AI-Assisted Programming
We have recently seen an explosion in the use of artificial intelligence (AI), with products like ChatGPT, GitHub Copilot, Tabnine, and many others. They are very useful and interesting tools, but not as much as the media claims.
Think of these tools as a machine that predicts which word is most likely to come next, adds it to the suggestion, and does this repeatedly in a loop until it reaches a certain number of words or until the probability of a good guess drops below an acceptable threshold, whichever comes first.
This causes some problems. For example, these tools don’t deal well with formal logic. If you ask for something like create an HTTP mux without using gorilla mux, it’s almost certain the tool will use gorilla mux. Likewise, they don’t handle math well; the algorithm is focused on conversation and doesn’t do calculations.
Even so, I’ve been using some of these tools since they launched, both directly and through their APIs, integrating them with my own projects. Despite these limitations, the results have been pretty good, especially for repetitive tasks.
Security
We are still discovering the security problems this kind of tool can cause. At a minimum, it’s as bad as the telemetry that already happens with several editors.
Basically everything you type is sent to the service, and they use your code to improve the product, feed the model back, and improve results, from knowing which code you accepted or rejected, to your edits and all of your code.
So some old rules apply very well here:
- Never use real keys in your code, don’t commit them, and so on. In fact, for work-related things it’s better not to even have access to anything; leave the access keys with the security team.
- The same care applies to customer data. Always use a mock, never real data.
- Be careful with code if it’s truly confidential. Today this isn’t a concern for me, since my work mostly involves interacting with APIs and there’s nothing really secret about it; the APIs are documented on the internet and anyone can access them. But early in my career there was a time when I worked on products where the equations were the heart of the business and kept under lock and key.
- In that case I wouldn’t use any of the AI products. It would be better to write a small library or API without any of these tools and have the rest of your system simply consume that library.
There is also an effort underway to create security filters that prevent sensitive information from leaking.
For now, as long as the proper precautions are taken, my opinion is that the level of security is acceptable. Many of my colleagues don’t share this opinion; it’s up to you to decide whether it’s worth the risk or not.
Reliability of the Results
For short snippets and more predictable parts, the reliability of the suggestions is excellent, but far from perfect. You need to closely supervise what’s being suggested. The inside joke we coined is that it’s like having a very inexperienced intern who is also very enthusiastic.
The License Problem
The available AI products were trained on everything developers could get their hands on, including GitHub, Stack Overflow, and so on.
The problem is that no attention was paid to software licenses, so you may end up accidentally including GPL code in your program because the tool suggested the snippet and you have no idea where it came from.
I don’t see copying small snippets of code as a problem, partly because I believe code should never have a license or a patent in the first place. Day to day, no one cares about copying a snippet of code from Stack Overflow and using it in their project.
As with the security question, this is an open issue; it depends on what you believe and the risks you’re willing to take.
Costs
At the time I’m writing this, GitHub Copilot costs $100 USD/year. In my opinion, it paid for itself in the first month; it saved me a lot of time.
The ChatGPT API has a variable cost depending on the size of the prompt and the model you choose. Since the price can vary quite a bit depending on what you want to create, there’s no easy way for me to list the values here. That said, in my tests the cost was quite affordable as long as you don’t have a very large volume of tokens.
Writing Good Prompts
Writing a good prompt has a huge impact on the quality of the AI’s response. Here are some tips.
- The less ambiguity, the better. It seems obvious, but when we’re talking we accidentally add a lot of ambiguity that we don’t notice.
- Writing correctly helps. In my tests, taking a little care with grammar and spelling produced slightly better results.
- Try to phrase sentences positively. “Do this” is better than “don’t do that,” because the AI doesn’t understand formal logic. A few tries may be needed before you get the result you want.
- You can specify things you want in the response. For example, you can ask for “all code snippets to be formatted with the markdown used by Telegram,” or, if the question isn’t related to Go, to return only the string “—1234—”. Then you can intercept that result and take the appropriate action.
- Keep the prompt size under control. Measuring a prompt is tricky; the number of tokens is roughly 3.5 per English word, and you have a fairly limited token budget. If you pass too large a prompt to OpenAI’s API, it will simply return an error.
Disabling GitHub Copilot
It can be useful to disable GitHub Copilot from time to time. Sometimes you know exactly what you want to write and don’t need help, and the AI’s suggestions are just a distraction. When that happens, it’s nice to be able to temporarily disable the AI. In vim/nvim this can be done with the commands “:Copilot disable” and “:Copilot enable”, or in VSC you can click the Copilot icon in the bottom-right corner of the screen.
To make things easier and type a little less, I created these two commands that I use in nvim.
cnoreabbrev dcp Copilot disable
cnoreabbrev ecp Copilot enable
Beyond the Editor
Other tools are emerging to help with developers’ everyday tasks beyond writing code itself. Some are more interesting than others.
On the Command Line
At the time I’m writing this, GitHub Copilot for the command line is in an experimental phase. Personally, I didn’t like the interface, and the results are below expectations.
Filling in the Commit
It’s easy and fun to extend git, so it’s no surprise that we have several tools to help write commits.
Community Chat
One of the most fun experiences with the OpenAI API was creating a bot for the Telegram channel of the Go study group.
I wanted the bot not to respond to a specific command. Instead, it would read all the messages and, when it detected that a message was a question, it would answer.
The result was great, but I ran into problems with prompt size, and as the volume of messages grew, the cost grew beyond what I was willing to spend on experimentation. I plan to revisit this experiment.
Do It Yourself
People have been using AI for some time, but they forget about it as it becomes common and expected in everyday life. For example, a spam filter is nothing more than a classifier (a Bayesian algorithm) trained to distinguish between good and bad emails; you can use the same algorithm to classify anything. And of course this is just the beginning; there’s a huge area to have fun with.