Changes Notes generated by AI are soulless

So recently, I have started having my run-ins with AI-generated change notes and PR descriptions. I won’t say where, or in which product. I won’t say if this is for some personal project or something at work. I will not name the programming language(s) involved in the software project(s). But let’s just say that AI-generated change notes and pull request descriptions are now a daily part of my life.

Change notes go by many names: change descriptions, code review notes, pull request descriptions. But they are all pointing to the same idea: you as a software engineer have made a change in your software program and to document what you have changed, your motivations for the change, and how tested your changes, you write some notes capturing those ideas — the what, why and how — in change notes, and you submit that change as a patch to the program, or you send out the change for others on your team to review. The change notes can be invaluable for others while reviewing (your) proposed changes, or when digging through code history to understand why certain changes got made. Personally, I find change notes as a useful documenting tool to remind myself of the reasons I made the change, i.e., the why, and how I tested the code, i.e., the how. Good value all round.

Enter AI.

AI-generated change notes are good at one thing: to offer banal descriptions of the specific code edits to a program that got made in the overall change. And the AI-created description has the potential to get into the minutiae of the code edits, in a seeming attempt to put the reader/reviewer into a state of sleep. But their real shortcomings are better highlighted in what they do not do.

They really do not offer any insight into why the change might have been made, or how the developer might have tested it … or should test it (may be generate me a test plan?). AI cannot possibly know why I do what I do, or how I might have tested it. But their seeming inability to even venture a guess tells me one of three things: (a) the product team behind this automated-change-notes feature has not gotten to it yet; or (b) they tried it and it was really bad; or (c) they tried it, and it just cannot do it.

All of this would not be a problem in and of itself. In fact, I would argue that if the developers generally are good about writing change notes, then leveraging the description of what the change is (assuming it is correct) to then amend it to include the motivation and testing approach for the change is a fantastic thing. It is a future I would love to live in. But this breaks down when the developers were not great at writing such change notes to begin with. In fact, let’s face facts: developers are lazy. I mean, I am lazy. I did all the work, now you want me to describe it as well? ugh!

When you give a lazy developer like me — who writes one-line change notes to describe a relatively nuanced and/or complex code change — a tool that generates copious amounts of change notes will feel like a boon! All of a sudden, my change notes look aplenty and it seems like I have spent time in writing them. It gives me an excuse to not write change notes entirely if I am being perfectly honest. And I am happy to ride this slippery slope by not verifying/checking if the change notes are correct.

And the real heart and soul behind what makes programming worthwhile: the why and the how, fall by the wayside. And in my experience as a programmer, why I wrote some code, or made a code change, probably has a bigger impact on the quality of that code than what the final version of my program ended up being. The motivations behind the program also end up influencing how to best test that program, or a change in the program.

May be change notes were never that great because they aren’t typically a high priority thing for developers. But there was an honesty when I saw laconic, or one-line change notes. Now, it feels like we are just masking over that reality with pointless descriptions that at best may just be redundant, or worse may simply be incorrect.

– vkp