Code generation tools are easy to start, but hard to master

I’ve been stuck in a constant loop of subscribe-unsubscribe from a code-generating AI service for about a year now.

I’m a software consultant and noticing a pattern in my perpetual on-again, off-again relationship with Copilot that reminds me of how I felt during my undergrad. Perpetually flipping between excitement and exhaustion, hype and despair, on a roller-coaster that goes something like:

1. Class registration opens. I look at options and get excited by the possibilities.
2. Term begins. The first few days are magic. New ideas are plentiful and I learn easily. I feel I could do this forever.
3. The learning curve climbs. I have to apply knowledge myself in new ways. It’s still fun, but I get pushed to the edge of my understanding.
4. Deadlines loom, as does my discomfort. Is education about blindly following directions? Or can I follow my heart to explore new things? I question why I do this, but want to see it through.
5. Panic sets in. I have to triage or drown. I fall back on habits I often hate but that worked before. Frustration turns inward and I blame myself for not being ‘on top of it’. Fun is a distant memory.
6. Somehow, I finish everything. It feels like nothing new came from this term. I’m tired. I’m getting out, and will never come back.
7. I rest. My brain decompresses and can actually internalize what I flooded it with. Time washes away the panic. I start to feel thankful for what I learned along the way.
8. Class registration opens…

Wait, I recognize this one

My experience as a software developer with Copilot matches pretty closely. I subscribe to the tool. I dig into a problem, start to write some code, and I’ll be! The autocomplete is so accurate I’m certain it read my mind. Except I didn’t even consciously think that yet, so it can also see the future. I know that’s what I wanted.

I dig deeper. The problem becomes more esoteric, painted with prior art that I need to account for.

Still, the incredibly fancy autocomplete AI tool hits the right thing often, and it’s done well for me so far. It sped my brain up, even if I have to stop and rework a suggestion or two for my context.

The reality of “I need to have something working” looms. My internal clock tends to turn up the pressure earlier than the deadline I’ve agreed to (or estimated). I’m deep in the problem, and have a nice mind-palace built of the system. Indirect cause and effect are clear to me.

The AI, in its rush to help, spits out a subtly incorrect block of code. On its own, incorrect is fine. More than once, however, the tool has spit out a brain worm. A new variable name, or a solution that is just reasonable enough to hijack whatever was going to come out of my subconscious.

I cannot un-think the suggestion, and the thing I wanted disappears. So I follow the AI and inevitably bonk my head on a syntax thing, or a side effect. The energy it takes to resist the magic computer breaks down the mindpalace.

Suddenly I look up, and I’m yak shaving again.

Now I’m upset, so I fall back to habits and old ideas to solve problems. I turn off Copilot and return to slower ways to work that have been effective so far. I wrap up the task and move onto something new. Somewhere a few weeks down the road, I remember I’m paying for a tool I’m not using.

Rest, repeat.

You might say I’m a bit of a legend

My brain is always hooked by the AI candy for the rote things.

Little side project idea? Auto-complete your way through the standard wiring to get at the fun part of the idea. Somewhere between that early joy and regular day-to-day work, I lose my control over the tool and it starts to control me.

I’m too aware of my fallible human nature, so I’m certain a lot of this problem has to do with operator error.

It’s probable I’m the character from an urban legend that came before self-driving cars were real, when cruise control was the hot new ticket in automobiles. The story is about a man who tries out the flashy cruise control feature on his brand new motor-home. He’s cruising on the highway and fires up that cool new tech. Satisfied, he gets up from the drivers seat and heads back to get a cup of coffee. Naturally the vehicle flies off the road, flips, and man sues the manufacturer because “instructions unclear.”

Maybe that’s exactly me with Copilot? (I promise I won’t bring a frivolous lawsuit!)

It’s rare when we get to spend time where a general code-completion tool is inherently an asset; the fundamentals of a framework, the well-trodden paths of data manipulation, clever JavaScript one-liners, or clear database queries. Especially in a language you don’t know. Those situations are the showcase for the value these tools hold.

For me, it helps shore up my leaky memory that cannot hold bash scripting or regex (even though I’d rather keep those things and forget the lyrics to I Want You by Savage Garden or the theme song to Arthur). I’ve also had success and fun doing an adversarial test-driven flow where I use the AI to write the code to pass the tests I write.

I get better results directing the LLM with a persona (Cranky Staff Engineer), and a method of thinking / responding (in the style of a succinct and firm pull request review of code you don’t like.)

Over time, I am writing better prompts to constrain the dreaming machine to help me be creative in turn, and in context. In those ways, I squeeze a lot more context-driven work out of a few suggestions.

Frankly, very little of what I’ve learned is helpful to you! It’s often not helpful to me, which is why I didn’t write a post called “How To Be A Staff Engineer By Using Copilot.”

This is more art than repeatable experiment, and it goes against a lot of ways I think when I am being a proper Software Consultant.

I’m still very hesitant to pull the tool off the shelf on client work, even on teams who are fully embracing them. I’m a huge sucker for fan of paying for tools really smart people have made that paper over my gaps. (Hi JetBrains, I love you.)

Due respect to the progress the builders of these tools have made. I’m poking fun at Copilot, but it’s grown leaps and bounds. There’s a reason I keep coming back. I just can’t quite figure out what it is, yet.

Every tool in its place

I truly want to find where Copilot can be a core part of my workflow. I’m just struggling to find where that’s productive when me and the tool are out of our depth together. It gets turned off, forgotten for a couple weeks.

Put another way, I have a blast until I get to the part of term where deadlines are looming, and I have to triage back to default habits and tools I know how to work with. That’s when I need my tools that don’t feel like they are actively trying to flip my winnebago on the highway. But I can think of a few ways I’m not using the tool to its greatest effect.

It’s a new term. I’m full of excitement and I know I can do it better this time! That registration button is open. I click it, and turn on the plugin in my IDE…

Are you a developer who’s tried an AI generation tool?

Join the conversation about this post over on our N.E.A.T. community (Not Everything’s About Technology).