Getting utility from LLMs

I've worked for GitHub for ~4 years at this point and for the past year or so Copilot has been at the centre of seemingly everything - figuring out how to incorporate LLMs into your work life is strongly encouraged. There are a lot of things I don't like about LLMs in terms of the impact on the tech sector, on the planet, and aspects of the tools themselves. However, if they're here to stay once the bubble bursts and we can make supporting them sustainable then having some ideas on how they can be used to good effect by leveraging what they're good at is probably positive.

In this post I'm focusing on what I've found and thought of in the context of my work with the help of smart people handing me good ideas (thank you again AI for the rest of us), various folks on LinkedIn, and my own trial and error.

I'm writing this by hand by the way.

Individual work

Where is the doc?

LLMs are extremely useful for fuzzy-searching large bodies of documentation for references even in cases where sorting is poor and doc titles are misleading. As long as you take any summary given to you with a healty dose of doubt you'll probably save hours by using this to dredge up source material from reliable but unwiedly sources (I use this method for looking up Azure docs).

Sort of wish people would just curate their docs better though.

When I'm writing code or I don't want to

For something small and well defined but unfamiliar or a bit tedious, LLMs can write a reasonably decent solution - this is especially the case if you provide your assistant with an instruction to keep asking clarifying questions until you tell it you've given enough detail and want it to build something. That said, it's worth examining whether what you're trying to solve is best done by writing a bunch of code in the first place.

Within larger codebases I've found that I generally am more efficient writing the code myself and using copilot to assist by doing things like providing comparisons between the feature I'm writing and similar ones in the same code (so a sort of gap analysis) or helping to root out annoying bugs (sometimes it provides some pretty wild suggestions but at least it often provides some useable direction).

I'd much prefer pair programming for real with an actual human and I think that would produce better code, but having copilot produce small useful things in languages I want to learn or generate coding exercises for me to solve has been interesting and useful (and not help I could readily get from a person).

I hate review season

What did you do over the past six months that mean you're a good worker? I have no idea and maybe I suppressed it to save my dwindling sanity. I hate review season, but if we need to write the weird little school reports to stay employed then I guess I'll figure out a way to do it that doesn't make my head explode. Enter Copilot CLI.

Last year's innovation for my self-reviews was a posix script that I wrote and aliased to the word brag - this little delight allows me to shove a one-line explanation of how I've added great value or done something cool into a terminal command and then chuck it into a GitHub repo with a timestamp.

This was great - it captured work that didn't have directly associated PRs or issues and made writing reflections a lot easier.

This year's innovation was adding Copilot CLI to the mix - all I had to do was ask it to summarise my pushed brags and commits to work repos over the past 6 months, I got a neat summary to talk over with my manager in our next 1:1. I had to do some minor tweaking of style and content (there were some plausible but wacky inferences) but overall it took me much less time and pain compared to previous years for something that was quickly good enough.

Sort of wish we revamped review systems to be more engaging though (I worked at one company where they actually worked when taken seriously).

Team toil

What did we do?

I could definitely see scope for teams to take a similar approach to my "give me all my commits from the past six months" and generate weekly snippets, save management toil in reporting work up, and help track what's really being done versus what your team says it does. Grabbing a summary of closed issues, pending reviews, or mentions across various platforms could really help with making work more visible and easier to spot patterns in.

For teams that are split across timezones this could be a serious gamechanger compared to having to wake up to back-scroll on shorter cadences (daily) and prevent siloing in the longer term.

This sort of information gathering removes the labour of tracking shipped work and provides a shared summary to trigger more interesting and in-depth conversations about work done.

What are we planning to do?

Project boards, opened issues, draft PRs - all can be fodder for the machine, and sprint/iteration/quarterly planning.

As someone in a remotely distributed team that works on many different projects, this ability to turn individual commitments into a more consolidated summary that the whole team can review can be very powerful. Importantly, it helps you stay grounded in the reality of what work has really been done - at least from one source and allows teams to focus on capturing more intangible stuff in meetings (which is the main reason to have a meeting instead of a written artefact).

I think project planning across distributed teams could be enhanced greatly by LLMs as well. Initial planning meetings can sometimes be chaotic and who gets to contribute can be determined more by group dynamic than relevance. What if your meeting was based off a shared proposal put together asynchronously? LLM assisstants, I've found, are actually really good at asking clarifying questions based off an initial statement - have each team member create a spec from their own perspective on what needs doing via this method and then have an LLM compare these specs, create a structured document with comparisons and trade-offs, and base your initial conversation on that.

Each individual spec would probably take about 10 mins, as would creating the initial planning draft with options, and once you've tweaked that as a team you can get your LLM to refine it into something more like a project plan with architecture diagrams. Heck, follow-up by seeing what copilot can generate from it and have a conversation about the thing in front of you instead of working from a blank slate.