You don't need to use AI to be a good software engineer

At my previous place of employment, like many other workplaces, AI hype was seeping in through already-existing cracks (e.g. funding shortfalls, leadership failings, etc).

About a year prior to when I presented these slides, at another workshop, I'd heard a speaker say, unchallenged, that anybody who refused to use AI should be fired for "refusing to use modern tools". After that, I tried to give presentations at these things whenever I reasonably could that challenged the almost exclusively pro-AI viewpoints of other speakers.

I gave this presentation during my last week there.

I have a degree in computational mathematics. This may be naïve of me, but I believe that if people had a better understanding of the math underneath LLMs, a lot of the mystery would be punctured and the inherent limitations of the technology would be more obvious. There's nothing like building your own ML model, from scratch, in Fortran 90, to break you from any illusions that these models could have any form of consciousness.

As I was looking for how other people have explained this, the explainers that made the most sense to me were by youtuber 3Blue1Brown.

There's also this great interactive visualization by The Polo Club of Data Science at Georgia Tech, my grad school alma mater, that I highly recommend playing around with.

But anyway, what are LLMs actually doing.

I think we're all pretty familiar with something like this--something that resembles a chat interface. You type something in, then you get some text back.

But what actually happens between those messages? How do we get from your input to the text you get back?

The first thing that happens, is that your initial message is converted from text to a vector representation. It gets broken into “tokens”. I’m showing them being broken into words here, and technically a token might be part of a word or punctuation or something, but for simplicity, by word is close enough here.

Each of these tokens is a huge vector. The values in this vector are relatively inscrutable. Each one doesn't only represent the word itself, but can be used to determine the relationship between itself and other words.

In the initial vector encodings, the encoding for "python" will always be the same. It will not distinguish between the python in our original prompt from the python in the sentence "Help! My pet python escaped!".

But, if you’ve ever used a chatbot, you know LLMs can clearly distinguish somehow between a pet python and the python programming language. So, how are they doing that?

They're doing it with transformers, the "T" in ChatGPT. Transformer blocks are made of attention layers and multilayer perception layers.

In an attention layer, the LLM sorts out how tokens in an input relate to each other.

In our example, we could say that "programming" and "language" modify or attend to the python token and pet attend to python in the second. They provide a more specific meaning or context to what the python token should represent.

So, the original encoding of python in that huge vector encoding space is modified slightly to more closely “point to” separate meanings, a python programming language in the top case and “python pet” in the second.

This is all done by training weights and other vectors. It’s 100% matrix math that does this; that’s the purpose of the "attention layers", to refine the meaning of each token based on other tokens in the input.

The other layers in transformers are multi-layer perceptron, or MLPs. These layers also refines the specific meanings of the vector representation of the tokens as they pass through with weighting matrices learned from the training data and softmax functions.

Then, the tokens pass through another attention layer, another MLP layer, and so on, further and further refining the original input tokens to get a really, impressively specific encoding of how all the tokens relate to each other.

And both these layers, the attention and the MLP layers, are made up of matrix multiplication and vector operations.

Definitely no “consciousness”, just math.

So finally, to put it all together, at the end of the transformer blocks, what we get, is a list of probabilities for the next token, which is what is used to build the “response” from the LLM.

A text input, to tokens encoded as vectors, and passed through an astonishing amount of matrix math, which, at the end, spit out what is a statistically likely next word. And that’s the bit I really want to emphasize--that LLM outputs are whatever would look statistically likely.

And, granted, that by itself is a bit nifty, that a huge pile of matrix multiplications end up outputting something comprehensible and recognizable as human language. But, what LLM responses are not is comprehensive, intelligent, or necessarily even correct, but purely statistically likely.

A lot of the inherent limitations of LLMs can be clearly inferred by understanding their design.

Perhaps the most obvious, LLMs will encode and reproduce incorrect and even harmful concepts.

I know they won’t tell you to put glue on your pizza any more, but, if you understand the underlying technology, you can kind of see how it got there. The concept of “eating glue” isn’t exactly not something that you could find being discussed on the internet, so, eventually, an LLM determined it was statistically likely and regurgitated it.

And there’s no way to stop this entirely. It’s not like there’s a distinct “eat glue” weight that can be turned down. There’s also not a distinct “racism” dial or “sexism” dial to turn off or even a “good code” to turn up.

You can use human in the loop training to punish responses that have characteristics you don’t want coming out the other side and reward those you do, but you absolutely cannot completely eliminate incorrectness or harmful concepts from LLMs.

Additionally, I’m not sure if you’ve ever looked at or tried to review LLM generated code, but from my experience, what I’ve learned is that it is much, much harder to tell when something is subtly wrong than when it’s blatantly and obviously wrong, and that it will take significantly more effort to check if every single line of code is doing exactly what it says it's doing.

And finally, if you are using these tools to generate code, ask yourself, is statistically average code generated by something that does not, can not, hold an internal understanding of your project really code you want to sign your name to?

Another obvious limitation is that LLMs will reproduce training data.

I can’t tell you exactly what’s in the training data, because the companies scraping all this data don’t exactly reveal that info publicly. It's everything they can get their hands on. Everything.

And, of course, this leads to obvious licensing and copyright issues.

This is a figure from the NYT OpenAI lawsuit. Everything in red represents a word for word match to a NYT article that was reproduced exactly by chatgpt, If it's hard to see, most of the passage is red.

I’m aware that courts might be still deciding or whatever if this counts as “fair use”, but from a non-legal perspective, I don’t think anybody could make the argument that this isn’t professionally troublesome.

I invite you to imagine if this is some code you’ve written. Do you check every function you’ve used a chatbot to write to make sure it wasn’t pulled line for line from another major package? Would you otherwise be okay going and copying and pasting large sections from major packages like numpy or astropy?

This isn’t something that is a remote possibility, using chatbots to generate code will eventually straight up lift code from another published package, which, at minimum, should be embarrassing for whoever generated it.

Given these inherent limitations of LLMs, I think there are incredibly foreseeable consequences of integrating LLMs into our work.

It's harder to trust the work your coworkers do.

It will be harder to trust the work done by others in our broader communities.

Using these tools in public-facing areas will lead to decreased trust in us*.

I like to think about the failure cases here. For example, a README page I write will never say anything I didn’t put on it. Documentation pages shouldn’t be able to misrepresent how to actually use the tool. Integrating LLMs into things like these makes those failure cases not a possibility, but an inevitability.

* To my previous employer, the perception of public trust would (or should) have been very very important.

“AI” does not have to be inevitable. Consider who it serves to not even try to imagine a different future where these products aren’t integrated into every aspect of our jobs and lives.

Messaging about how “you’re going to be left behind” is a marketing tool. AI is interpolative. It always reproduce what’s been fed into it.

There isn’t a way to never have them return things that aren't accurate.

You aren't going to fall behind if you refuse to use AI.

Don't use it.

The claim from [institutions like my old employer] that “things are moving so fast” when they decide not to make a coherent AI policy is inadequate at this point. Chat bots and image generators have been widely available for years. We need clear messaging and guidelines for their usage now.

Trust in public science is at an alarming low and it’s irresponsible to allow autonomous, or “agentic” chatbots the access we’ve already given them.

Who am I to suggest guidelines? Nobody. I shouldn’t be the one suggesting guidelines. But in the silence from our organization, we have to start with something.

Since I’m already here making suggestions, my moonshot ask for AI policies would look something like:

• First, heavily restrict or straight up disallow the use of LLMs for professional work, and especially on shared systems like HPCs
• Next, no public LLM interfaces in documentation, databases, anything.
• Also, no generated images or text in presentations, websites, newsletters etc
• And in professional or student training we create, any AI education content should prioritize the underlying principles of AI/ML, not be focused on how to integrate these tools with the work we do.

Finally, I want to ask that whenever faced with a question of “how do we integrate AI safely and responsibly”, that the option for refusal and rejection is always an option. Less “how should we do this” to “should we”.