It is no surprise to anyone that knows me, that I am no fan of LLMs and the hype that follows them; in fact, most of the people who know me have accurately assumed that I’m not particularly enthusiastic without the need to even ask. I’m asked why I dislike them, but never seem to be asked whether I dislike them. My answer changes all of the time; I flip between criticising their environmental impact, their effect on capital allocation, the extreme labour theft, determinism, etc and avoid talking about the technology itself because debating whether a machine can autonomously do X or Y seems unimportant when it costs $20 million and burns though tons of natural gas in the process; It’s like a debate about the morality of soviet gulags focusing mostly on whether the food was good or not.
But the thing that interests me is why the people close to me already know that I disapprove. I’ve not rejected new technology before, at least no more than the next man. What makes this technology different?
The Vibes Are Off Link to heading
I think it goes without saying that the way people push and hype LLMs is annoying, suspicious, but also religious. AGI (which is just around the corner bro) is going to be humanity’s saviour! It will enable us to colonise Mars (famous for its beautiful nature and fresh air) and solve world hunger (presumably due to how edible last-gen Nvidia cards are?)! These are claims wholly believed by the technology’s proponents, but also mostly by its opponents, who seem to buy into everything other than the promise of utopia.
I think dubious marketing claims are not enough to hate a thing, but we are getting closer. I remember once taking the night train from Oslo to Bergen. I thought it would be fine to get a chair instead of a bed (a terrible mistake) so of course ended up in the food car for the entire night chatting to a middle aged woman who was becoming less intelligible with every can of beer, and a jazz musician who was a fun mixture of kind, enthusiastic and undiagnosed adult ADHD. The woman turned out to be one of the first women to ever work on an oil rig, so as you can imagine, we spent most of the night listening to crazy stories of discrimination, defiance, revenge and solidarity, broken only by her recurrent hysterical laughter at the fact that a passenger had purchased a hot-dog, earlier in the night, for his girlfriend as a token of love so that she would let him back into their shared sleeping compartment. But at around 4 in the morning, the topic changed. The musician had recently gone to try and sleep in his firm, allocated chair and this was the moment that the, now fairly intoxicated, woman decided to unload her political views. It started with covid vaccine conspiracy and swiftly moved on to her telling me that white people can’t go to Birmingham any more and so on and so on. Its uncomfortable being in those situations as an immigrant, but the madness of telling a British person that the city in Britain that they were in just a few weeks prior have sharia law was not lost on me. I pushed back, of course I did. I outright rejected her views about the UK, even inviting her to play “count the white people” in the video circulating in rightwing circles of Robert Jenrick MP walking through Birmingham complaining that he can’t see any white people (plot twist, almost everyone in the background of the video is white). I asked question after question about her vaccine views, hoping that she would find some kind of internal contradiction or that I would discover that our worlds are not so far apart, but instead the chasm only grew. Further and further apart we drifted until it seemed absurd that a small food car could contain two people so vastly far away from one another. I didn’t feel angry about her views; I don’t know if I was even sad, but I defiantly felt lonely. I felt the weight of the futility of trying to make a connection with someone who’s eyes delivered to her brain the inverse of what my eyes delivered to my own. It was the first time I’d experienced the phenomenon, and sadly has not been the last. It’s that feeling that I feel when I see LLMs fill up my feed. That feeling that the reality in which you occupy is light-years away from the reality of another person.
But for the purposes of working out why the vibes are off, it’s helpful! It tells me that this diametric opposition, which my contemporaries can smell on me, is almost definitely political or philosophical, but it goes deeper than labour and intellectual property theft. It is around now that a professor in our department sent me a link to a paper on The TESCREAL bundle. Read it if you haven’t already. And please don’t spend as much time trying to work out how to pronounce TESCREAL as I have, it distracts from how awesome the paper is. It asks the question “why do they care so much about AGI?” I mean it’s a good question right? It would be way cheaper to make way more capable AI systems tuned to specific tasks. Where does this obsession come from? And the answer is eugenics. Yes. Eugenics. I’ll write it three more times so that you know it isn’t one of my many typos: Eugenics eugenics eugenics. The link isn’t even dubious. You can follow the history of philosophy from first generation eugenics to AGI obsession. If you don’t believe me, go read the paper and come back when you are done. I will not insult it by summarising it in my own words.
But just because the philosophic origin of the AGI obsession is eugenics, doesn’t mean that all people who love AGI secretly want IQ tests and embryo selection. You can find a thing cool without buying into the whole why and where.
I was wrong Link to heading
So it turns out that you can’t love AGI and join in with the hype without wanting embryo selection based on IQ, or at least not logically constantly. I was a bit slow to this awful realisation, probably because I like to think positively about my fellow humans, but an event that took place two weeks ago changed my mind.
The event was a coffee discussion about the Hugging Face attack. It was prompted because in our department discord, a few PhD students were quite aggressively pushing claims about the ramifications of this attack. The professor who sent me the TESCREAL paper agreed to chat with them (probably so all of the employees in our department could be saved from the constant pinging of discord notifications). I of course came. And it was about as productive as… well… as these things often are. They would start talking about AI agents as if they have hopes and dreams, we would interrupt and ask them to use language which doesn’t anthropomorphise. They would push back saying that when using such language, we make better predictions about the behaviour of these systems and so it is the most appropriate language to use irrespective of the connotations. We would then go on to argue that the anthropomorphising language presupposes the very things they want to conclude and so is an obstacle to us engaging in the discussion, especially given that we understand the process in which an LLM transforms text and have perfectly good language to discuss the situation with which leads to better predictions. And we’d go back to the beginning. We went around and around in circles. At some point, the professor left because he had more important things to be doing with his time. I sure don’t! Hours (I am not exaggerating) we went around in various circles. We visited some other places along some of the arcs. I learnt that the particular chaps I was arguing with think that intelligence is a 1 dimensional metric which machines and people can be measured by, and that any push back against this core belief was met with “It"s just a fact”. Right. Sure. I wonder where we will travel next when we circle back to them describing these system with overly anthropomorphising language?
But at some point I realised something. Believing that “that intelligence is a 1 dimensional metric which machines and people can be measured by” is a eugenics belief. We are back to IQ and skull measuring. I thought I’d test the water. I said something about eugenics (I don’t recall what exactly). It was of course met with “you can’t call everything you don’t like Nazi” which is true. I didn’t know how to play it after that, but it turns out I didn’t need to play another card. Both of the people who had just a second ago denied that they were eugenicists started to defend embryo selection based on genes linked to higher IQs. I couldn’t believe it. The other guy started rambling on about “genetic drift”, that humans no longer have “natural selection” and so we are “just breeding more and more idiots1”. At this point I stopped engaging but it kept me up at night. I’m not going to claim that every argument about hugging face will turn out the way mine did, nor will I claim that every vibe coder is a closet eugenicist, but the two guys on the opposing team did make me realise something, although I disagree with their assumptions, their world view is frighteningly consistent.
The Shortest Alt-Right Pipeline Link to heading
I’m not happy. But at least I understand why the vibes felt so off; it’s because when I see these people on the internet or in person, my limbic system has been hearing nothing but Wagner2. I feel that vast chasm again.
So I should probably finish by being precise about the pipeline/logical consistency that I found:
- You believe AGI is achievable through the training of LLMs and/or reinforcement learning.
- Then, you believe that general intelligence is expressible as a fitness function and measurable.
- You believe that AGI is a good thing to have
- Then you believe that the world is better with more generally “intelligent” entities
- By 2 and 4, plus the knowledge that resources are scarce, lead you to wonder, why not do it to people too?
I have seen papers that link IQ scores to certain genes. I have no idea if their results are dubious because I’m too hung up on not believing that IQ has any relevance or even link to intelligence and so they fall into the ‘I don’t care’ pile.
The TESCREAL paper does a great job at showing how a pre-existing belief in eugenics can lead you to loving AI. But my observation is the other way around. It’s that buying into the marketing about LLMs (at least with respect to AGI, killing all humans, (obscene claim)+) places you firmly on point 1. If you feel excitement about it then you have achieved point 3 and your only hope to be logically consistent is to buy a skull measurer and join MENSA.
But at least the people that know me can sense, without asking, that I am in no way a eugenicist.