Romance of Five Kingdoms
Login Register
All posts

Romance of Five Kingdoms uses transformer-based models, more commonly (if somewhat misleadingly) known as generative AI, a bunch. It’s used for coding, bug hunting, animating NPCs and living world features, producing stock images, and more. In the past few years this technology has become an increasing point of political and social tension, and more and more right-wing coded. Since that’s not close to where I live politically, I thought I’d explain why I don’t really agree with the commonly understood story. Most people probably don’t care about this and already have their own strong opinions, which is fine, but if you’re curious, or yourself more on the fence, it might be helpful.

So first, let’s go over how we got here. One big moment was GPT-3.5: OpenAI released a chatbot onto the internet and people were amazed at how humanlike it seemed.1 The other big thing was the scaling laws paper, released in 2020, which seemed to show that these models got ‘smarter’ the more compute and data you put into them, in a linear way that showed no signs of slowing down.2 Thus it was hypothesised that if you just kept scaling, you would eventually create something much, much smarter than any human: a superintelligence.

One problem the companies faced at the time, however, was that there was no clear commercial application for the models, but they needed lots of money to do their training and win this AI superintelligence scaling race. Fortunately for them, investors are pretty used to giving money to companies that aren’t currently profitable, and there’s a long history of Silicon Valley companies in particular running at a loss for a long time while they build up, until they establish some kind of monopoly and can then turn profitable.

This meant that a company’s success was largely driven by the degree to which it could convince people that the technology would at some point in the future make money. A lot of people have thought it very strange that AI CEOs are always talking about how their product is going to steal your job and maybe end the world. That would be very odd if what they wanted was for normal people to be their users, but it’s very understandable if what they care about is investors, and convincing those investors that this technology will one day make loads of money.

One thing that started to happen as the tech got hyped is that it folded very neatly into a certain right-wing anti-talent, anti-elite idea. One thing the far right, particularly wealthy right-wingers, really dislike is human talent. Elon Musk can be the richest guy, but he wants to be funny and isn’t, and no amount of money can make him funny. In general, power and money can’t create talent or reliably control the people who have it. Much has been written about the rightward swing of tech CEOs, and how a lot of this was down to them feeling forced to make their companies more woke than they wanted them to be. If you want the smartest and most talented people, they’re mostly going to be progressive, and they’re mostly going to be well paid enough at these companies to leave for another job if they don’t like your politics. The idea of a computer that’s just as smart as or smarter than the smartest person, that can do creative things as well, like draw or write fiction, and that never says no, never disagrees, and will always just do what you say because you have the money, is extremely appealing to these people, and they became huge cheerleaders for the technology.

Then people can also point out that these models are trained on the writing or art of millions of people, but none of those people will get paid for that, or have any say in what the models do or are trained for. And it seems pretty obviously unfair that if there are two inputs to the end product, only one side is getting compensated or getting a say.

On top of that you have the slop problem. The internet has been going downhill in quality for a long time, but one thing that somewhat mitigated this was that it takes effort to make stuff. Suddenly, people could make stuff way, way more easily, so very quickly the internet went from like 75% slop to like 90% slop, which was generally unpleasant for everyone.

And then you have companies forcing the technology into lots of products and services where it isn’t needed or isn’t helpful, often making things worse, and you might’ve seen it be crazy bad at something that seems simple, like counting the number of Rs in strawberry,3 or not knowing you shouldn’t put glue on pizza.4

Then there’s the environmental impact. Data centres need electricity, and if you want to scale to superintelligence you need a lot of it, which is often produced with carbon fuels or increases energy prices. The data centres also need cooling, and the preferred way of doing that requires clean water. You can find stories of a data centre going up somewhere, the clean water going to it, and the town being left with dirty water for actually, like, living.

Then there’s the bubble problem, where the investment in the technology seems to far outstrip its promise, leading almost everyone to conclude that this is a speculative bubble that’s going to cause a market crash.

So you get to this point where it’s like: this is a technology that’s going to steal your job, do every job it takes badly and unreliably, championed and controlled by seemingly the worst people on the planet, stealing from everyone constantly, filling the internet with crap, fucking up the economy, and destroying the planet. It’s certainly no wonder that the vast majority of people who hear that are going to say: I hate that and I want it to die.

So I think that’s a completely understandable position. However, now I want to complicate the narrative a little.

AI is a marketing term; it doesn’t really mean anything. It’s just generally applied to any sophisticated technology someone wants to make sound more impressive. LLMs are statistical models of language as used by people, and because language is a very human thing, this makes them strikingly human-like compared to other computer-based technologies. However, language is not the same as reason or knowledge. They overlap, and there are things you can do to make them overlap more tightly, but they are always going to be different. It’s like translating from English to German: there are good and bad translations, but there can never be a perfect translation.

This is why these models can seem strikingly intelligent in some cases and completely moronic in others; assessing them as either dumb or smart is missing the point. It’s also why you don’t hear much about scaling laws or superintelligence anymore. Scaling laws didn’t produce superintelligence because they were never scaling ‘intelligence’. They were scaling language.

This doesn’t mean they’re productively useless, though. On a well-defined, discrete task where outputs can be checked or low levels of mistakes can be tolerated, they can be extremely powerful, and there are also specific tasks closely related to language at which they particularly excel. One reason they’ve proven so good at coding is that coding is largely a language task and its outputs can be checked: you can see if a bit of code compiles, or whether a function returns the output you expect.

However, you can start to see how this creates a lot of problems for the “AI is going to take your job, be superintelligent, or make talent and creativity a purchasable commodity” line. LLMs have no meaningful possibility of replacing any job that isn’t brainless and rote. They have no real broad understanding of a job, no sense of values with which to prioritise things, they go increasingly insane the longer they run unsupervised, and they don’t know when they’re lying. On the artistic front, AI is absolutely going to take the job of stock photographers and essay mill writers; it isn’t going to take the job of artists hired for their taste, aesthetic understanding, or creativity, or writers hired for their thoughts or wit or talent. And while there absolutely could be cases where using the technology means the work of what used to be six people can be done by four, it’s not the four doing what they did before plus two AI workers; it’s four people using AI for the more boring and repetitive parts of their job so they can be more productive overall. And if what the technology is doing isn’t replacing workers but making them more productive, then rather than saving two salaries, it’s pretty hard to argue you shouldn’t be giving a lot of that saved money to those four people, particularly since their jobs are now more intellectually challenging. While it’s a bit of a tangent, it’s also the case that even if AI did replace half the jobs, you could just have everyone work 20-hour weeks instead of half of people being unemployed.

It also means the idea that these models are going to supplant talent and creativity and turn them into purchasable commodities starts to look like the opposite of true. People’s creativity and creative intelligence become even more valuable when they’re the main things the LLMs can’t do. Since to make an LLM more creative you’d also have to make it lie more, and it doesn’t know when it’s lying.

Another thing a lot of people don’t know about is the increasing prevalence of open-weight models. Right now, the generally considered third smartest LLM is GLM 5.2, which is open weights, meaning anyone can just download it and run it on their own computer if it’s beefy enough. It’s actually much cheaper to make the second best model than the best one, partly because of distilling, which just means training on the output of the smarter model. Big AI companies are so freaked out by this that they’re lobbying the US government to try to get it to do something about it, and have supposedly even tried to add booby traps to their models to detect when they’re being distilled and give worse outputs.5 Definitely ironic that companies which trained on the output of others without consent are getting so up in arms about it being done to them.

A lot of the most obnoxious behaviour we’re seeing, from huge data centre build-outs to cramming AI into everything to talking about how it’s going to take all the jobs, is mostly a product of desperation. Here’s a quote from the Head of Strategic Futures at OpenAI: “One probable outcome of an open-weight-model-dominant world is full AI communism.” He goes on to describe this future of AI as a public good as a “dystopian hellscape.”6 There’s no clear roadmap for capitalists with a lot of money to control the technology, or to use it to make themselves money, and so they’re left just acting as confidently as they can for their shareholders and fellow capitalists: this totally is going to make them tonnes of money and increase their control and everything, at some point.

On top of all that, the technology is vastly more useful for poorer and more disadvantaged people. LLMs make pretty good academic tutors, since a lot of tutoring is language tasks, reworking some text into a question and so on. There was a study in Nigeria that showed quite significant benefits from giving students access to an LLM tutor.7 It’s still not as good as a capable human tutor, of course, but a lot of people can’t afford one of those. Asking a chatbot legal questions is much worse than having an actual lawyer, but much better than getting no advice at all.8 LLMs are vastly more beneficial as tools for people with less education and/or less money. This is also true for businesses and developers and the like: small and independent companies or endeavours benefit much more from an OK-if-you-check-it human stand-in for tasks they can’t do themselves, whereas big businesses can just afford to hire people.

I view this less as a fundamentally evil, anti-human capitalist technology, and more as an unusually human and socialist technology that evil anti-human capitalists are trying as hard as they can to control.

This is also why I think the IP theft view of all this is sort of backwards. For sure, if these models were controlled by one corporation, it would be unjust for the people who wrote the training data to get nothing and have no say. But when we’re talking about a situation where there are hundreds of models, lots of them free for anyone to use, I’d say the idea that this is a common good that can’t be controlled is actually good. Particularly given how IP works in practice.

This is kind of a tangent, but intellectual property really is among the most evil forces in the modern world. All property is to some degree arbitrary; it kind of boils down to the state saying, if anybody but you takes that, we’ll beat them up. But with physical property, there’s some sense behind it. It’s useful that if I bring a sandwich to work, nobody else can eat it. If someone just copies my sandwich recipe, though, I’m not actually any worse off. Intellectual property is a relatively recent phenomenon, because you need a pretty invasive state for it to even be possible to know that someone a state over is telling a story you told, or who came up with the idea for a gizmo first. For most of human development, if someone had a good idea, other people just used it freely, and they used the ideas of other people freely in return. Artists and musicians and the like were generally just paid to create art and music, usually by noble patrons.

While we’re generally told that IP exists to reward inventors and creatives, there are lots of other ways to do that which work much better. Stipends, awards, and grants are already frequently used in a lot of areas and do the same thing. They also do it better. With IP, you only get money after the thing is made, which biases towards people who already have money and security to live on while they work. It’s also only as valuable as your ability to enforce it, or pay for lawyers. And it requires a whole police infrastructure to try to catch and punish people using it inappropriately. Models where people are paid to create, or given rewards after the fact for particularly notable work, pay people when they need to be paid, and don’t require a whole police state to function. They also reward people for doing the actual thing. The best way to make money as a novelist at the moment isn’t to write a good or even popular novel; it’s to write a novel someone wants to turn into a movie. While it’s true that copyright and patents work a bit differently, they share most of the same basic problems.

What’s really pernicious about IP, though, is that it can be signed away. If you have a white collar job, there’s a good chance your contract states that the company you work for simply owns any IP you create. Stan Lee, despite creating comic book characters worth billions, was not rich, because he never owned the characters; the company he worked for did.9 A struggling musician who wants to sign to a label will almost certainly be offered a deal where they sign over the rights to their own songs, and the label knows they really have no choice but to sign. Individuals owning their own IP is by far the minority case. The overwhelming majority of IP is owned by companies and the powerful, not the person or people who had the actual idea or did the work.

As bad as this is in the creative industries, it’s much worse in science and technology, where the relevant flavour is patents, and patents have an extra nasty rule that copyright doesn’t. A patent goes to whoever does something first. So if a lab in India starts working on something, and a little while later a US lab starts working on the same thing, it’s likely the US lab will get there first because it’s bigger and has more money. It then gets the IP, and the Indian lab gets nothing. You can’t independently develop the same thing as anyone else; whoever gets there first gets everything. So bigger and richer countries just need to get to things a little bit ahead, and then nobody can catch up. Instead, everyone else has to, by law, not invent the thing, and pay you for it instead. As well as entrenching inequality, this leads to insane injustices, like countries not being able to make their own vaccines during the pandemic, or South Africa not being allowed to produce antiretrovirals during the AIDS crisis.10

So the fact that LLM training isn’t considered to violate copyright is actually a good thing. If it was, what would happen is that companies like Amazon would add to their terms of service that if you sell a book through them, you consent to them selling the contents as training data. A couple of other companies would do similar things, and then you’d have a block of about five companies owning the vast majority of written IP, which you’d need to pay a bunch of money to in order to train your model. And the models that don’t have billions to pay would be declared illegal. Companies are very, very used to controlling and monopolising things through IP (it’s kind of why IP was created), and their inability to do so here is actually a huge problem for them.

Even the slop problem I consider to be not a problem with generative models, but with algorithmic media feeds. You’re not seeing slop on network television; you’re seeing it in places where human taste, or any sort of standard for quality, was replaced with a tasteless maths algorithm that, since it can’t assess quality, just goes by likes, clicks, scroll speed, and other things that prioritise clickbait, ragebait, and producing tonnes of crap instead of fewer good things. So maybe I’m crazy, but maybe we just create and consume media curated by a person, or at least put together with some concern for quality?

From an environmental standpoint, data centres are actually unusually environmentally friendly as far as industry goes. They don’t produce emissions or pollution, and they can be placed anywhere, so you can put them next to a green energy source and not need all the transmission lines and so on. Because they can go anywhere, you can also do things like have a data centre on one side of the world online when the sun’s up there powering solar panels, then switch it off and use one on the opposite side of the world as the sun moves round.11 That’s less cost-effective, but environmentally speaking, something which fits into green energy patterns well but less cost-effectively is pretty good compared to other industries. And given that electricity is dropping to free at midday in many places now due to solar abundance, it’s not as cost-ineffective as you’d think.12 On water use, there are lots of ways to cool data centres that don’t consume fresh water: grey water cooling, closed systems, and so on.13 And the water use still isn’t that high on an industrial level. One million queries take about 2000 litres of water;14 by contrast, the average pair of jeans requires about 8,000 litres to produce.15

It reminds me of a case a few years ago when there was a drought in California and all the water was pre-sold to Nestlé.16 Yes, capitalists are going to capitalist. They’ll always do things in the cheapest way irrespective of the damage, no matter the industry, but that’s where laws and regulations should come in.

Now, I think what some people might say here is that I’m being naive. That sure, in a decent, sensibly run society this tech would be used to do the boring and repetitive tasks at work, lead to lower working hours and higher wages, and benefit independent creators and the most disadvantaged people, and so on. But we don’t live in that world, and in this world nobody’s going to pass an environmental regulation, or get fewer hours, or anything else. Everything is just going to be controlled by the awful people and deployed the way they want, and that’s just how stuff is.

And there’s definitely some truth to that. But even taking the real hard-line, ruthless pragmatism approach, it still seems a lot less naive than the more popular anti-AI line. Passing environmental regulations on companies happens all the time. Technological developments leading to lower working hours or increased wages have happened in the past. Open-weight models being free to download and use is already a thing. This stuff doesn’t seem that crazy, and when I look at the alternatives, they seem to be:

  • Try to block the building of data centres.
  • Consumer boycott things which use the tech.
  • Harass people online for using it.

As mentioned, data centres can go anywhere, so at most a block will cause a bit of expense as they relocate while most corporations aren’t vulnerable to consumer boycotts. Some trading company spending millions a month on tokens to analyse every social media feed to make slightly better stock market bets isn’t a consumer-facing company, and most big companies aren’t. Even for those that are, if it’s a large company there are so many layers that there’s next to no accountability. Video game companies are consumer-facing companies, but big video game companies do crazy evil shit all the time, because it doesn’t matter much if you have an advertising budget. Only small solo or indie studios are actually vulnerable to that sort of thing. And harassing people is pretty unpleasant generally, and since there’s no way to know for sure who’s using AI, it leads to a lot of AI-vestigations, which is also really crazy and unpleasant.

For instance, you could easily see a world in which indie RPGs started using computer-generated voices. They don’t have the budget for voice actors, and while obviously not as good there is a section of players for whom it’s much better than no voices at all. This creates upward pressure on AAA studios, since why are people paying twice as much for your game now? The natural response would have to be for AAA studios to hire more and better voice talent, to lean on that quality gap as their market advantage. This would obviously be better for voice actors, and for indie game devs.

Or, you could easily see a world in which no indie studio would ever integrate computer voices, because the review bombing campaigns and other backlash would do far more damage than any benefit it would secure. Meanwhile, AAA studios start laying off voice actors and replacing them with digital voices to save a bit of money, because their size and advertising budgets make them effectively immune to these kinds of backlashes.

As an example of the latter world, Expedition 33 had their indie game award revoked because they admitted to using gen-AI to produce placeholder assets.17 In other words, for using a computer to do some boring, low-importance work so that the artists who would otherwise have had to do it could work on interesting, important final game art instead.

Is that supposed to be good? Is this actually the pragmatic approach that’s supposed to help things?

It’s really hard for me not to see the strident anti-AI crowd as being functionally aligned with the strident pro-AI bros. They both seem to be pushing for a future in which the technology is just for the rich and for exploitative cost-cutting, and workers must never be allowed shorter working weeks or less boring jobs. It also doesn’t help that the internet feels like it’s drowning in both anti-AI and pro-AI grifters, each making extreme and simplistic claims for clicks and popularity.

For me personally, there are really two reasons why I lean on the tech, even though it’s probably objectively stupid given the amount of animosity it generates.

The first is just time and money. I build these games on my own, in my free time, for free. If I was a billionaire I’d absolutely hire a bunch of coders and artists and really go hog wild, but since I can’t do that and have to do everything myself, I use absolutely anything and everything that will be at all helpful and allow me to make the coolest stuff I can with the limited resources I have. There’s plenty of stuff where it absolutely would be better if I hand-wrote it all myself, but then the game would never come out. So instead I triage: the things that really need me and the extra time, and the stuff where good enough is acceptable.

The second is that the technology is unusually human. A lot of this is framing. If you compare an LLM to a person, then yeah, of course the LLM is way less human-y than the person. But if you compare an LLM to a computerised mathematical function, it’s way more human. I don’t build LLM tech into the game to replace people; there are no LLM players or player surrogates. I use it entirely to make systems that otherwise wouldn’t exist, or would be clunky and deterministic. Whether you’re going on a Jianghu adventure or going into a delve, you’ll have the most fun doing it alongside another PC, with the computer presenting obstacles or other challenges that you’re overcoming, and someone else seeing you do that, and interacting with you about it. But in a delve, the only way to overcome an obstacle is to roll dice, or hit a thing, or solve a puzzle. In an adventure, you can overcome obstacles by being creative, and make narrative character choices that change how things play out. They’re both best as communal experiences, but one is a lot richer, more creative, and more human than the other.

I really want to build worlds and games that are more than just chatting in a tavern and then going and doing fight-maths. Where emotions, relationships, creativity, and all these other things are also part of it. Where the computer system that runs the world can actually, even if in a limited or imperfect way, recognise that you just did something clever or expressed something meaningful, and react to that. It’s maybe impossible and certainly challenging, but I think it’s worth striving for all the same.

Further Reading

References

  1. Wikipedia: ChatGPT; History.com: ChatGPT Released by OpenAI. ↩
  2. Kaplan et al., "Scaling Laws for Neural Language Models" (arXiv:2001.08361); Hoffmann et al., "Training Compute-Optimal Large Language Models" (arXiv:2203.15556). ↩
  3. Hacker News: tokenization and the strawberry problem; secwest.net: Strawberry. ↩
  4. Forbes, "Google AI Glue To Pizza: Viral Blunders"; Futurism, "Google Suggested Adding Glue to Pizza". ↩
  5. Bloomberg, "OpenAI Accuses DeepSeek of Distilling US Models"; Rest of World, "OpenAI-DeepSeek distillation dispute"; Berkeley Law, "The Innovation Dilemma: AI Distillation in OpenAI v. DeepSeek". ↩
  6. Dean W. Ball, Head of Strategic Futures at OpenAI, quoted in TechCrunch, "Kimi: Threat or menace?"; The Decoder, "China's Kimi K3 is forcing Western AI labs to question their compute advantage". ↩
  7. De Simone et al., "From Chalkboards to Chatbots: Evaluating the Impact of Generative AI on Learning Outcomes in Nigeria" (World Bank Policy Research Working Paper 11125); World Bank Open Knowledge Repository; World Bank Blogs: From chalkboards to chatbots in Nigeria. ↩
  8. Thomson Reuters Institute, "Chatbots for justice"; American Bar Association, "Access to justice: How AI-powered software can bridge the gap"; Bloomberg Law
  9. Money.com, "Stan Lee Net Worth and the Marvel Universe"; Celebrity Net Worth: Stan Lee. ↩
  10. WTO, "TRIPS Council discusses COVID-19 vaccine decision"; Congressional Research Service, "The WTO TRIPS Agreement and COVID-19" (R47231); South African History Online, "Pharmaceutical companies drop lawsuit against South Africa"; Barnard (2002), "In the High Court of South Africa, Case No. 4138/98" (PubMed); Harvard Berkman Klein Center, "The South Africa AIDS Controversy". ↩
  11. Google, "We now do more computing where there's cleaner energy"; Radovanovic et al., "Carbon-Aware Computing for Datacenters" (arXiv:2106.11750); Electricity Maps, "Google data centers shift their computations to cleaner times and locations". ↩
  12. REsurety, "Negative Prices in CAISO"; California Public Advocates Office, "Net Energy Metering history fact sheet"; Futurism, "California Solar Electricity Prices Go Negative". ↩
  13. Li, Yang, Islam and Ren, "Making AI Less Thirsty" (Communications of the ACM); arXiv:2304.03271. ↩
  14. Sam Altman, "The Gentle Singularity"; Jegham et al., "How Hungry is AI? Benchmarking Energy, Water, and Carbon Footprint of LLM Inference" (arXiv:2505.09598); Li, Yang, Islam and Ren, "Making AI Less Thirsty" (Communications of the ACM). ↩
  15. Oxfam GB, "Buying one pair of second-hand jeans and a t-shirt could save the equivalent of 20,000 bottles of water"; UN News, "UN launches drive to highlight environmental cost of staying fashionable". ↩
  16. NPR, "Nestle Offered Permit To Continue Taking Water From California Stream"; EcoWatch, "Nestle Water San Bernardino"; Story of Stuff, "Unbottle Water: San Bernardino". ↩
  17. AV Club, "Clair Obscur has its Indie Game Awards rescinded over gen-AI"; Game Rant, "Clair Obscur: Expedition 33 Gen-AI and the Indie Game Awards 2025". ↩