Getting played by AI

The monthly online meetings have featured two recent games by Jim that proved to be ‘engined’ by AI/LLMs. Both games were set in Jim’s Universe campaign. The first a ‘murder mystery’ with the players forming a team of specialists investigating a curious and deadly anomaly; the second a committee game with the player teams representing factions in an urban civil unrest scenario.

It is interesting to talk about AI in gaming, but looking back, the post game discussions almost exclusively focussed on the GM/control/umpire side of the game, not on the player experience. And so I will try to focus on that part, this being an offside report.

So how was it, played in an LLM environment?

Apparently not all that different from playing a human GM, as the first game we didn’t know (although some smarter people than me might have suspected it was an experiment) and only found out afterwards. It does help that there was a human intermediary. Of course, hiding the fact that it was AI was much easier online than face to face.

In both games the players were asked to enter 2 or 3 actions, in the first game as a line in discord, in the second in an online sheet. In the first game there was a limit in the number of characters we could use. The responses came back through the same platforms.

While it is useful for the LLM to keep the number of prompts limited, it restricts the ability of a player to explore the problem before her or him. I would often have a small knowledge question that a GM could have cleared up quickly, which would affect the choice of my next step. Now it felt that it slowed down my actions during the game and limited their effectives having to go through very basic things.

I’m not sure how it would have affected my experience of the game if I had known beforehand that the game was powered by an LLM. I think I would have thought harder about wording the instructions, and would have tried to slip in bonus actions to get the most out of them. And future games might turn around players only trying to optimise their prompts, rather than to roleplay. This may not be all that different from players flooding their human GM with context to get the same result in a non-LLM game. I don’t play these games to win, but it affects my perception of ‘getting a fair chance’.

I’m not sure the game itself needs a human ‘driver’ with the LLM being the ‘engine’ apart from guiding the players through the game and copy pasting the player actions into the LLM and then copy pasting the results back. But I personally want a human as GM. I don’t like playing online games because I play games for the interaction with humans. Preferable people I like.

So while I enjoyed the games, I don’t look forward to always gaming/RPGing like this.

Finally a few thoughts on the onside part of the game

The post game chat brought up that there are certainly limits to what LLMs can do gamewise. Eg they don’t really understand maps. But it works fine for RPGs or similar investigator/low level games that are light on mechanics and heavy on narrative. So LLMs are limited in the kind of games they be used for.

Jim noted (I think in both cases, but certainly the last) that it took the LLM 3 or 4 turns to ‘get into it’ and adding more narrative and flavour text to the responses. This is something that a GM/umpire can tackle by doing a warming up session, in which he plays a few turns with the LLM before the game starts.

Although Jim said that running a game with AI is much less work and stress than running it without, which is great of course, there is a huge investment to get into that position. If I recall correctly, Jim intimated that it had taken him a year of feeding the LLM with the game background and training it. This inherently includes Jim investing a lot of time in getting to know how to work with LLMs, like writing good prompts and training.

I guess that’s a useful skill to invest time in, but I can also see that as an effective block on most people trying to do this in their free time with the aim of running an RPG campaign for friends. Not everybody has been running a gaming universe for over 30 years (partly online of course, because all the verbal input is lost from the point of view of the LLM). Could a similar effect be achieved with a smaller ‘knowledge base’?

It might be considered worthwhile to provide a similar environment for Gms by an RPG company with a broad following. The LLM might feed lore to the players in a natural way and provide chance encounters with NPCs that a GM would need more time to prepare. But is it less time/money intensive than creating a fully automated game? And what would players enjoy more?