top of page
Simeon Spencer

LLMs Playing Pokemon Could Be the First Step to Human-level AI?



If I told you large language models and AIs could eventually reach human-level performance or achieve artificial general intelligence by playing Pokémon, would you believe me? Probably not, at least until now.


Recently, AI researchers at the Georgia Institute of Technology have developed PokéLLMon, the world's first large language model (LLM) agent that can fight Pokémon battles against humans on Pokemon Showdown, an online simulator for all versions of Pokemon games. But here’s the twist: PokéLLMon, despite its advanced AI pedigree, isn't wiping the floor with its human opponents. It's not the StockFish of the Pokémon world, delivering checkmate after checkmate in chess. Instead, it's holding its own, with a win rate that hovers around the halfway mark - 49% in general competitions and a slightly better showing at 56% in matches where it's specifically invited to compete. And this begs the question: How does an AI, seemingly middling in its Pokémon prowess, edge us closer to the holy grail of AI, artificial general intelligence?


Games are the Best Testing Ground to Develop AI to Resemble Human Behavior


Let's take a moment to ponder the game of chess. You start with a modest army of 16 pieces with 6 unique movement patterns. Initially, you're faced with 20 potential moves. Make those moves, and suddenly, you're staring down the barrel of 400 different game states. It sounds deceptively simple, yet the rabbit hole of chess's complexity goes deep, with the number of possible games in a standard 40-move matchup spiraling into the realm of 10^120. Despite this mind-boggling complexity, our AIs today have mastered chess to a startling degree. Take StockFish, for example, the champion of the AI chess world with an Elo score that towers at 3634 as of February 2024, leaving the top human grandmaster, Magnus Carlsen, and his 2830 rating, in the digital dust.


But here's where the plot thickens: crank up the variables, and even the sharpest AIs start to fumble. To mimic the richness of human experience – encompassing everything from games to work, to the intricacies of human relationships – an AI would need to juggle an astronomical number of parameters. Right now, that's a bridge too far for our digital companions.


So, where do we go from here? The answer lies in finding a sweet spot – a challenge that's a notch above chess but doesn't quite dive into the complexity of real-life human existence. Enter the arena of video games, a proving ground where AI can stretch its legs in environments that demand a blend of intuition and strategy, qualities we often think of as uniquely human. Pokémon, despite its cute façade, requires more strategy than you may have expected.


Pokemon is More Complex Than You Think



At first glance, the world of Pokémon might seem like a vibrant splash of colors, populated with creatures that are more at home in a child's imagination than in a strategic battleground. But beneath its cartoonish cover lies a complex and deeply engaging universe that has captivated a global audience, nurturing not only a vast community but a fiercely competitive gaming scene. This competitive environment serves as the perfect testing ground for PokéLLMon, allowing it to measure its mettle against human opponents with precision on leaderboards.


For the few uninitiated, the premise of Pokémon battles might seem straightforward: you step into the arena with a squad of six Pokémon, holding four abilities each. The game unfolds in turns, where you can unleash one of these abilities or swap out your current combatant for another teammate. The objective? Outmaneuver your opponent to knock out their six Pokémon.

 


However, the simplicity of this setup belies the strategic depth that the game demands. Crafting a team isn't just about picking your favorite monsters; it involves a meticulous selection process where each Pokémon's abilities, the types they belong to, and their compatibility with the rest of the team are all crucial considerations. Pokemon has evolved from just your 151 Pokemon in the 1st generation to 1,025 Pokemon now as of the 9th generation, a whole different game. Each of these 1,025 Pokemons are spread across 18 different types (e.g. fire, water, electric, etc.) each with its own set of strengths and weaknesses, and these are just the basics.


Beyond just picking a potent lineup, staying competitive in Pokémon requires keeping abreast of the ever-shifting meta-game, where certain Pokémon rise to prominence with each game update. Crafting a balanced team capable of countering popular strategies is just the beginning. Players must also select the most effective moves from a dizzying array of options, tailor their Pokémon's stats for specific tactical roles, choose the right items for them to hold, and even consider their natures—all factors that can tip the scales in a closely fought battle.


This level of strategic depth means that Pokémon offers something for everyone: it's accessible enough for children to enjoy, yet it boasts layers of complexity that can satisfy the most hardcore competitive players.



Given this backdrop, PokéLLMon's performance as an average player is not something to be scoffed at. Unlike StockFish, which operates within the relatively static confines of chess, PokéLLMon faces a far more dynamic challenge. It must navigate a labyrinthine web of over a thousand Pokémon, hundreds of moves, and an intricate system of types and counters, not to mention predicting the strategies of human opponents. Currently limited to battling with a random selection of Pokémon, PokéLLMon hasn't yet begun to explore the full depth of team-building strategy. But its foray into this vibrant and complex world marks an important step in the evolution of AI, showcasing the potential for these systems to engage with and adapt to environments of staggering complexity.


Pushing Towards Human-level AI One Game at A Time?


So, we've seen AI stretch its legs beyond the chessboard, stepping into the arena of more nuanced and tactical games like Pokémon. It's a leap forward, no doubt, but we're still a long way from creating an AI that can seamlessly blend into the intricacies of everyday human life. However, Pokémon might just be the opening act in a series of increasingly complex virtual sandboxes where AI can hone its skills.


The next frontier? Entering the vast, unpredictable worlds of massively multiplayer online role-playing games (MMORPGs) like World of Warcraft or Runescape with millions of players. These digital realms introduce AI to a whole new set of challenges: collaborating with human teammates, navigating the social dynamics of trade and teamwork, and mastering game mechanics that make Pokémon's complexity look tame in comparison.


Imagine an AI that doesn't just participate in these online worlds but thrives in them, indistinguishable from human players. It would need to understand and predict a plethora of human behaviors and strategies, adapt to unforeseen circumstances, and make decisions that affect not just itself but its team. Achieving such a level of sophistication in AI would mark a significant milestone on the path to artificial general intelligence.


Games, with their controlled environments and complex interactions, offer a clear and structured way to progress toward this goal. They provide a space where AI can experiment, learn, and evolve in conditions that mimic the real world's unpredictability but within the safety of coded rules and objectives. As we push the boundaries of what AI can do within these virtual playgrounds, we edge closer to the day when artificial intelligence can navigate the full spectrum of human experiences and challenges.

Comments


Want to Know When We Post?

LLMs Playing Pokemon Could Be the First Step to Human-level AI?

If I told you large language models and AIs could eventually reach human-level performance or achieve artificial general intelligence by playing Pokémon, would you believe me? Probably not, at least until now. Recently, AI researchers at the Georgia Institute of Technology have developed PokéLLMon, the world's first large language model (LLM) agent that can fight Pokémon battles against humans on ....

blurred text.png

Already Accessed Free Article !

bottom of page