Google DeepMind’s AI Conquers ‘Goat Simulator 3’: A New Frontier in Gaming

Google DeepMind’s SIMA: The AI Goat Herder

In the whimsical world of Goat Simulator 3, where domesticated goats embark on improbable quests, sometimes even donning jetpacks, Google DeepMind has unveiled a groundbreaking AI program called SIMA (Scalable Instructable Multiworld Agent). This innovative software is capable of learning to complete tasks across various games, adapting its knowledge from one game to another.

“SIMA is greater than the sum of its parts. It is able to take advantage of the shared concepts in the game, to learn better skills and to learn to be better at carrying out instructions.”

Frederic Besse, a research engineer at Google DeepMind, emphasizes the significance of SIMA’s adaptability. As tech giants like Google and OpenAI vie for dominance in the generative AI arena, expanding the scope of data that algorithms can learn from presents a path to enhanced capabilities.

Beyond Chatbots: AI Takes Control

DeepMind’s latest foray into video games suggests that AI systems like OpenAI’s ChatGPT and Google’s Gemini could soon transcend mere chatting and content generation, potentially taking control of computers and executing complex commands. This ambitious goal is being pursued by both independent AI enthusiasts and major companies, including Google DeepMind, whose CEO, deepmind.com/about/leadership”>Demis Hassabis, recently expressed the company’s substantial investment in this direction.

Linxi “Jim” Fan, a senior research scientist at Nvidia, commends the Google DeepMind work, drawing parallels to earlier projects like OpenAI’s World of Bits and the 2022 effort called VPT, which involved agents learning tool use in Minecraft. Fan notes that while the number of environments is still limited, SIMA demonstrates stronger generalization to new games and is on the right track.

SIMA: Putting a New Spin on Game-Playing Agents

SIMA represents DeepMind’s fresh approach to game-playing agents, a technology the company has pioneered in the past. From mastering classic Atari video games using reinforcement learning in 2013 to developing AlphaGo, which defeated a world champion in the ancient board game Go in 2016, DeepMind has consistently pushed the boundaries of AI in gaming.

For the SIMA project, the Google DeepMind team collaborated with several game studios to gather keyboard and mouse data from human players across 10 different games with 3D environments, such as No Man’s Sky, Teardown, Hydroneer, and Satisfactory. The collected data was then annotated with descriptive labels to associate the clicks and taps with specific actions, like a goat searching for its jetpack or a human character digging for gold.

The labeled data was fed into a language model, similar to those powering modern chatbots, enabling SIMA to execute actions based on typed commands. Human evaluations of SIMA’s performance within different games were used to fine-tune its capabilities further.

After extensive training, SIMA can perform over 600 actions, ranging from exploration and combat to tool use, in response to hundreds of human-issued commands like “Turn left,” “Go to the spaceship,” “Go through the gate,” or “Chop down a tree.” The researchers deliberately avoided games featuring violent actions to align with Google’s ethical guidelines on AI.

The Future of AI in Gaming and Beyond

While SIMA remains a research project, Tim Harley, another member of the Google DeepMind team, envisions a future where agents like SIMA could play alongside humans in games. However, for AI agents to perform useful office or everyday admin tasks, they must become more reliable. Harley and Besse at DeepMind are currently working on techniques to enhance the reliability of these agents.

As AI continues to evolve and expand its capabilities, the potential applications extend far beyond the realm of gaming. The advancements made by projects like SIMA pave the way for AI to become an increasingly integral part of our daily lives, assisting us in various tasks and opening up new possibilities for human-AI collaboration.