World models' are the future of AI, but how do they work?

In a few short years, artificial intelligence (AI) has transformed how we interact with computers and threatens to upend wide swathes of the job market. But large language models (LLMs) still struggle with the messy realities of the physical world. Researchers are betting that a new type of AI, called "world models," could fix that.

At a fundamental level, world models do exactly what the name suggests: They build a mathematical model of the world that can then be used to make predictions about how it will change in response to certain actions or changing conditions. The "world" in this context doesn't necessarily mean the entire physical reality. Instead, it refers to the environment the model operates within, which could be anything from a warehouse to a video game.

Exactly what counts as a world model and how best to build such a model remain topics of considerable debate among AI researchers. But world models would represent a significant advance over LLMs, which, despite their impressive capabilities, simply predict the most likely next word in a sequence.

The hope is that by developing a richer understanding of complex environments, world models could allow AI to finally break out of the chat interface, with potentially game-changing applications in areas like robotics, autonomous driving and scientific discovery.

"The idea has strong connections with the intuitive models in our human minds," said Yunzhu Li , an assistant professor of computer science at Columbia University. "We can imagine how the environment is going to change, how an object is going to move when you apply a specific action. And we basically want to also build this kind of model for any robots or any virtual agent so they can imagine the effects of their actions."

Building world models in your mind

While world models are the latest buzzword in Silicon Valley, the concept has deep roots. It first came to prominence in the 1950s, Manling Li , an assistant professor of computer science at Northwestern University, told Live Science. It arose when cognitive scientists attempted to describe the mental models people used to simulate their environments in their heads.

The concept is also deeply connected to, and often inspired by, control theory, Manling Li said. This is a branch of applied mathematics used to create models of physical systems so they can be predictably controlled. It powers everything from thermostats to aircraft autopilot systems.

However, the term "world models" today refers primarily to neural networks that learn models of their environment by training on data. The modern incarnation of the idea can be traced to a 2018 study titled "World Models ," by scientist David Ha and deep learning pioneer Jürgen Schmidhuber . Early models from Google, like PlaNet and Dreamer , were among the first to solve tasks by first making predictions about the outcome of different actions.

While the idea behind a world model is fairly intuitive, a more precise definition is any system capable of "action-conditioned future prediction," Yunzhu Li said. This essentially means the model can predict how a particular action will change the state of the world around it.

Making those predictions, Manling Li said, consists of two key tasks: state estimation and state transition. State estimation refers to the ability to perceive the current state of the environment and encode it into a format that the model can compute, while state transition means the ability to predict how a particular action will cause the environment to evolve.

The data that powers new realities

Deep-learning-based world models learn to do both tasks by training on vast quantities of data. But exactly what kind of data and how that data should be encoded and processed are design choices, with different groups taking a variety of approaches, Yunzhu Li said.

World models are trained primarily on video data, although they can also be trained on 3D data captured by light detection and ranging (lidar) or other depth sensors, audio data and even text that explains the relationships between elements in the environment. Crucially, Yunzhu Li said, this has to be paired with action data — things like robot joint angles, movement readings from an inertial sensor, or event text labels describing what action was taken.

Robots, including humanoids, will be increasingly reliant on strong world models in order to interact with the physical realm. (Image credit: China News Service via Getty Images)

Typically, this data is arranged into sequences of state-action pairs — essentially, recordings of what the world looked like and what action was applied at each step. The AI then uses this data to learn a statistical model of what impacts different actions have on its environment, which can be used to make predictions.

This data can be processed in different ways, Yunzhu Li said. One of the most popular approaches is to operate directly on raw pixel data, which represents the state of the world as a series of images and predicts how actions will change them. Another is to use the data to learn 3D geometric representations of the world that more explicitly encode spatial and physical relationships among objects in a scene.

The power of math-based abstractions

More recently, however, there's been growing interest in approaches that operate on a more abstract level. When a neural network learns from image data, it creates high-dimensional numerical representations of the real-world elements that make up the visual scene — known as embeddings — that exist in a mathematical space known as the model's "latent space."

In a pixel-based model, these abstract representations are reconstructed into pixels to make predictions about what will happen next. But it's also possible to do those simulations within the latent space by directly predicting the embedding of the environment's next state. This approach has been popularized by computer scientist Yann LeCun , Meta's fo...

Indexed and credited by AIPROPX. Originating outlet: Live Science. Open at source →