Runway and Niantic Advance AI World Models for Real-Time Video and Virtual Environments

Runway, a video-generation startup that works with Hollywood studios, including Lionsgate, recently launched a product that uses world models to create game environments, personalized stories, and characters generated in real time.

“Traditional video footage is a brute-force approach to generating pixels: you try to cram motion into a few frames to create the illusion of movement, but the model doesn’t actually understand or analyze what’s happening in this scene.”

Previous video-generation systems had physics that differed from the real world; general-purpose world models aim to fix these discrepancies and provide more plausible dynamic interactions in virtual environments.

To build such models, companies need vast amounts of data about the world – from spatial coordinates to surface details and the movements of objects, which becomes the foundation for training and improving world models.

Niantic, a San Francisco-based company, collects data from 10 million locations through its games, including Pokémon Go, which has about 30 million monthly users interacting with a global map.

Niantic has been working on Pokémon Go for nine years, and even after the game was sold to American Scopely in June, its players continue to provide anonymized data through scanning public landmarks to help build a world model.

“We have a good start on this problem.”

Both industry giants – Niantic and Nvidia – are working to close gaps so that their world models can generate or predict environments. Nvidia’s Omniverse platform supports these efforts, creating and running simulations for robotics and engineering applications, based on real data from video games and simulations.

Chief Executive Officer of Nvidia Jensen Huang stated that the next major growth phase for the company is tied to “physical AI,” and these new models could radically transform the robotics industry.

“The next big growth phase is tied to ‘physical AI’ – these new models will revolutionize the field of robotics.”

Some scientists and leaders, including Yann LeCun of Meta, believe that the vision of a new generation of AI systems with human-level intelligence may require about ten years.

“This prospect of a new generation of AI systems that give machines human-level intelligence may take approximately 10 years.”

Despite ongoing discussions about terminology, the potential of world models remains vast: they open up opportunities to serve other industries and enhance the knowledge that computers have already demonstrated in handling data and knowledge.

“World models open up opportunities to serve other industries and enhance what computers have done for working with knowledge.”

Growing interest in world models underscores the shift in investment and scientific focus from discrete language models to all-encompassing systems that combine data from diverse sources for rapid adaptation of business and science to real needs.