“The ChatGPT moment for robotics is coming,” Nvidia CEO Jensen Huang said at CES, unveiling Cosmos, a platform that aims to democratize physical artificial intelligence.
Cosmos includes a set of World Foundation Models (WFMs), a family of models capable of generating realistic, physics-aware videos from a combination of inputs, such as text, images, and videos, as well as data from sensors or robot movements. These 3D videos allow AI models to be trained without using real data. WFMs, openly licensed via Nvidia’s API, NGC catalogs, GitHub and Hugging Face, are designed to be customized according to the specific needs of developers.
Nvidia explains that WFMs can not only generate detailed videos from prompts or images, but also predict the evolution of the scene thus created. “WFMs are essential for developers of physical artificial intelligence,” says Ming-Yu Liu, vice president of research at Nvidia. They can imagine many different environments and simulate the future, which allows us to make better decisions based on those simulations.”
Creating models of real environments requires enormous amounts of data, which is difficult and expensive to collect. WFMs make it possible to generate synthetic data, offering according to Nvidia a rich and varied set to improve the model training process. Developers can simulate and test their systems in a controlled setting, without the risks associated with real-world testing.
-Applications in industry
WFMs promise to transform workflows and AI development across various industries. In the automobile industry, for example, an autonomous car can be tested in different weather conditions or simulated traffic scenarios. In robotics, WFMs can be used to simulate and verify the behavior of robotic systems in various environments. These simulations aim to ensure that robots perform their tasks safely and efficiently before deployment.
Another recently unveiled platform, Genesis, aims to revolutionize robot training in virtual environments. While Cosmos benefits from NVIDIA’s extensive ecosystem and hardware optimization, Genesis stands out with its universal physics engine and framework developed specifically for Python.
We also remember that Nvidia had been singled out for the way it trained its AI models. The provider would have used a huge quantity of YouTube and Netflix videos to do this.
