NVIDIA lance Cosmos 3 Edge pour robots et agents vision

Q: Quels sont les avantages de Cosmos 3 Edge pour les systèmes physiques ?

Cosmos 3 Edge permet aux robots et agents vision de comprendre et d'agir en temps réel dans des environnements complexes, améliorant ainsi l'efficacité et la précision des opérations.

Back to Articles The real world is vast and to operate in it physical AI systems need to understand how a scene is changing, anticipate what could happen next, and determine how an action will affect the world.Machines operate at the edge in factories, warehouses, hospitals and beyond. To understand and act in these environments, they need models that deliver data center–level performance on memory-constrained systems. Today we are releasing NVIDIA Cosmos 3 Edge on Hugging Face’s Cosmos 3 repository. It’s a 4-billion-parameter open world model that helps robots and vision AI agents understand their surroundings, reason in real time and generate robot actions on edge devices. Download the model: https://huggingface.co/nvidia/Cosmos3-Edge The model delivers memory-efficient, high-throughput inference across NVIDIA edge computers including NVIDIA RTX PRO GPUs, NVIDIA DGX, NVIDIA GeForce RTX™ GPUs, NVIDIA Jetson including the newly announced Jetson T2000 and T3000 modules. Designed as a compact open model that can serve as a small vision language model (VLM) with best-in-class throughput and accuracy with real-time inference. As a post-trained world action model (WAM), Cosmos 3 Edge operates at robot-control resolution (640×360 observations), delivering real-time reasoning and generating 32 actions per inference on NVIDIA Jetson Thor - while achieving real-time control at 15 Hz. Among similar size (4B parameters) models, Cosmos 3 Edge ranks #1 on VANTAGE-Bench for vision analytics and state-of-the-art for robot policy learning, setting the state of the art for smart infrastructure and robotics. What Is World Modeling? A world model learns how an environment changes over time. It represents objects, motion, spatial relationships, and the effects of actions. Consider a robot reaching for an object. Recognizing the object is only the first step. The robot must also understand where the object is, how its gripper is moving, what may happen when contact occurs, and which action is most likely to complete the task successfully. A world model helps the robot reason about these relationships. It can predict the visual result of an action, infer the action that caused a change, or generate an action to produce a desired outcome. Cosmos 3 Edge brings these capabilities together in one model, on-device. Its shared representation enables physical AI systems to understand the current world state, simulate possible futures, and connect those futures to actions. Two Transformer Towers, One Shared Representation Cosmos 3 combines two transformer towers: Autoregressive tower: Processes vision and text tokens for understanding and reasoning. Diffusion tower: Processes vision, audio, and action tokens for prediction, generation, and neural simulation. The two towers maintain separate normalization layers and multilayer perceptrons. They share multimodal attention layers, which align information across language, video, audio, and action. This design allows the model to reason about a scene before generating an output. Depending on the task, Cosmos 3 can produce reasoning tokens from the autoregressive tower or denoised video and action tokens from the diffusion tower. The attention pattern is also adapted to each type of information. Language uses causal attention, where each token attends to the tokens that came before it. Diffusion tokens attend more broadly to the available context, supporting coherent prediction and generation. Together, these components give the model a common representation of what is happening, what may happen next, and how an action could influence the outcome. A Common Representation for Action Physical systems describe actions differently. A vehicle may represent an action through ego pose and movement. A camera uses camera motion. A robot arm uses the pose of its end effector, while a hand or gripper also needs to represent grasp state. Cosmos 3 maps these different embodiments into a common action representation. Actions are encoded as compact geometric vectors that capture: Translation Rotation Manipulation state This creates a direct connection between control and the visual structure of the world. The model can associate changes in pixels with physical motion, spatial relationships, and control inputs. As a result, generated video becomes more than a visual prediction. It can represent how the world is expected to change in response to an action. This gives developers training data grounded in motion, control, and cause and effect. Policy Mode As a policy, Cosmos 3 predicts an action together with its expected visual consequence. The model can generate what the system should do and simulate what is likely to happen next. This connects world modeling directly to robot policy training and evaluation. Current state in, action and visual consequence out. These modes enable one model to learn cause, effect, and policy. Action can

Accueil

Outils

Annuaire

Apprendre