Advanced Robotics Applications In Engineering

Exciting updates on Project GR00T! We discover a systematic way to scale up robot data, tackling the most painful pain point in robotics. The idea is simple: human collects demonstration on a real robot, and we multiply that data 1000x or more in simulation. Let’s break it down: 1. We use Apple Vision Pro (yes!!) to give the human operator first person control of the humanoid. Vision Pro parses human hand pose and retargets the motion to the robot hand, all in real time. From the human’s point of view, they are immersed in another body like the Avatar. Teleoperation is slow and time-consuming, but we can afford to collect a small amount of data.  2. We use RoboCasa, a generative simulation framework, to multiply the demonstration data by varying the visual appearance and layout of the environment. In Jensen’s keynote video below, the humanoid is now placing the cup in hundreds of kitchens with a huge diversity of textures, furniture, and object placement. We only have 1 physical kitchen at the GEAR Lab in NVIDIA HQ, but we can conjure up infinite ones in simulation. 3. Finally, we apply MimicGen, a technique to multiply the above data even more by varying the *motion* of the robot. MimicGen generates vast number of new action trajectories based on the original human data, and filters out failed ones (e.g. those that drop the cup) to form a much larger dataset. To sum up, given 1 human trajectory with Vision Pro  -> RoboCasa produces N (varying visuals)  -> MimicGen further augments to NxM (varying motions). This is the way to trade compute for expensive human data by GPU-accelerated simulation. A while ago, I mentioned that teleoperation is fundamentally not scalable, because we are always limited by 24 hrs/robot/day in the world of atoms. Our new GR00T synthetic data pipeline breaks this barrier in the world of bits. Scaling has been so much fun for LLMs, and it's finally our turn to have fun in robotics! We are creating tools to enable everyone in the ecosystem to scale up with us: - RoboCasa: our generative simulation framework (Yuke Zhu). It's fully open-source! Here you go: http://robocasa.ai - MimicGen: our generative action framework (Ajay Mandlekar). The code is open-source for robot arms, but we will have another version for humanoid and 5-finger hands: https://lnkd.in/gsRArQXy - We are building a state-of-the-art Apple Vision Pro -> humanoid robot "Avatar" stack. Xiaolong Wang group’s open-source libraries laid the foundation: https://lnkd.in/gUYye7yt - Watch Jensen's keynote yesterday. He cannot hide his excitement about Project GR00T and robot foundation models! https://lnkd.in/g3hZteCG Finally, GEAR lab is hiring! We want the best roboticists in the world to join us on this moon-landing mission to solve physical AGI: https://lnkd.in/gTancpNK

Build your first robot in simulation! 👾 📌 If you’re self-learning robotics, this is genuinely one of the better repos to save for later. NVIDIA Robotics released a "Getting Started with Isaac Sim" tutorial series covering everything from building your first robot to hardware-in-the-loop deployment. What's inside? → Building Your First Robot Explore the Isaac Sim interface, construct a simple robot model (chassis, wheels, joints), configure physics properties, implement control mechanisms using OmniGraph and ROS 2, integrate sensors (RGB cameras, 2D lidar), and stream sensor data to ROS 2 for real-time visualization in RViz. → Ingesting Robot Assets Import URDF files, prepare simulation environments, add sensors to existing robot models, and access pre-built robots to accelerate development. → Synthetic Data Generation Learn perception models for dynamic robotic tasks, understand synthetic data generation, apply domain randomization with Replicator, generate synthetic datasets, and fine-tune AI perception models with validation. → Software-in-the-Loop (SIL) Build intelligent robots, implement SIL workflows, use OmniGraph for robot control, master Isaac Sim Python scripting, deploy image segmentation with ROS 2 and Isaac ROS, and test with and without simulation. → Hardware-in-the-Loop (HIL) Understand HIL fundamentals, learn NVIDIA Jetson platform, set up the Jetson environment, and deploy Isaac ROS on Jetson hardware. The progression makes sense: start with basics (build a robot), add perception (sensors and data), generate training data (synthetic generation), develop software (SIL), then deploy to hardware (HIL). Each module builds on the previous one. For robotics teams, this is the path to faster iteration. Simulate first, validate in software-in-the-loop, generate synthetic training data at scale, then deploy to hardware with confidence. 🎓 If this helps at least one engineer to become more fluent in the world of robotics, means a lot to me! 🫶🏼 Here's the course (it's free): https://lnkd.in/dRYdkmdi ~~ ♻️ Join the weekly robotics newsletter, and never miss any news → ziegler.substack.com

A few years ago, I learned the hard way that jumping straight into hardware, sensors, motors, and wiring can lead to costly mistakes and late-night headaches. That’s when I discovered the true importance of #simulation in robotics and engineering. During the early phase of my final-year thesis, I spent weeks recreating our school cafeteria with Iman Tokosi in Blender, exporting it as an SDF model and loading it into Gazebo using #ROS2. Suddenly, I could drive a virtual robot through aisles and around tables without the fear of damaging anything real. It was challenging and eye-opening, and it saved me countless hours and resources. Then came the moment that changed everything: integrating #SLAM so the robot could build its own map while moving, and setting up #Nav2 to let it plan and follow paths autonomously. Watching it navigate the environment with precision and independence was a powerful confirmation that the system worked. Now, imagine a world where every structure, product, and system is simulated down to the smallest detail. The result? Reduced costs, faster development, increased reliability, enhanced safety, and stronger adherence to standards. Some may still view simulation as “just for show,” but I’ve experienced firsthand that it’s the foundation of true innovation. Are you leveraging simulation in your next robotics or engineering project? Let’s connect and exchange ideas!

This is the moment simulation becomes more important than prototyping. In our last posts, Pascalis and I showed two things: First, how you can generate a full production and warehouse environment in NVIDIA Omniverse using Claude Code and the USDA data format. Second, how NVIDIA’s new Kimodo model can generate robot motions from simple text prompts. Now we are taking the next step: Transferring robot motion into Omniverse and merging both use cases. Omniverse is not just for static visualizations. It allows dynamic simulation of movements, interactions and behavior with CAD components inside a virtual environment. And this is where it gets interesting for future product development. The vision is clear: If we can model production environments, warehouses, and real operating environments of products, we can simulate mechatronic products in realistic conditions before they physically exist. Environment → Sensor & actuator interaction → Model-in-the-loop simulation. Very similar to how autonomous vehicles are developed today, but applied to all kinds of mechatronic products. The effects are huge: • Less physical prototyping • Earlier insights without building hardware • Faster iteration cycles • Better product decisions earlier in development • Simulation becomes the main development environment Omniverse already shows how granular these simulations can be created today. Not through months of manual modeling, but increasingly through prompts that generate environments, movements and soon maybe even control logic. We are moving from designing products to designing behavior in simulated worlds first. And that will fundamentally change how we develop products. Curious to hear your thoughts! When will simulation become the primary development environment in your industry? Vlad Larichev | Rüdiger Stern | Rick Bouter | Ruben Hetfleisch | Dr.-Ing. Tobias Guggenberger

𝗗𝗶𝗱 𝘆𝗼𝘂 𝗽𝗹𝗮𝗰𝗲 𝗮 "𝗖𝗮𝗻𝗮𝗿𝘆" 𝗶𝗻 𝘆𝗼𝘂𝗿 𝗣𝗿𝗼𝗰𝗲𝘀𝘀 𝗔𝘂𝘁𝗼𝗺𝗮𝘁𝗶𝗼𝗻? The sort of early warning detection system which monitors your automated processes and sings when irregularities occur? Why a 🐤 𝗰𝗮𝗻𝗮𝗿𝘆 you ask? Around 1911, miners started to take canary birds into the coal mines to detect the accumulation of toxic gases. These birds, would even sense the smallest traces and emissions, starting to erratically chirp and with that giving miners early warnings to immediately evacuate the mine. Just as the canaries once did in the mines, 𝗮 𝗱𝗶𝗴𝗶𝘁𝗮𝗹 "𝗰𝗮𝗻𝗮𝗿𝘆" can play a vital role in monitoring the health of your automated workflows signalling potential issues before they escalate and perhaps, cause scaled harm. But how do you implement a digital canary into your workflows in your process automation? 𝗜𝗻𝗰𝗼𝗿𝗽𝗼𝗿𝗮𝘁𝗲 𝗶𝘁 𝗳𝗿𝗼𝗺 𝘀𝘁𝗮𝗿𝘁: into your design by using code, reconciliation reports, and validation rules to establish effective in-process control checks and monitoring mechanisms and visual dashboards to analyse red flags. Find here 5 examples how to get early alerts in your process automation, even if your automation bots don't know how to sing: ▪️𝗣𝗿𝗼𝗰𝗲𝘀𝘀𝗶𝗻𝗴 𝗖𝗵𝗲𝗰𝗸𝘀: Implement automated checks at various stages of the process to ensure accuracy and completeness and volume variations. ▪️𝗜𝗻𝘁𝗲𝗴𝗿𝗮𝘁𝗶𝗼𝗻 𝗘𝗿𝗿𝗼𝗿 𝗩𝗮𝗹𝗶𝗱𝗮𝘁𝗶𝗼𝗻: Monitor integration and break points like API's for errors or failures to maintain seamless data flow across systems. ▪️𝗗𝗮𝘁𝗮 𝗜𝗻𝘁𝗲𝗴𝗿𝗶𝘁𝘆 𝗦𝗰𝗮𝗻𝘀: Validate for duplicate records or inconsistencies to maintain data integrity and remove manual overrides or corrections. ▪️𝗨𝘀𝗮𝗯𝗶𝗹𝗶𝘁𝘆 𝗙𝗲𝗲𝗱𝗯𝗮𝗰𝗸𝘀: Analyse insights from user feedbacks to check on usability issues, frequent issues and detect sentiment drops with NLP / AI. ▪️𝗖𝗼𝗺𝗽𝗹𝗶𝗮𝗻𝗰𝗲 𝗖𝗼𝗰𝗸𝗽𝗶𝘁: Create a centralised dashboard to monitor compliance metrics to detect red flags and and detect deviations from policies. By integrating digital canaries into your process automation strategy, you are not only enhance your ability to detect and respond to issues rapidly but also promote a culture of self-monitoring and continuous improvement. So, did you already place a digital "canary" into your process design and automations? If not, maybe it's time to reconsider adding this early warning system to your automation approach ensuring the health and resilience of your tasks, data & process performance. What early warning systems have worked for you best? #processautomation #intelligentautomation #rpa #processexcellence

A humanoid robot costs $90K to break once. AI lets you break thousands... and learn from every fall. My background is mechanical engineering, robotics, and integration & test. But this field is moving so fast with AI that reading articles wasn't cutting it anymore. I felt out of the loop, so... I recently upgraded my personal setup to support AI training workloads and ran my first experiment: Teaching a bipedal (two-legged) humanoid robot to navigate a custom parkour course using reinforcement learning in NVIDIA Isaac Lab 5.1. But before I share what I learned, let me explain what's actually happening under the hood. A GPU-accelerated AI agent runs thousands of virtual robots in parallel. Each one learns from its own falls and successes simultaneously. The AI develops a "control policy," which is the brain that tells a robot how to move through the physical world. Why does this matter? Because what once required million-dollar labs and months of physical testing can now run on a single AI-capable GPU in hours. Robotics R&D is becoming software-first. Here's what that looked like for this experiment: 76 minutes of CUDA-accelerated training time. 393 million training steps. 4,096 robots learning in parallel on my RTX 5080. So what did I learn so far? Three things stood out to me: 》The setup before you can hit "Run" is a challenge. It took me seven hours to troubleshoot versioning, packages, and dependencies before I could run anything. I forced myself to do it manually because I wanted to understand what's under the hood. YouTube tutorials hit their limit quickly, but thankfully the NVIDIA developer forums saved me. 》The cost case is undeniable. A Unitree H1 costs around $90K. I *virtually* crashed thousands of them. My damage bill? $0. Simulation lets you fail-forward at scale. This gets you to a solid starting point for physical testing, but... 》The Sim-to-Real gap is real. This policy works well in simulation, but I couldn't get a feel for stress points, sensor behavior, or true stability. Failure is not predictable and happens at the edges. The next step would be to transfer this policy to a physical robot, gather real-world data, and continuously aligning the simulation to close that gap. The key thing here is: Testing real hardware is expensive. Simulation in software is cheap. How can you leverage both, intelligently? The benefit isn't limited to cost savings. This workflow also compresses developmental cycles and allows you to field systems faster. Do you think virtual simulation is a game-changer that is here to stay, or a fad? How would you build confidence in a robotic control policy that is trained in a virtual world? #robotics #ai #nvidia #omniverse #isaaclab ~~~~~~~~ Citations: NVIDIA IsaacLab -> https://lnkd.in/ekVMDnDc RSL-RL -> https://lnkd.in/eJye3XTW Unitree H1-> unitree.com/h1/ Note: this is an educational personal project. Opinions are my own, no affiliation or endorsement.

Closing the sim-to-real gap in humanoid robotics requires massive simulation throughput and high-fidelity physics validation. WPP recently detailed their engineering pipeline, showing how they reduced reinforcement learning cycle times for complex humanoid locomotion from 24 hours down to less than 60 minutes. The hardware architecture relies on Google Cloud’s new G4 VMs (powered by NVIDIA RTX PRO 6000 Blackwell GPUs) running NVIDIA Isaac Sim, integrated closely with DeepMind’s MuJoCo physics engine. The mechanics: The team mapped raw human mocap data (over 200 degrees of freedom) down to a constrained 29-DOF OpenUSD digital twin. By leveraging a P2P GPU topology to bypass central processing bottlenecks, the infrastructure executed over 3 billion simulations in under an hour. The virtual environment continuously introduced physical micro-variances—simulated pushes, shifting floor friction, and momentum changes—to train the model against the chaos of the real world. The resulting reinforcement learning model was condensed into a highly efficient ONNX policy and deployed directly to the physical robot. This edge policy processes live IMU and joint telemetry to output immediate, stabilized motor commands. Reaching this scale of simulation volume is the precise engineering mechanism that allows control policies to handle unstructured physical deployment. To support the research, Unitree has open-sourced the underlying RL code on GitHub. Blog post : https://lnkd.in/g4-gWzTP #Robotics #PhysicalAI #ReinforcementLearning #MuJoCo #GoogleCloud #IsaacSim #Engineering

Robots are moving into new kinds of work. Tasks that were once too variable, manual, or uneconomical to automate are now being handled across environments like security, food processing, and field services. This month’s news surfaced signals like these: 🐕 Faraday Future’s FX Aegis quadruped cleared U.S. compliance testing and is moving into formal sales, designed for security, patrol, and companionship use cases that require mobility across complex environments. 🥩 Chef Robotics expanded its platform into meat tray assembly, applying robotic picking systems to one of the more difficult categories in food processing, where products vary in shape, size, and texture. 🧼 Lucid Bots is scaling autonomous exterior cleaning through a robotics-as-a-service model, with close to 1,000 robots deployed across commercial jobs ranging from building washing to pressure cleaning. Why this matters: Robotics is showing up across more industries, not just a few controlled environments. The same advances in perception, manipulation, and autonomy are now being used in very different kinds of work. Adoption is starting to move faster across more categories. Read more: https://lnkd.in/g4Y6evXU #spatialcomputing #physicalAI #robotics

Just an ordinary day at a robotics company. Progress looks like chaos. We watch the viral backflips and perfect precision. We rarely see the thousands of slips, collisions, and face-plants that happen on the lab floor to get there. This isn't clumsy engineering; it's the "Sim-to-Real" gap in action. The difference between code and concrete is the most valuable data a robotics company possesses: ✔️ Reinforcement Learning (The Grind): In a simulation, a robot can train for 1,000 years in a single day. But real-world physics is unforgiving. Every one of these falls is a high-fidelity data point that refines the neural network's balance policy. ✔️ Resilience over Perfection: The goal isn't to build a robot that never falls. It's to build a system that can recover from failure in milliseconds, autonomously, without human intervention. ✔️ Domain Randomization: You see chaos; the algorithm sees variety. Kicking the robot, slippery floors, and random obstacles are features, not bugs. They force the model to generalize beyond its training set.

Another robotics masterpiece from our friends from Disney Research! Recent progress in physics-based character control has improved learning from unstructured motion data, but it's still hard to create a single control policy that handles diverse, unseen motions and works on real robots. To solve this, the team at Disney proposes a new two-stage technique. In the first stage, an autoencoder is used to learn a latent space encoding from short motion clips. In the second stage, this encoding helps train a policy that maps kinematic input to dynamic output, ensuring accurate and adaptable movements. By keeping these stages separate, the method benefits from better motion encoding and avoids common issues like mode collapse. This technique has shown to be effective in simulations and has successfully brought dynamic motions to a real bipedal robot, marking an important step forward in robot control. You can find the full paper here: https://lnkd.in/d-kzexdJ What Markus Gross, Moritz Baecher and the rest of the gang are bringing to life is unbelievable!