Bridging the Sim-to-Real gap with Florencia Grosso

InOrbit, in collaboration with Ekumen and NVIDIA, recently announced some exciting advancements in bringing RobOps to simulations. This work leverages the NVIDIA Isaac Sim™ extensible robotics simulator powered by NVIDIA Omniverse™, blending dynamic 3D simulation technology with real-world automation environments, all managed by the InOrbit platform. Bringing these two domains together, effectively bridging simulation to reality and leveraging InOrbit for multi-vendor robot operations, is no easy task. Today we explore the Sim-to-Real gap with InOrbit’s Sr. Robotics Engineer Florencia Grosso (Flor).

We spoke with Flor some time ago about her work at InOrbit, and she hasn’t slowed down in the slightest. She is one of 50 women in robotics you need to know about 2023. She was a recent speaker at ROSCon 2023, where she was part of the ROSCon Diversity Committee. She recently joined us as a panelist for Tech Together at the InOrbit Robot Space to share her story with other inspiring women in robotics. Flor works at InOrbit and as a Manager and Robotics Engineer at Ekumen. She was the lead in the recent simulation work with NVIDIA, liaising technical development between the teams.

Flor has a unique insight into why companies building and deploying robots should be using simulations, why bridging the simulation-to-reality domain gap is so important, and how the industry can accelerate the adoption of simulation as it moves beyond traditional limitations.

InOrbit: Let’s set the stage for everyone, how are simulations generally used in robotics?

Flor: I believe that we are in the middle of a change in the paradigm regarding simulations. Simulations used to be the last link in the chain of robot development. Teams built sims once everything else (robot hardware, software, etc) was complete. Now sims are becoming an important tool to validate ideas, even in the early stages.

Simulations are useful when validating prototypes before getting fully hands-on with new hardware. Depending on the robot's size and cost, this can be significantly cheaper and sometimes faster than building the real robot first. Just imagine the cost of building a full-scale automated construction crane, compared to its digital version. Simulations also provide guidance on important details such as where to place sensors.

By leveraging the power of AI, simulations can also enable testing different configurations or conditions of the environment where the robot will run, teaching it to anticipate different scenarios by randomizing data. Simulations have become a key tool in reinforcement learning, teaching robots how to behave under different conditions by relying on data-driven trial and error techniques. Both of these areas are currently growing at a fast pace, powered by cutting-edge tools like NVIDIA Omniverse.

One interesting application that we are leveraging as part of our recent demo of a virtual Robot Space is how simulations are also handy for non-roboticists. Robotics end users, whether they are running logistics at a hospital or operating warehouses around the world, are gaining more focus. Increasingly, end users must integrate fleets of robots from different vendors in order to automate different process workflows. They need a way to validate that the setup they are building matches their needs, and then drive continuous improvements over time. Simulators are a way to accomplish this.

Of course, simulators continue to be key to feature testing and debugging. Working with different engineering teams, every time I report an issue I also supply the relevant data to reproduce the problem on a simulator.

InOrbit: What are the challenges to effectively using robotics sims?

Flor: Even though simulations can be faster and cheaper than trying to implement a solution in hardware, they still require considerable work to function properly. Building and managing a good simulation is multi-disciplinary work, from replicating the environment and robot models (which in some cases involves detailed 3D designs) to accurately representing robot behavior and interactions with the world.

Another challenge is the complexity of the hardware. As robotics solutions get more refined and require features like photorealistic multi-robot simulations, intelligent hardware needed to run them is in high demand and has a higher cost. The recent surge in AI applications, which rely on the same type of hardware, has made this even more challenging. I expect these challenges to be addressed as the technology is adopted more widely, including solutions like cloud-based simulations, which are far easier to scale.

Robotics has been expanding over the last few years and now encompasses a wider range of fields (from indoors to outdoors, aerial and aquatic robots) and implementations (ROS, non-ROS or hybrid). However, the native simulation tooling is not always ready to support all of these cases. This is a temporary problem though, as simulator developers continue to update their product support and other companies integrate additional advanced tools with existing simulators.  

The combination of all these challenges and the state of the art in simulations results in the so-called Sim-to-Real gap.

InOrbit: What is this Sim-to-Real gap?

Flor: The simulation-to-reality gap is the discrepancy between how a robot behaves in a simulation versus the real world. Many of us have been working to bridge the gap, and our latest work takes another step in that direction.

There is a lot of detail needed to capture a real environment accurately in simulation. Factors like changing light or temperature, which may be difficult to replicate in sim, can have a major impact on real robots. From simulation physics to robot sensor fidelity, the gap occurs when the physical environment and response to those conditions can't be modeled in sufficient detail. But I would say that overall the gap is getting narrower, as different teams and companies in the robotics ecosystem put time and effort into bridging it. Some areas, such as computer vision applications, have received more attention, resulting in reducing the gap.

We may still need some time to improve robotics tools and our modeling of the real world to achieve an exact replication of robot behavior. The good news is that this development is progressing at an accelerated pace, allowing us to see changes in a relatively short time.

Today we have photorealistic simulations that look more real than ever. From the different textures and lighting that can be adjusted to match various scenarios, it is possible to train robot models over a larger number of dynamic missions that very closely match reality. When this is paired with real robot data and RobOps principles, the sim-to-real gap gets even narrower.

InOrbit: Can you explain this project to bring RobOps to simulations?

Flor: Throughout the development of the InOrbit platform for robot operations, we have relied on both simulations and real robots to test new features and explore different solutions. However, this is the first time that we merge both worlds together, with a simulation that replicates in exacting detail both the environment and the robots at our very real demo facility and concept store, the InOrbit Robot Space. It is definitely a step towards closing the Sim-to-Real gap, and we can’t be more proud of taking it.

The integration was a joint effort between different teams and companies. Our first step was to design a 3D version of the Robot Space that followed every detail of the real version. This task was carried out expertly by our talented UX Designer, Ryutaro Sakai, using Blender and NVIDIA Omniverse. The different iterations of this model were shared with the team at Ekumen, led by Christian Barceló, who worked with Isaac Sim to provide the necessary feedback and make any adjustments that were needed. The Ekumen team executed the backend work needed to get two of the robot models currently running at the Robot Space available in sim, and replicating their counterparts’ real behavior. With that in place, the InOrbit team completed the integration with their RobOps platform, enabling robot missions and more to execute in simulation and reality. You can see this in action in our very cool demo video.

What I’m describing above in a simple paragraph is a summary of weeks’ worth of collaboration between different teams, including enormous support provided by NVIDIA’s technical team. This was a true collaborative effort!

InOrbit Why is this work important?

Flor: This work is important for a few reasons. Most importantly, it shows the power of integrating RobOps tools to support orchestration with advanced simulations for various end user applications. It also highlights the value of joining multi-disciplinary efforts within the robotics ecosystem.

I think this is a winning scenario for everyone. By leveraging advanced simulations, InOrbit users can now validate how a solution offered matches their needs while making complex tasks like interoperability easier by providing a single pane of glass that connects with different robot models and vendors. Robot developers can also benefit from integrating InOrbit with simulations of their robots to inform real-world value throughout the build and deployment process.

This is a collaborative integration that highlights the significance of not reinventing the wheel. Instead of starting from scratch when integrating a simulation, we hope companies will leverage this work and the power of existing platforms like InOrbit and NVIDIA Isaac Sim to drive their simulations. Just as this project saw us relying on each other, from the expertise in robotics development at Ekumen, to the cutting-edge RobOps platform of InOrbit, to the photo-realistic and innovative robotics simulator offered by NVIDIA Omniverse, we know those who rely on us will find value.

InOrbit That's very exciting, so what's the next step?

Flor: Oh, this is just the tip of the iceberg. We now have a photorealistic representation of a multi-robot environment that matches our real-world environment. We intend to add more robot types and behaviors to leverage the simulations’ full potential. We are also expanding the physical space by 3X to enable new use cases, and our simulations can reflect that and even grow beyond that, simulating for example a large warehouse that extends the physical space. However, this is only one particular simulation. We see a lot of value as we extend this integration to our customers, with a focus on end users who are increasingly adopting robots into their operations.

Next, we plan to explore the amazing forensic opportunities for AMR incident management enabled by sims that are directly linked to InOrbit RobOps tools. This is an area we believe has a lot of potential moving forward.

I’ll leave you with this: the teams at NVIDIA, Ekumen and InOrbit are all incredibly excited to push this technology forward as we continue to develop interesting applications and strengthen this integration which starts to bridge the gap between simulation and reality.

—

A huge thanks to Flor for sharing her insights. As she said we are excited to share more with you in the weeks and months to come as we continue to bridge the Sim-to-Real gap with a foundation of effective RobOps. Click here to explore our work bringing RobOps to advanced simulations in more detail.

BLOG UPDATE

Flor graciously took some additional time to sit down with us for a video interview on the state of robot simulations. Please enjoy!