Over the last year, we have been talking with robotics companies and operators in Silicon Valley and around the world. We have learned a few things about the challenges of operating robots at scale. This has mostly confirmed that, compared to the great advances in software and hardware components that are helping robots enter every industry, robot operations and infrastructure are still immature.
We have found that most robotics companies are using tools designed for developers in the lab to operate robots in the field. This is often an after-thought or a distraction from the “cool work” of building robots. In the words of the CEO of a successful robotics company with hundreds of robots in the field: “our engineers have cobbled together a bunch of tools.”
Is this how companies should be managing costly machines that weigh hundreds or thousands of pounds and move around on their own amongst people? It is more than a little ironic that, for an industry that is bringing about massive changes through automation, many of its own operational practices are extremely manual, repetitive and error-prone.
Part of this may be explained away as a matter of industry maturity. The number of companies with thousands of autonomous service robots in operation is still relatively small. However, the journey that robotics companies follow as they scale is markedly similar, regardless of their specific application or industry. Robotics startups that may not yet be feeling the pain of trying to run or support hundreds of robots in far-flung locations would be well served by learning from their more advanced brethren.
Below is a list of some of the most egregious issues we’ve found in robot operations. Not every company is incurring all of them or to the same degree, and some of the companies we are working with at InOrbit are already far ahead of the curve, but as a whole this paints a pretty grim picture – or rather, there’s a lot of room for improvement.
Poor security practices
Many robots share the same access password or have generic management accounts as a way to reduce the management complexity, at the expense of reduced security
Reliance on SSH
Creating an SSH connection and running tools over a point-to-point connection is a catch-all solution in the absence of proper tools designed robot operations in the field
Inadequate incident response
When an issue occurs, it can take a long time for the problem to propagate and tracking of the resolution is often incomplete, making it hard to manage or get to the root cause
Lack of real-time notifications
Related to the previous point, many systems are able to perform limited logging but are missing a real-time mechanism to detect and respond to issues as they occur
Frequent navigation exceptions
Deployment in the real world is messy, resulting in frequent issues that result in the robot leaving autonomous mode, reducing operational efficiency and asset utilization.\
Running scripts and commands manually
Robot configuration and management is often done by running sequences of commands or scripts manually, resulting in “snowflake” robots that are impossible to manage at scale
Networking issues including VPN traversal
It is common for roboticists to continue using in the field tools that were meant for the lab where they run on the same local network, but operating in customer-controlled networks
So what can be done about it? Plenty.
In terms of tooling, the InOrbit platform offers solutions to most of the issues above. Given our robot-agnostic approach and ease of platform extensibility, InOrbit can be deployed quickly for any size robot fleet, including heterogeneous fleets with different types of robots for different tasks. If you want to dig deeper, you can try InOrbit for free, just go to inorbit.ai/quickstart.
However, our mission goes beyond that: we want to help robotics companies get to operations at global scale. To get there, the industry needs to build operational best practices. We are helping create a movement, similar to the adoption of DevOps practices and tools.
To that end, we have founded a Robot Operations Working Group. We are starting small, with a few experienced leaders and passionate robotics practitioners. Our first virtual Meetup will be later this week. If you are interested in joining this “cross-industry group aimed at developing and sharing best practices and advancing the state of the art for the operation at scale of autonomous robots”, you can learn more at ROWG.