New Approaches to Robot Interoperability: Large AI Models as an Opportunity for Collaborative Action

Whether in intralogistics, manufacturing or rescue operations: when different robots are required to carry out tasks together, they must be able to exchange information, understand its meaning and coordinate their actions. Until now, achieving the necessary interoperability – for example, through standardisation – has involved a great deal of effort. Large AI models open up new possibilities for achieving this. A new discussion paper from the Plattform Lernende Systeme shows how such AI models can support the interaction of robotic systems and what challenges still need to be overcome to ensure reliable and safe deployment.

A drone surveys the site, a robot dog inspects hard-to-reach areas, and a vehicle transports materials: in the event of a fire at a chemical factory, different robotic systems could complement one another’s capabilities. Robots can also work alongside other robots and humans during fruit harvesting or the transport of goods. In practice, however, differing interfaces and, in some cases, closed operating systems make collaboration difficult.

Four levels of interoperability

Interoperability lays the foundation for communication and cooperation, but does not in itself bring about joint action. To date, it has primarily been achieved through standards for interfaces, protocols or data formats. Such solutions are complex and often lack flexibility. Establishing a shared understanding of information and its significance for action is particularly challenging.

The discussion paper distinguishes between four levels of interoperability: the technical level connects systems with one another, whilst the syntactic level ensures compatible data formats. The semantic level concerns the meaning of the information exchanged, and the pragmatic level concerns the understanding of the consequences of actions within a specific context. It is only at this final level that the actions of systems can best be coordinated with one another.

What large AI models are capable of

“Large AI models can help to achieve interoperability between different robotic systems at a semantic or even pragmatic level in an elegant and efficient manner,” says Jürgen Beyerer, Executive Director and Head of the Fraunhofer Institute for Optronics, System Technologies and Image Processing (IOSB) in Karlsruhe, and holder of the Chair of Interactive Real-Time Systems at the Karlsruhe Institute of Technology (KIT).

In addition to large language models, models based on logical reasoning or those that combine language, vision and action are also suitable for this purpose. They can help to categorise information from different systems and make it usable for coordination. Crucial factors are the world knowledge represented in the model – which is linked to the type and scope of the data used in model training – as well as additional contextual information made available to the model during operation.

Collaboration requires reliable knowledge of the world

“Robots can carry out complex sequences of actions or missions as a team if they are able to interact physically, exchange information and interpret it reliably. The crux of the challenge often lies in the systems’ autonomy and their model of the world. Today’s large generative AI models can fill gaps in this area and coordinate robots; future models could even establish new forms of interoperability,” explains Sirko Straube, Research Area Manager for Emergency Response and Recovery Management at the Robotics Innovation Centre of DFKI GmbH.

However, current models have their limitations. They frequently lack activity data and domain-specific data. They sometimes produce erroneous statements and have only a limited understanding of cause and effect. Furthermore, they are not yet able to reliably translate objects, positions or actions into precise specifications for physical models. As a result, interoperability can occasionally ‘break down’ – posing risks to functional and process safety. More reliable multimodal AI models are promising for the future; these will be able to verify facts in real time and better assess the consequences of actions using physical simulations and world models. For practical application, however, these capabilities must be reliably available.

“Interoperability between robots is a crucial building block for the future, which will drive forward the practical use and collaboration of diverse robotic systems in the long term. Researchers are already developing promising projects and concepts – the task now is to work together to transform these into marketable, application-oriented solutions. This requires close collaboration between users, robotics manufacturers and the research community to jointly take robotics to the next level of development,” summarises Johannes Hinckeldeyn, Director of Advanced Corte Technologies at the KION Group.

About the brochure

The position paper ‘Interoperability between Robotic Systems – The Potential of Large AI Models’ was authored by members of the ‘Learning Robotic Systems’ working group and the ‘Technological Enablers and Data Science’ working group of the Learning Systems Platform. It is available for download free of charge.