Better than test driving

Companies such as Google drive millions of kilometres to test the functions of their autonomous systems. To guarantee the safety of these systems, however, experts believe that test driving is far from sufficient.

The Federal Ministry for Economic Affairs and Energy, in cooperation with the automotive industry, suppliers and other experts, conducted the Pegasus project to develop a feasible and affordable, yet safe alternative for verification and validation of autonomous systems. To find out more about which processes and process modifications are required, we talked to Josef Horstkötter, Dipl.Ing., Senior Consultant and owner of F+S Fleckner and Simon Informationstechnik GmbH.

Mr. Horstkötter, what is the difficulty in verifying systems for autonomous driving?

Josef Horstkötter, Dipl.-Ing.: According to the recommendations of the Ethics Committee autonomous vehicles and vehicle functions are acceptable even if they are not absolutely safe. But they must be much safer than a human driver. Human beings are very good car drivers. On motorways, there is only one accident with serious injuries or fatalities for every 600 million passenger kilometres. In accordance with today’s state of the art, one would have to test drive more than six billion kilometres to prove that a system is twice as good as the average driver. Of course, that is far beyond the scope of possibility.

Other verification and validation methods are therefore needed that are affordable, while also clearly proving the safety of the system. Otherwise, autonomous vehicles will not gain social acceptance.

And these methods were developed by the Pegasus project?

Horstkötter: Partially. The Pegasus project was conceived as a central element in establishing the safety of a system at level 3 3. In particular, it should provide proof for the safety of the system. The main focus in this respect is verification and validation (V&V). In addition to the method developed by Pegasus, however, other tasks are also necessary in order to qualify a system as safe, such as “Safety by Design” or in-service tests.

Specifically, Pegasus developed a method for obtaining reliable proof of the safety of autonomous systems, in addition to an exemplary collection of verification and validation tools that will be expanded in the future.

How did the Pegasus project arrive at these results?

Horstkötter: The project was divided into four sub-projects with the goal of answering two central questions: “How safe is safe enough?” and “How can this be verified?”.

To find out, one group analysed the driving skills of human beings. Another expanded the established development processes to include dependency on the intended area of use, which is known as the Operational Design Domain (ODD), for example. A third group examined how systems can be tested, and another group focused on being able to use and develop the results in future projects, in order to actually provide a basis for a new state of the art.

What do human driving skills have to do with autonomous systems?

Horstkötter: To prove that a system is better than a human being, one first has to know how well the human would perform the same task. The statistical average for driving on a motorway stated in the introduction is insufficient. In addition to the average, one also has to take into account special scenarios, such as driving at night, and in rain, fog and construction zones, etc. In every scenario the autonomous system must be at least as good as the human counterpart. Otherwise, it has to turn the controls over to the human driver.

It must also be taken into account how well a human is able to take over control of the vehicle when the system reaches its limits. The driver cannot be overly distracted, and must have enough time to assess the situation. Any accidents resulting from this would potentially be the fault of the system and not of the human being.

About the third group: both software and system tests are long established. What still has to be developed in this area?

Horstkötter: The classic implementation tests (module test, software integration test, hardware tests, system integration tests and system tests) still have to be conducted, of course. They are primarily based on requirements, and it is possible to create good test cases for clear requirements.

However, in the case of complex autonomous systems such as those we are discussing, such clear requirements are often lacking. Although we know that they reach their limits under certain conditions, we do not know enough about the extent or details of such circumstances. These missing requirements must be determined by means of simulations and tests with the real system (sensors, actuators, algorithms). That is a cyclic process of testing, improving, and testing again, until proof is obtained that the system is safe enough.

How does that work in practice?

Horstkötter: To verify the autonomous system it is necessary to identify and verify as many situations and scenarios as possible. This can be done on the basis of simulation, test driving in test areas, or in field tests.

Simulation is the most economical and reproducible method. That is why the focus in simulation is on identifying the system limits and optimizing the system. A database with a description of scenarios on six independent levels helps to achieve the necessary completeness.

Field tests attempt to reproduce the limits identified in the simulation in order to verify that the real system does not behave more critically than the simulated system. The simulation is also limited in the case of extreme vehicle dynamics or sensor phenomena, where real tests provide more useful information.

In a field test one can test critical cases, and also experience “surprises” and identify new scenarios or parameters that were not previously obvious.

And that is then the newly developed method?

Horstkötter Laughs: A very general overview. Even a general description of the method would be beyond the scope of this interview. The finished concept often seems simple and logical, but to develop it from “nothing” is very difficult. And of course, I have only presented the concept here, it’s the little things that cause big problems, and many details have been worked out that still need to be implemented.

As with many publicly funded projects, the results are available to the public at www.PegasusProjekt.de/en/home. We have examined the results intensively in order to be able to provide solutions that are custom tailored to the requirements of our customers. That is the added value we deliver.

Then we wish you much success in applying the project results in your customer projects!