Research and Practice on Verification Test of Intended Functional Safety Awareness System Based on HWA System

Abstract

With the development of driving automation technology, more and more vehicles are equipped with intelligent driving functions. The Ministry of Industry and Information Technology (MIIT) issued in 2021 “Intelligent connected Vehicle production Enterprises and Product Access Management Guide (Trial)” (draft for comment) proposed that “intelligent connected vehicle products should meet the process assurance requirements of functional safety, intended functional safety and cybersecurity.” As the “eyes” of the HWA functional system, the perception system, due to its complexity, may still have safety problems caused by functional limitations such as sensors or algorithms in the absence of faults. Therefore, the intended functional safety analysis and verification methods for the perception system are particularly important to ensure that the system does not have unreasonable safety risks. This paper describes the analysis method of the intended functional safety function deficiency, and puts forward the test verification strategy and test verification method of known hazard scenarios. Taking the HWA system as an example, this paper analyzes the hazard scenario of the HWA perception system based on camera and millimeter wave radar, conducts testing and verification evaluation for typical hazard scenarios, identifies the insufficient function of the HWA perception system, and puts forward functional modification suggestions to ensure that the perception system does not have unreasonable risks.

1 Introduction

With the continuous improvement of vehicle intelligence level, intelligent driving system adopts a large number of perception sensors, perception fusion technology, planning algorithms, etc. In the complex vehicle operating environment, even if the intelligent driving system does not have functional safety-related failure problems, there may still be safety problems caused by functional limitations such as sensors or algorithms. The concept of anticipatory functional safety comes into being, which is mainly used to solve the safety problems caused by insufficient functions and human misuse of intelligent driving vehicles. In June 2022, the International Organization for Standardization (ISO) published the intended Functional Safety standard ISO 21448 [1].In order to accelerate the application and implementation of intended functional safety technologies and standards in the country, The National Automotive Standards Technical Committee has formulated the “China Functional Safety (Functional Safety) and intended Functional Safety (SOTIF) technology and standards Research Medium and Long Term Plan (2020–2025)” and “China Functional Safety (Functional Safety)Safety) and Intended Functional Safety (SOTIF) technology and standard system, the planning and standard system to the national standards GB/T34590 “Road vehicle functional safety”, GB/T “Road vehicle intended functional safety” as the guidance and research main line, Based on China's national conditions, research on technologies and standards applicable to the functional safety and intended functional safety of new energy vehicles, autonomous vehicles, traditional vehicle vehicles and key electronic control systems in China is carried out. At the same time, the concept of intended functional safety has been introduced by the United Nations, the European Union and other relevant organizations into the safety related regulations of intelligent connected vehicles, and the “Intelligent Connected Vehicle Production Enterprises and Product Access Management Guide (Trial)” (draft for comment) issued in 2021 in China puts forward the requirements of intended functional safety for enterprises and products. This makes the design, development and testing of intended functional safety become an unavoidable technical problem for enterprises.

In 2016, the autopilot Tesla rear-ended the road sweeper, resulting in the death of the driver [2]; In 2018, an Uber self-driving car hit a woman crossing the road during a road test[3].In 2020, an ideal car collided with a right-front lane changing truck when the driver assistance system was turned on [4];The analysis of a number of autonomous vehicle accidents found that the performance limitation of the autonomous driving perception system could not identify the target in front of it, which was the main cause of many accidents[5]. By applying the thought of intended functional safety analysis, this paper puts forward the method of analysis and test verification of the intended functional safety hazard scenario of HWA perception system, and realizes the intended functional safety goal through systematic and comprehensive analysis and test guarantee system to ensure that there is no unreasonable risk.

2 Intended Functional Safety Test Validation Scenario Analysis Method

Intended functional safety mainly protects the hazards caused by insufficient performance of electronic and electrical systems of intelligent driving vehicles or personnel misuse. The operation scenarios of intelligent driving vehicles are divided into four areas, namely known safety, known insecurity, unknown insecurity and unknown safety. The overall research idea is to find the scenarios that may cause harm by the system through systematic analysis. Then the function improvement and test verification of the system are carried out to make the unsafe scenarios as little as possible, and finally the risk acceptance criteria are met through the test verification. The methods used to analyze the intended functional safety performance deficiency and trigger conditions of the system include HAZOP, FTA, STPA, etc. Leveson proposed the STPA analysis method based on system theory in 2011[6], which is used to evaluate the safety of complex systems and identify safety constraints and requirements. The interaction between system, scenario and human can be regarded as the source of harm, and this method is more suitable for the analysis of intended functional safety trigger conditions of intelligent driving system. The method regards the whole system as a whole, systematically analyzes the possible unsafe control behavior, and combines the unsafe control behavior with trigger conditions to form a test scenario for the intended functional safety test verification.

The analysis of the intended functional safety hazard scenario is completed through two steps: hazard analysis, performance deficiency and trigger condition analysis. The specific analysis contents and steps are shown in Fig. 1. Firstly, the hazard analysis of the vehicle level is carried out according to the functional specifications and the corresponding vehicle level safety objectives are determined. The safety objectives are the basic guidance of the subsequent analysis and the target of the intended functional safety testing and verification. After that, it is necessary to establish a system control architecture by using STPA method according to functional specifications and system architecture design to analyze unsafe control behaviors that may lead to hazards. At the same time, it is necessary to analyze the trigger conditions that may lead to hazards in various parts of the system at the technical level, including perception layer, control layer, execution layer and human–computer interaction, and finally form a hazard test scenario.

Fig. 1

Analysis steps of the intended functional safety test scenario

2.1 Vehicle-Level Hazard Analysis and Definition of System Safety Objectives

Highway Assist (HWA) can be opened on structured roads such as expressways and urban expressways. It has the functions of lateral and longitudinal control such as lane keeping, car following, lane changing, overtaking, etc. It is one of the mainstream intelligent driving systems currently studied. This paper takes 5R1V architecture to realize HWA function perception system as an example to analyze and study.

Generally, the safety objective of the intended functional safety of intelligent driving system is put forward according to the functional specification according to the requirements of horizontal control, vertical control and human–machine interaction system respectively. For HWA sstems with horizontal and longitudinal control capabilities, the requirements for horizontal control include no unintended steering, insufficient steering force, excessive or insufficient acceleration during steering, etc. The requirements of longitudinal control capability include no unanticipated acceleration, unanticipated braking and insufficient braking force. In addition to the control requirements related to vehicle motion, the intended functional safety also focuses on the hazards of human misuse, and the safety objectives of the human–computer interaction system include the absence of inappropriate switching between the system and the driver's control control and inappropriate switching between the system and other systems. Table 1 below shows the unacceptable whole-vehicle hazard analysis of the HWA function and the corresponding safety objectives.

Table 1 Hazard analysis and safety objectives of H vehicle class

2.2 Analysis of Trigger Conditions of Sensing System

Trigger conditions are the direct causes of harmful events in the system. One or more trigger conditions may lead to harmful events in a specific operation scenario. In this paper, the analysis of trigger conditions is divided into two parts. The first step is to analyze the situation that may cause vehicle harm due to the sensor components carried by the intelligent driving car; the second step is to identify unsafe control behaviors using STPA, and form a hazard test scenario after combination. In this 5R1V sensing system, five millimeter wave radars and one camera sensing system are used.

Based on the sensors and system components carried by autonomous vehicles, the analysis of system and sensor interference elements, accumulation of expert experience, accumulation in existing databases, component-level safety analysis and other aspects are analyzed in Fig. 2. Among them, it is mainly based on the working principle of the sensor to analyze and summarize the interference elements of the sensor, for example, according to the camera mainly based on the working principle of image recognition, easy to be affected by the environment and light, the trigger conditions of the camera consider heavy rain, night, fog and other special environments; Millimeter wave radar echo signal is easy to be affected by metal material objects and misidentification occurs, and its triggering conditions consider static targets, speed limit signs, manhole covers, etc. According to the analysis of expert experience, when the target object is similar to the background color of the camera, the camera may produce missing recognition, such as a white truck in front of a sunny day; One of the limitations of millimeter wave radar is that the signal from the target is very different: when the motorcycle and truck are driving in front of the vehicle, because the motorcycle is relatively weak in reflection compared to the truck, the reflected signal of the motorcycle may be buried by the strong reflected signal of the truck, resulting in missing identification of the motorcycle. Analysis from available accident databases or test results, such as from an autonomous vehicle accident, suggests that the limited vertical separation capability of millimeter-wave radars may lead to a lack of differentiation between road infrastructure and stationary targets. Component level safety analysis is also performed, such as the automatic Emergency braking system (AEB) based on the camera, the processing error during image extraction and filtering, and the unintended opening of the AEB. The trigger conditions of human–computer interaction are analyzed according to the personnel misuse analysis method.

Fig. 2

Trigger condition identification method

2.3 Hazard Scenario Analysis Based on STPA

STPA analysis method mainly consists of four steps, namely defining analysis purpose, establishing control structure, identifying unsafe control behavior, and identifying causative scenarios. Defining the purpose of the analysis This part of the content analysis was performed in 1.1. The STPA method analyzes the unsafe control behavior of the system based on the control structure, in which the control flow chart of the system needs to specify the control components of the system, the executive components, the control or display instructions sent by the control components to the executive components, and the feedback signals sent by the executive components to the control components. This paper focuses on scenario generation and uses the HWP system control model built by Feng Hao [7].

According to the system architecture and the Unsafe Control behavior guide words, the corresponding Unsafe Control Action (UCA) is obtained. UCA = scenario + design operation scenario + guide word + control behavior + hazard event, for example, on the highway, the car is following the car, and the perception module does not detect the deceleration of the vehicle in front, resulting in a collision with the vehicle in front. Finally, the cause scenario is identified. The occurrence of unsafe control behavior is in a certain scenario, and there are performance limitations that will lead to harm. Therefore, according to the system architecture, it is necessary to filter the related items that cause the occurrence of hazards from the trigger conditions such as components and systems, and form the occurrence scenario of hazards, that is, the hazard scenario to be tested. Table 2 is part of harm scenarios of HWA sensing system.

Table 2 Harm scenario of HWA sensing system

3 Intended Functional Safety Test Validation Method

3.1 Testing and Verification Policies

For intended functional safety testing methods, a “three-pillar” approach is typically used for testing, consisting of simulation testing, closed site testing, and actual road testing, and the three pillar testing methods work together to ensure that the product does not present unreasonable risks.

Simulation test refers to the digital restoration of intelligent connected vehicles and their application scenarios by means of actual collection, computational reasoning and other modeling methods, and the establishment of vehicle and environment simulation models as close as possible to the real world. Through the analysis of vehicle operation in the simulation environment, the safety of vehicles in ODC is evaluated. In order to achieve the purpose of effective testing of intelligent connected vehicles [8]. The simulation test is mainly based on the product ODC to test the safety of the automatic driving system in nominal scenarios, dangerous scenarios and edge scenarios. On the basis of considering the reliability assessment of simulation model tools, tool chains such as SIL, HIL and VIL are applied for testing.

Closed site test refers to the real vehicle test for intelligent connected vehicles carried out in a closed site, which is used to verify the functions and performance of vehicles in typical scenarios [9]. The closed site test should focus on the ability of vehicles to cope with typical traffic environment [10].

The actual road test refers to the real car test for intelligent connected vehicles carried out on the open road. The test road should be selected according to the design and operation range of the automatic driving system, and the test mileage or time should be determined to ensure the full coverage of ODC scenario elements. Meanwhile, the test vehicle should be monitored to verify the ability of the declared automatic driving function to cope with the real traffic environment.

Based on the three-pillar test method, this paper proposes the intended functional safety test verification strategy combined with the three-pillar method for known scenarios, as shown in Fig. 3 below. The known hazard scenarios analyzed in the intended functional safety development stage are distributed rationally and verified through simulation test and real vehicle test. Simulation test can test all known scenarios, including test scenarios that can not be realized by real vehicles; The closed site test mainly tests the extreme danger scenario, and verifies the reliability of the simulation test. The actual road test can cover a large number of known scenarios, ensure the test coverage of known scenarios, and find unknown unsafe scenarios.

Fig. 3

Schematic diagram of test verification strategy

3.2 Test and Evaluation of Known Hazard Scenarios

According to the intended functional safety hazard analysis and trigger condition analysis of the HWA perception system mentioned above, fog is one of the influencing factors of the perception system. Therefore, this paper focuses on the HWA highway cruise assistance perception system, and designs the vehicle recognition test scenario of the front vehicle stationary accident under different visibility conditions in foggy environment. Since the foggy environment cannot be simulated by the real environment, therefore, simulation tests were carried out to verify the recognition ability of the perception system to the vehicle hazard scenario in a stationary accident in front of the foggy day, to verify the insufficient function of the HWA perception system, and to propose modification measures.

The test scenario was built through the simulation software Carmaker, and the sensor system under test was integrated by the dSPACE hardware in the loop simulation test system, where the front camera was installed in the video black box and the millimeter-wave radar was installed in the millimeter-wave radar echo simulator. The test equipment is shown in Fig. 4, and the test scenario is shown in Fig. 5. In a fog-filled environment, the vehicle under test is driving at a speed of 80 km/h on the express road when the HWA function is enabled, and there is a stationary accident vehicle in front of the vehicle under test in the lane, visibility of 50 m, 100 m and 200 m are set respectively in the fog-filled environment.

Fig. 4

dSPACE hardware-in-the-loop simulation test system

Fig. 5

Test scenario of the front static vehicle under different visibility in foggy days

The test results under different visibility conditions in fog were obtained through simulation test, as shown in Fig 6. Read test data from CANoe, including lane line information identified by the sensing system, type of vehicle ahead, speed, and position information, Table 3 is the test results under different visibility conditions in foggy days.

Fig. 6

Test result reading

Table 3 Test results under different visibility conditions in foggy days

It can be seen from the test results that fog has a great impact on the recognition ability of the perception system, and under different visibility, the recognition ability of the perception system does not meet the passing requirements. From the analysis of the expected functional safety processes, functional modifications should be made for the sensing system. The purpose of functional modifications is to identify and implement measures to address the risks associated with SOTIF and to update the input information for the Specification Definitions and Design. Therefore, propose functional modifications for this test results: add additional sensing device to improve the recognition of fog days; or improved sensor recognition algorithm; or limit the system operation in the functional specification.

4 Summary

Based on the thought of intended functional safety analysis and test verification, this paper proposes analysis and verification strategies and methods for the intended functional safety hazard scenarios of intelligent driving vehicles. In addition, based on the perception system of HWA functions, the hazard scenarios are obtained through systematic analysis methods, and reasonable verification strategies are selected for testing and verification. Through testing the recognition ability of the sensing system to the vehicle hazard scenario in a stationary accident in front of the foggy day, the function deficiency of the HWA sensing system is verified, and the modification measures are proposed. This method can reduce the risk of hazards in intelligent driving system by ensuring the sufficiency of hazard scenario analysis, which is of great significance for ensuring the intended functional safety of intelligent driving vehicles.