Keywords

1 Introduction

With the continuous development of the level of driving automation, the SOTIF need for driving automation system has become increasingly prominent [1]. As an important link to ensure that vehicles meet the requirements of SOTIF, unknown scenario verification method urgently needs research and breakthrough. As an important pillar of intelligent connected vehicle safety, expected functional safety is mentioned in many domestic and foreign intelligent connected vehicle policies and regulations such as Opinions on Strengthening intelligent Connected Vehicle Production Enterprises and Product Access Management [2], and UN R157. At present, there are mainly the following problems in the confirmation test of expected functional safety:

  1. 1.

    How to customize the second-level acceptance criteria according to the product functional definition and the launch city.

  2. 2.

    How to generate validation objectives according to the second level acceptance criteria.

  3. 3.

    How to develop appropriate validation test plans according to validation objectives and product attributes.

At present, there is a lack of custom formulation methods based on target market for the formulation of acceptable criteria for expected functional safety. This paper mainly studies the second-level acceptance criteria and related validation tests. This paper studies and proposes a customized SOTIF validation test method for highway assistance (HWA) functions based on Design Operation Domain (ODD). A validation test method based on multi-dimensional key elements of the target market is designed. Based on the HWP function, experiments are carried out to verify the effectiveness and practicability of the method.

2 Brief Description of Expected Functional Safety Validation Test Requirements

For autonomous driving functions, it is crucial to demonstrate that the system does not create unreasonable risks at SOTIF level. In order to meet the requirements of SOTIF, autonomous driving systems generally carry out safety demonstration from two layers of acceptance criteria. Acceptance criteria consist of first-level acceptance criteria and second-level acceptance criteria [3]. The first level of acceptance criteria is the hazard behavior acceptance criteria, which is used to determine whether the vehicle's behavior constitutes a hazardous behavior [4]. The criteria here are “controllable” (C = 0) or severity rated “no harm” (S = 0). The second level of acceptance criteria is the residual risk acceptance criteria., which is used to determine whether the residual risk is at a reasonable level during the operation of the vehicle. The first level of acceptance criteria is the micro-level acceptance criteria. The second level of acceptance criteria is an acceptance criterion that evaluates the risks at the vehicle level from a macro perspective. The second layer of acceptance criteria mainly considers whether unknown scenarios can accept risks at the vehicle level. Validation testing for SOTIF is primarily a validation of the second level of acceptance criteria, i.e. testing for residual risks in unknown scenarios.

After the first layer of acceptance criteria defines hazardous behavior at the micro level, the second layer accepts that the criteria constrain the residual risk at the vehicle level at the macro level. The confirmation verification strategy is the strategy of actual verification and confirmation testing of the two-layer acceptance criteria. This article focuses on a detailed study of the confirmation test methods for the second acceptance criteria, taking into account the characteristics of the target market.

3 Expected Functional Safety Confirmation Test Method

The SOTIF validation test method mainly includes the formulation of the confirmation goal and the confirmation test execution method.

3.1 Identification Targets Based on Target Markets

Based on the acceptance criteria, the export process for confirming the target is:

  1. 1.

    Confirm the second level of acceptance guideline.

  2. 2.

    Select the export method to confirm the destination.

  3. 3.

    Select the safety margin and other relevant parameters such as the confidence level.

  4. 4.

    Get the target confirmed.

The second level of acceptance criteria mainly deals with the residual risk at the vehicle level from the macro level, mainly to determine whether the hazard risk accumulated during operation is at a reasonable acceptable level. The second level of acceptance criteria can be demonstrated in terms of the number of vehicle accidents and/or the number of casualties. Starting with the official statistics of the target market is a recommended path, and this article will provide a target market data derivation idea to demonstrate the implementation of the second level acceptance standard at the macro level.

Considering the information of the target market in the process of formulating the second-level acceptance criteria can ensure that the acceptance criteria are formulated in line with the conditions of the target market. The derivation process of the second level acceptance criterion mainly has the following steps: 1. Hazard scenarios with unreasonable risk need to be considered; 2. Data sources derived from the definition of acceptance of guideline statistical information, in the general case accident statistics B; 3. Consider the safety margin Y; 4. Provide an acceptance criterion AH with a safety margin, and finally derive an acceptance criterion with a safety margin \({\text{A}}_{{\text{H}}} = {1}/\left( {{\text{b}}*{\text{Y}}} \right)\).

The second level of acceptance criteria is a risk assessment of the autonomous driving system at a holistic level, so the derivation data should be based on the information that the target is placed on the market. At present, there is a lack of systematic combing of the method of generating second-tier acceptance criteria based on target city customization, and this study provides a second-layer acceptance criterion setting method based on target market.

In the data sources for the second tier of acceptance criteria, the data need to consider the following factors:

  1. 1.

    Statistical area, statistical area contains the name of the country/city, the description of the region, and the general description of the statistical area. The area description includes the statistical time period (all-day/daytime/night-specific time), statistical length, weather characteristics, and road structure characteristics. The description of the generality of statistics contains the typicality of the statistical region, the description contains the reason for the selection of the region, and the typical characterization of the region. If the ODD is different from the statistical area, the extension description of the statistical area to the ODD should be added to explain the degree of relationship between the accident rate of the current statistical area and the accident rate of the ODD range.

  2. 2.

    The length of statistics, the start date and end date of statistics, should generally be maintained for at least 1 year, to ensure that the information of the four seasons can be counted, and there is no fluctuation in the accident rate caused by seasonal climate change.

  3. 3.

    ODD settings of vehicles, such as urban non-expressways, urban expressways, highways. The ODD here is used for comparison with statistical regions for stripping the data.

  4. 4.

    Regional accident statistics, here is the focus of accident investigation, including statistical time period, total number of times, statistical standard description, statistical data to accident estimation ratio, the number of casualties caused by casualties, the depth of the accident, etc. The statistical standard description here contains statistical standards, statistical standards and accident promotion instructions. (According to the classification description of the ESC for functional safety, those that do not cause personal injuries can be classified as E0, so the focus should be on accidents that include human injuries.)

  5. 5.

    Regional motor vehicle statistics, in the statistical time & statistics of the number of motor vehicles in the region, motor vehicle evaluation driving time, average driving mileage of motor vehicles. Here, in a practical study, it is more accurate to calculate the traffic flow of the expressway.

  6. 6.

    The method of statistics for the above areas, including the statistical sources and methods of the above data and the reasonableness of the methods.

  7. 7.

    Use the data to select a method that generates acceptance criteria, such as GAMBAB.

The resources that can be used include: data from the National Bureau of Statistics, cooperation and exchange with local traffic management departments, road test development, literature, network data, etc. Here, the HWA function target ODD is used as an example in Chongqing. First, it is determined that the ODD of the HWA function is a high-speed structured road.

Taking Chongqing as an example, the following data were obtained from the national statistical yearbook and literature (see Tables 1 and 2):

Table 1 Statistics on targeted acceptance criteria in target cities
Full size table
Table 2 Examples of second-tier acceptance criteria in target cities
Full size table

It can be seen that on the expressway in Chongqing, the average mileage of driver accidents is 10159635.08 km. From the GAMAB principle, autonomous driving functions need to be at least better than the driver's ability, i.e. less average accident mileage.

From the above table, it can be obtained that with a safety margin of 2 and a confidence level of 0.99, and an ESC probability of 0.1, Chongqing's confirmed target is 93,573,500 km.

3.2 Requirements for Confirming the Test Method

Confirming the test method requires, on the one hand, whether the system meets the residual risk acceptance criteria, that is, whether there is sufficient anticipation of the risk of unknown scenarios. On the other hand, it is necessary to fully discover the expected functional safety risks of the system and make timely functional modifications to ensure that the system fully meets the expected functional safety requirements. Therefore, for confirmatory testing, it is critical to meet the test adequacy of unknown scenarios and the data traceability of functional modifications for expected functional safety issues.

3.3 Select Development Mode for the Test Method

The confirmation strategy of unknown scenarios mainly assigns mileage and duration under different test methods according to the key points of simulation and real vehicle testing (see Fig. 1).

Fig. 1
A block diagram for the unknown scene confirmation has a three-step testing process including simulation, simulation plus real vehicle testing, and open road testing.

Confirm method for unknown scenarios

Full size image

Simulation test: According to the ODD function under test, set the distribution ratio of environmental requirements and traffic vehicle types, and automatically generate simulation test scenarios. The simulation test of unknown scenarios focuses on the random combination and coverage of different scenarios and scene parameters.

Real vehicle test: consider the project cycle and mileage, covering different road types, time types, weather types, and driver types. For unknown scenario testing, open road testing is necessary, and the route of open road needs to fully cover the key elements of ODD (duration, road type, etc.).

In order to explore the unknown hazard scenario of the HWP function and confirm that the probability of encountering the unknown hazard scenario is low enough, according to the limitation of its ODD, the large mileage test of the open road is the most close to the user use scenario and the most realistic test method, so this method is selected.

In view of the test adequacy of unknown scenarios, it is necessary to pay attention to the route selection of the confirmation test and the selection of the time of the confirmation test in the way of confirming the test method.

To confirm the selection of road sections for the test, it is necessary to traverse the expressway according to the distribution of high-speed routes in the target city and the needs of scene traversal, and evenly distribute the time on each highway according to the distribution of traffic flow.

To confirm the selection of road sections for the test, it is necessary to traverse the expressway according to the distribution of high-speed routes in the target city and the needs of scene traversal, and evenly distribute the time on each highway according to the distribution of traffic flow. The time period for the confirmation test was selected to be distributed according to the percentage of day, night, and dusk available from local highway traffic statistics.

3.4 Test Equipment for Unknown Scenarios

In terms of the data traceability required for functional modification of expected functional safety issues, it is especially critical to confirm the equipment mode, data recording method, data statistics and other methods of testing in unknown scenarios.

The types of data that need to be recorded to confirm the test are: external environment information, vehicle motion information, driver operation in the car, internal interaction information of the vehicle, and vehicle controller signals. In order to ensure the traceability and accuracy of the data, the test is carried out using a combination sensor and a data synchronization device.

Confirm the equipment method of the test (see Fig. 2): use external cameras to record the surrounding environment information of the vehicle, internal cameras to record the driver's hand and foot operations, and use the in-car camera to record the human–computer interaction display, use the inertial navigation system to record the movement information of the car, use the intelligent camera to record the relative position of the wheels and lane lines, use lidar to record the relative distance from the vehicle in front, and finally use the terminal equipment to synchronize all nodes and record the key nodes and information of the test for labeling.

Fig. 2
An illustration has model vehicles with various test equipment used in vehicle navigation and monitoring, including cameras for recording different aspects like the vehicle’s surrounding environment, driver’s operation, and human-computer interaction display.

Test equipment

Full size image

During the test, the data is recorded through the truth value of the vehicle and the subjective point of the safety officer. The different risky behaviors are recorded as follows (see Table. 3).

Table 3 Data recording
Full size table

The subjective data is the safety officer's continuous labeling of the scene in the passenger seat during the test execution, and the objective data is the camera, lidar and related controller signals.

4 Experimental Validation

According to the above process, this paper carries out a confirmation test based on a prototype of HWP, selects the local highway to carry out forward and reverse bidirectional traversal according to the method proposed in this paper, and the test duration distribution is defined according to the statistical definition of the highway traffic time period, and finally carries out the verification test according to this test strategy. The high-speed test period has been opened for 1989.7 km and has been tested for a total of 1940 min. There are 182 times hazard behavior detected, and the average mileage of hazard behavior was 10.9 km (see Table. 4).

Table 4 Statistical results of the test period
Full size table

The distribution of risky behaviors and the main causes are shown in the Table 5.

Table 5 Distribution of hazard behaviors and scene elements
Full size table

Next, more vehicles with HWA systems on the market will be tested and verified to further confirm the feasibility of the test method.

5 Conclusion

In this paper, a customized SO verification test method for highway assistance (HWA) function based on design operation domain (ODD) is studied and proposed, and a test route formulation method based on multi-dimensional key elements is designed. This method fully considers the data dimensions and environmental characteristics involved when deriving acceptance criteria and confirming targets, and effectively meets the adequacy requirements of unknown scenario experiments. This paper designs a vehicle test method for HWA for expected functional safety, explains the selection principle of the function to be tested, the basis of the test method—safety analysis and the design of the test method, and conducts verification tests. The experimental results show that the test method designed in this paper is reasonable and implementable.