This paper presents a reliability analysis method for automated vehicles equipped with adaptive cruise control (ACC) and autonomous emergency braking (AEB) systems to avoid collision with an obstacle in front of the vehicle. The proposed approach consists of two main elements, namely uncertainty modeling of traffic conditions and model-based reliability analysis. In the uncertainty modeling step, a recently developed Gaussian mixture copula (GMC) method is employed to accurately represent the uncertainty in the road traffic conditions using the real-world data, and to capture the complicated correlations between different variables. Based on the uncertainty modeling of traffic conditions, an adaptive Kriging surrogate modeling method with an active learning function is then used to efficiently and accurately evaluate the collision-avoidance reliability of an automated vehicle. The application of the proposed method to the Department of Transportation Safety Pilot Model Deployment database and an in-house built Advanced Driver Assist Systems with ACC and AEB controllers demonstrate the effectiveness of the proposed method in evaluating the collision-avoidance reliability.

## Introduction

Autonomous vehicle (AV) is considered as one of the game-changing technologies that will change the transportation styles of current society. Research shows that AV has promising potential in improving the road safety by eliminating human errors [1], and easing the traffic congestions through optimal control [2,3]. Before the mass production and the deployment of AVs in real traffic environment, the safety and reliability of AVs are recognized as one of the most important factors that need to be assured.

Vehicle collision avoidance system is an essential element of the autonomous driving technology. It plays a vital role in avoiding collision during the driving process and warning the passengers to switch to traditional driving if they feel uncomfortable. Analyzing the risk of collision of AVs in real-world traffic condition is an indispensable step in the development of AVs and has gained much attention in recent years. For instance, European New Car Assessment Program (NCAP) [4] has developed multiple testing scenarios to verify the versatility of the control logic of AVs in the collision avoidance situation. In order to provide accurate warning in a timely manner to the driver, Wu et al. [5] proposed a method for the vehicle motion prediction and collision risk assessment based on the simulation of vehicular cyber physical system. Chang et al. [6] suggested a collision avoidance system based on short range communications. Deng et al. [7] investigated the risk of collision in the car-following scenario under bad weather conditions. Similarly, Zhao et al. [810] applied importance sampling approach to test-based collision risk assessment of AVs. They investigated the performance of collision avoidance systems in different traffic conditions, such as car-following scenario and lane-change scenario. Inspired by the application of importance sampling approach to the collision-avoidance performance evaluation of AVs [8], various approaches have been proposed recently including piecewise mixture models-based method [11], application to cut-in scenario [12], and integration of importance sampling approach with kernel method [13].

Even though various approaches have been developed to evaluate the collision risk of vehicles under various traffic conditions, most of the approaches make assumptions on the randomness or uncertainty variables in the collision-avoidance system. To release this assumption and simplification, naturalistic driving data [10] have been extracted to capture the variability in the real-world traffic conditions. The current methods [10,11], however, usually employ a specific statistical model such as exponential distribution or piecewise mixture models. These statistical models cannot accurately capture the complicated correlations between different random variables and thus lead to errors in the collision reliability analysis of AVs. In addition, a major challenge in the collision reliability analysis of AVs is how to effectively reduce the number of function evaluations to the computer simulation model or the number of vehicle tests if testing-based method is employed. The application of importance sampling approach to collision avoidance system performance analysis has shown promising potential in reducing the number of tests and reducing the testing time [8]. The required number of tests, however, is still very high to reach to an accurate estimation of the collision-avoidance reliability. Especially when the collision-avoidance reliability is high, the required computational or testing resources are practically not affordable.

Aims to efficiently perform reliability analysis without sacrificing the accuracy, various adaptive surrogate modeling-based methods have been proposed in recent years. Two most widely used approaches are the efficient global reliability analysis method [14] and adaptive-Kriging Monte Carlo simulation (AK-MCS) approach [15]. Following these two methods, various adaptive surrogate modeling-based reliability analysis methods have been proposed, such as the global sensitivity analysis enhanced method [16], adaptive surrogate-based system reliability analysis [17], and single-loop Kriging for time-dependent problems [18]. Results of numerical case studies of these methods have shown that the adaptive surrogate modeling-based reliability analysis methods can dramatically reduce the number of function evaluations in reliability analysis. It provides a promising direction to overcome the challenges in the collision-avoidance reliability analysis of AVs.

This paper aims to develop a systematic collision-avoidance reliability analysis framework for AVs equipped with adaptive cruise control (ACC) and autonomous emergency braking (AEB) systems. The developed framework consists of two main elements, namely uncertainty modeling and reliability analysis. In the uncertainty modeling module, a recently developed Gaussian mixture copula (GMC) is employed to model the uncertain traffic conditions based on the naturalistic driving data. The Gaussian mixture copula allows for not only accurate modeling of the marginal distributions of various traffic conditions but also accurate representation of the complicated dependence between different variables. In the reliability analysis element, the adaptive surrogate modeling-based reliability analysis method with active learning function is employed to efficiently perform collision-avoidance reliability analysis. The synthesis of these two elements enables us to accurately and efficiently evaluate the collision-avoidance reliability of an AV control system. Based on the developed collision-avoidance reliability analysis framework, the potential application of the proposed method in reliability-based design optimization of controller parameters is discussed. The effectiveness of the developed approaches is demonstrated using the Department of Transportation Safety Pilot Model Deployment database and an in-house built Advanced Driver Assist Systems. The contributions of this research are therefore summarized as: (1) uncertainty modeling of the uncertainty sources in the real-world driving environment using a new Gaussian mixture copula method based on naturalistic driving data; (2) employment of adaptive surrogate modeling approach to reduce the number of simulations/tests in AV collision reliability analysis; and (3) synthesis of these two approaches and demonstration of the developed framework using the Safety Pilot Model Deployment database.

The remainder of this paper is organized as follows: Section 2 provides a background introduction of collision-avoidance reliability analysis of automated vehicles. Section 3 gives the adaptive surrogate modeling-based collision-avoidance reliability analysis method. Section 4 illustrates the effectiveness and the implementation procedure of the proposed method using a case study, and Sec. 5 presents the concluding remarks.

## Collision Avoidance System of an Automated Vehicle

This section briefly introduces the background of collision-avoidance reliability analysis of an automated vehicle.

### Collision Avoidance System of an Automated Vehicle.

Collision avoidance system is usually designed to avoid the collision between an automated vehicle and an obstacle in front of the vehicle (as shown in Fig. 1). The obstacle can be any object in front of the automated vehicle, such as a leading passenger car, a bicyclist, a truck, or any object that appears suddenly on the road and in front of the vehicle. These obstacles can be detected by the sensors equipped on the automated vehicle. The commonly used sensors include ultrasonic, LIDAR, radar, and camera [19]. The detection sensor can determine the distance between itself and the obstacle, as well as the relative speed between the two objects. Based on the information from the detection sensors, the collision-avoidance system will implement certain control strategy to decide if a certain braking system should be activated and thus avoid the collision with the obstacle.

In the collision-avoidance system control logic, one of the most important variables is the time-to-collision (TTC) [20]. Defining the speed of the obstacle as $v0$, the speed of the automated vehicle as $vA$, and the relative distance between these two objects as $ldis$ (see Fig. 1), the TTC at a specific time instant t is defined as
$TTC=ldis(t)v0(t)−vA(t)$
(1)
in which $ldis(t)$, $v0(t)$, and $vA(t)$ are, respectively, relative distance, speed of obstacle, and speed of automated vehicle at time t.

Note that the TTC is also affected by other vehicle physical parameters and environmental variables such as the friction coefficient between the vehicle and road surface, the road condition, and the weather. More sophisticated simulations or expressions need to be employed to compute the TTC if the physical parameters need to be considered. Since $vA(t)$ is changing over time and is affected by the dynamics of vehicle systems and controllers, vehicle dynamic simulations need to be employed to predict $vA(t)$ and thus compute the TTC over a simulation period. In this paper, the speed of the obstacle $v0$ is assumed to be constant in this short time period of collision-avoidance event. Since the speed of the obstacle at that short time period or time instant is just one realization of the random speed, the collision-avoidance reliability obtained in this paper is only the reliability at that short time period or time instant. The developed method is also applicable in the case that $v0$ is modeled as a time-series model. When the time-series model is used to represent $v0$, the correlation of the speeds between different time instants needs to be considered and the reliability becomes time-dependent. In that case, the time-dependent reliability analysis approaches [18,21,22] need to be employed to consider the correlations of speed over time.

Depending on the TTC at any time instant, the collision-avoidance system will take a specific action according to a predefined control logic. Figure 2 gives a schematic control logic for collision avoidance based on TTC.

As shown in Fig. 2, if $TTC>t1$, the automated vehicle is safe at this moment and it will continue with the normal driving using ACC. If $t2, the risk of collision is increasing and a warning will be given to the passengers. The system will let the passengers to make decision in this case. Otherwise, the AEB system will be active to avoid the collision. Even for the AEB system, there are also several braking stages, such as full brake and other braking scenarios [23]. For different companies, the control logic may be different. In this paper, an in-house-built ACC and AEB systems are used to illustrate the proposed collision-avoidance reliability analysis in Sec. 4. The developed framework is applicable to any automated vehicle collision simulation models, such as the commercial software carsim [24].

### Reliability of Collision Avoidance System.

The advanced driver assistant system with adaptive cruise control and autonomous emergency breaking is able to dynamically adjust the vehicle speed to avoid collision. The automated vehicle, however, still has a certain probability of collision with the obstacle due to various uncertainty sources in the traffic conditions and vehicle systems.

Defining the failure event as the relative distance ($ldis$) between the automated vehicle and the obstacle being less than zero (i.e., collision happens), the reliability of a collision-avoidance system is given by
$R=Pr{Ldis(X, β, τ)≥0, ∀τ∈[0, t]}$
(2)

where $Pr{⋅}$ stands for probability operator, $X$ is a vector of random variables (e.g., $V0$, $VA$, and $ldis$) in the collision-avoidance system, $β$ is a vector of controller parameters, and $[0, t]$ is a time period of interest.

Alternatively, Eq. (2) can be written as
$R=Pr{Lmin(X, β)≥0}$
(3)
where $Lmin=minτ∈[0, t](Ldis(X, β, τ))$ is the minimum distance between the vehicle and obstacle over the simulation time period. Evaluating the reliability of collision-avoidance system not only allows us to assess the collision risk in the early design stage but also provides a potential way to optimize the controller parameters using reliability-based design optimization approach as follows:
$min C(β)subject to:Pr{Ldts(X, β, τ)≥0, ∀τ∈[0, t]}≥R0βL≤β≤βU$
(4)
in which $C(β)$ is a cost function related to the controller parameters $β$ (e.g., fuel consumption, quantity related to user experience); $βL$ and $βU$ are, respectively, the lower and upper bounds of the controller parameters; and $R0$ is a specific reliability requirement for the collision-avoidance system.

The reliability-based optimization of the controller parameters (i.e., Eq. (4)) enables us to optimize the controller parameters in the early design stage with the consideration of reliability constraints. It can dramatically accelerate the development process of the autonomous vehicles. As an essential step in solving the optimization model, evaluating the reliability constraints (i.e., Eq. (3)), however, is not straightforward. The following two main challenges need to be addressed to accurately and efficiently solve Eq. (3):

• First, as inputs of Eq. (3), the random variables $X$ need to be accurately modeled based on the real-world traffic data. The modeling of these random variables requires statistical approaches that are able to accurately model both the marginal probability density function (PDF) of each random variable and the joint PDF of multiple variables by accounting for the complicated dependence between variables. How to accurately model the uncertainty sources in the collision-avoidance system based on data (which could be noisy) is the first challenge that needs to be solved.

• Second, collision-avoidance reliability needs to execute the collision-avoidance vehicle simulation numerous times (e.g., greater than 1 × 105) at different realizations of random variables [8]. This could be very computationally expensive if sophisticated computer simulation model (e.g., carsim [24] or Hardware-in-the-loop simulations) is employed. How to reduce the required computational/testing effort in collision-avoidance reliability analysis without sacrificing the accuracy is the second challenge that needs to be addressed.

Next, a collision-avoidance reliability analysis framework is developed to assess the reliability of the AV collision-avoidance system by addressing the above two challenges.

## Collision Avoidance Reliability Analysis Based on Adaptive Surrogate Modeling

There are two main elements in the developed collision-avoidance reliability analysis framework, namely: (1) uncertainty modeling and (2) reliability analysis. The uncertainty modeling element, which addresses the first challenge, aims to accurately represent the uncertainty in the traffic conditions such as vehicle speed and relative distance between the automated vehicle and the obstacle (see Fig. 1). Following that, the reliability analysis element evaluates the collision-avoidance reliability by addressing the second challenge. In the subsequent sections (i.e., Secs. 3.1 and 3.2), these two elements are explained in details.

### Uncertainty Modeling.

The objective of uncertainty modeling is to build a probabilistic model using data analytic methods to accurately represent both the marginal and joint PDFs of the random variables in the collision-avoidance system. In the past decades, various data-driven approaches have been developed to approximate the PDF of random variables based on data. The commonly used approaches can be roughly classified into two groups: parametric approach and nonparametric approach.

• Parametric approach: This group of approaches fit the data to a specific probability distribution, such as exponential, Gaussian, or Weibull distributions [2527]. For these specific distributions, methods are then developed to estimate the distribution parameters using maximum likelihood estimation method or Bayesian approach [2830].

• Nonparametric approach: Without assuming a certain distribution type, nonparametric approaches represent any arbitrary distribution shapes (e.g., multimodal distributions) purely based on data using kernel functions or mixture models. The commonly used nonparametric approaches include kernel density smoothing function [31,32], Gaussian mixture model (GMM) [33], and copula functions [34].

The above data-driven uncertainty modeling approaches [28,31,33] have their own advantages and disadvantages. For example, kernel density smoothing method can flexibly represent an arbitrary distribution based on data. However, it has difficulty in handling high-dimensional problems. The same issue exists for copula function-based methods. GMM is the most flexible method among the available methods. The use of Gaussian components in the mixture model, however, adds limitations to the GMM method. To overcome the disadvantages of different approaches in data-driven uncertainty modeling, a GMC method is developed recently in Ref. [35]. The GMC method seamlessly integrates kernel density function and GMM with copula function. It inherits the advantages of these three methods and performs better than all these three approaches for different data sets [35]. In this paper, the GMC method is employed to model the uncertainty sources in the collision-avoidance system based on data. Next, we briefly review the GMC method.

In Sklar's theorem [36], an n-dimensional distribution function, $F(x1, x2, …, xn)$ is connected with the marginal distributions, $F1(x1), …, Fn(xn)$ as follows:
$F(x1, x2, …, xn)=C(F1(x1), F2(x2), …, Fn(xn))=C(u1, u2, …, un)$
(5)

where $C(⋅)$ is a copula function and $Fi(xi)=ui$ is the cumulative density function (CDF) of the ith random variable $Xi, ∀i=1, …, n$, and n is the number of random variables. $C(u1, u2, …, un)$ is a unique function if $Fi(xi), ∀i=1, …, n$ are all continuous [34].

In the above equation, $C(u1, u2, …, un)$ is well studied only for few copula functions, such as Gaussian copula, student's t copula, and several other bivariate copulas [37]. In many practical applications, the current available copula functions, however, cannot accurately capture the complicated dependences between multiple variables. Inspired by the fact that kernel density function or GMM model can be employed to approximate an arbitrary probability distribution, a GMC model is developed in Ref. [35]. The GMC model preserves the characteristics of the copula function given in Eq. (5) (i.e., the copula function is independent from the marginal distribution). The marginal CDFs (i.e., $F1(x1), …, Fn(xn)$) are connected with $F(x1, x2, …, xn)$ in GMC through a generalized copula as follows:
$F(x)=C(u1, u2, …, un; θ)=Cnew(Φ−1(u1), Φ−1(u2), …, Φ−1(un); θ)$
(6)

where $Φ−1(⋅)$ is the inverse CDF of a standard normal variable, $Cnew(⋅, ⋅, …, ⋅; θ)$ is an unknown new copula function, and $θ$ is a vector of unknown parameters of the new copula function. Note that if a multivariate Gaussian distribution is employed for $Cnew(⋅)$, the new copula will become a Gaussian copula.

Based on Eq. (5), the joint PDF of random variables $Xi, ∀i=1, …, n$ is given by
$fX(x)=cnew(u1, u2, …, un; θ)fX1(x1)…fXn(xn)$
(7)
in which $cnew(u1, u2, …, un; θ)$ is the PDF of the new copula function and is given by
$cnew(u1, u2,…, un; θ)=∂nCnew(Φ−1(u1), Φ−1(u2), …, Φ−1(un); θ)∂Φ−1(u1)∂Φ−1(u2)⋯∂Φ−1(un)×1ϕ(Φ−1(u1))1ϕ(Φ−1(u2))⋯1ϕ(Φ−1(un))$
(8)

where $ϕ(⋅)$ is the PDF of a standard normal variable.

Defining $fnew(u1, u2, …, un; θ)=(∂nCnew(Φ−1(u1), Φ−1(u2), …, Φ−1(un); θ)/∂Φ−1(u1)∂Φ−1(u2)…∂Φ−1(un))$ and combining Eqs. (7) and (8), we have [35]
$fx(x)=cnew(u1,u2,…,un;θ)fx1(x1)⋅⋅⋅fxn(xn)=1ϕ(Φ−1(u1))⋅⋅⋅1ϕ(Φ−1(un))fnew(u1,u2,…,un;θ)fx1(x1)fx2(x2)⋅⋅⋅fxn(xn)$
(9)
in which $fnew(u1, u2, …, un; θ)$ is the PDF function of $Cnew(⋅)$.
Since $fnew(u1, u2, …, un; θ)$ is an unknown PDF function, a GMM model is then employed to approximate this unknown PDF in the transformed space based on data. Note that the data have been transformed into the standard normal space in Eq. (5). This is also the reason that the GMC model performs better than GMM. Based on the GMM representation, we have
$fnew(u1, u2,⋅⋅⋅, un; θ)≈∑i=1Kλiφ(z,μi,Σi)$
(10)

where $z=[Φ−1(u1), Φ−1(u2), …, Φ−1(un)]$, K is the number of Gaussian components, $λi$, $μi$, and $Σi$, which are parameters $θ$ of the new copula function, are, respectively, the weight, mean, and covariance of the ith Gaussian component.

As indicated in Eq. (10), we first transform the original data into the standard normal space. This step is similar to that of Gaussian copula and is accomplished through kernel density smoothing function. The GMC model is, therefore, applicable to uncertainty modeling with complicated marginal and joint PDFs. Based on the GMM approximation, we have the PDF of the new copula function as
$cnew(u1,u2,⋅⋅⋅, un;θ)≈1φ(Φ−1(u1)) … 1φ(Φ−1(un)) ∑i=1Kλiφ(z,μi,Σi)$
(11)
The joint PDF of random variables, $fx(x)$, is given by
$fx(x)≈1φ(Φ−1(u1))…1φ(Φ−1(un))∑i=1Kλiφ(z,μi,Σi)fX1(x1)fX2(x2)⋅⋅⋅fXn(xn).$
(12)

The generalized procedure for uncertainty modeling in collision-avoidance system using GMC can be summarized as below:

• (1)

Collect data of random variables in the collision-avoidance systems of automated vehicles.

• (2)

Approximate the marginal distribution of each random variable using kernel density smoothing function method [31,32].

• (3)

Convert the original data into the standard normal space using kernel density smoothing function and learn a GMM model in the transformed space;

• (4)

Integrate the GMM model with kernel density function to obtain the GMC model.

• (5)

Predict the joint PDF of any uncertainty realization using the learned GMC model.

In Sec. 3.2, we will discuss how to perform collision-avoidance reliability analysis based on the uncertainty modeling.

### Collision-Avoidance Reliability Analysis Based on Adaptive Surrogate Modeling Uncertainty Propagation.

A straightforward way of performing collision-avoidance reliability is to repeatedly run the collision-avoidance simulations or tests at different realizations of random variables using MCS method. The MCS method, however, is computationally prohibitive if high-fidelity simulation model is used or actual tests are performed. As mentioned previously, importance sampling approach has been employed in collision-avoidance analysis in Refs. [8] and [9]. But the required computational effort is still high due to the iterative procedure to determine the optimal proposal density function using the cross-entropy method [38].

Since the development of the first adaptive surrogate modeling-based reliability analysis method in 2008 by Bichon et al. [14], adaptive surrogate modeling has gained much attention in recent year in the reliability analysis field due to its promising potential in overcoming the drawbacks of traditional reliability analysis methods, such as the first-order reliability method [39] and second-order reliability method [40]. It has shown that the adaptive surrogate modeling-based reliability analysis method can dramatically improve the efficiency of reliability analysis while maintaining satisfactory accuracy [41].

The basic idea of adaptive surrogate modeling-based reliability analysis methods is to then adaptively refining the surrogate model in the limit state (i.e., failure boundary) region. The refining of the surrogate model nearing the limit state allows us to efficiently and accurately build a classifier to classify the random samples into fail or safe regions and thus reduce the required computational effort. Figure 3 gives a generalized flowchart of adaptive surrogate modeling-based reliability analysis method. As indicated in the figure, there are two main elements in the flowchart, namely surrogate modeling and learning function. The surrogate model is a data-driven model such as support vector machine [42], polynomial chaos expansion [43], and Gaussian process model [22], which is built using training points to replace the original model. The learning function is a function used to identify new training points near the limit state for the refinement of the surrogate model. In this paper, the adaptive Gaussian process surrogate modeling-based reliability analysis method is employed for the collision-avoidance reliability analysis. Next, we briefly review the Gaussian process surrogate modeling method and various learning functions. Based on that, we discuss the application of the method in the reliability analysis of the collision-avoidance system.

Let $Lmin$ be our quantity of interest in collision-avoidance reliability analysis (see Eq. (3)), for given controller parameters $β$ and a number of training points, Gaussian process surrogate model approximates $Lmin$ as below:
$Lmin(x,β)=g(x,α)+e(x)$
(13)
where $x$ is a vector of random variables in the collision-avoidance system as explained in Eq. (2), $g(x,α)$ is a trend function, $α$ is a vector of parameters of the trend function, and $e(x)$ is a Gaussian process with zero mean and covariance function given by
$Cov(e(x(i)),e(x(j)))=σe2ρ(x(i)−x(j);η)$
(14)
in which $σe2$ is the variance of the Gaussian process, $ρ(x(i)−x(j);η)$ is a correlation function or Kernel function, and $η$ are parameters of the correlation function.
In the surrogate model training phase, the hyper-parameters (i.e., $α$, $σe$, $η$) need to be estimated using maximum likelihood estimation method, least-square method, or Bayesian method [44]. After the parameters are estimated, for any given input setting $x$, the Gaussian process surrogate model predicts the response as
$L(x)∼N(L̂(x), σL2(x))$
(15)
in which $L̂(x)$ is the mean prediction and $σL2(x)$ is the variance of the prediction. More detailed descriptions of Gaussian process surrogate modeling are available in Ref. [44].
As indicated in Fig. 3, the surrogate model built using the initial training points may not accurately represent the original model. To refine the surrogate model near the limit state and thus reduce the required number of computer simulations/tests in reliability analysis, a learning function called the expected feasibility function (EFF) is proposed in Ref. [14] as below
$EFF(x)=L̂(x)[2Φ(−L̂(x)σL(x))−Φ(εL−L̂(x)σL(x))−Φ(εU−L̂(x)σL(x))]−[−Φ(εL−L̂(x)σL(x))−Φ(εU−L̂(x)σL(x))]−σL(x)[2φ(−L̂(x)σL(x))−φ(εL−L̂(x)σL(x))−φ(εU−L̂(x)σL(x))]$
(16)

where $εL=−2σL(x)$ and $εU=2σL(x)$ are user-specified error bounds.

Inspired by the EFF function [14], a much simpler U function is then proposed as [15]
$U(x)=|L̂(x)|σL(x)$
(17)
The EFF and U functions have been the most widely used learning functions. But they both have some drawbacks. For instance, it has been pointed out in Ref. [16] that both the EFF and U functions cannot account for the complicated correlations among the Gaussian process predictions when they are utilized to select the new training points. Inspired by the EFF and U functions and with a hope to further improve the efficiency of reliability analysis, various learning functions have been proposed in recent years. For example, the H function [45]
$H(x)=|−∫−εεfL(x)(l)ln(fL(x)(l))dl|$
(18)

where $fL(x)(l)$ is the PDF of the uncertain prediction.

The expected risk function (ERF) learning function [46,47]
$ERF(x)={ ∫0+∞lfL(x)(l)dl,if L̂(x)≤0 ∫−∞0−lfL(x)(l)dl,if L̂(x)≤0$
(19)
The maximum confidence enhancement (MCE) function is proposed by accounting for the joint PDF into the learning function [48]
$EIMCE(x)=(1−Φ(|L̂(x)|σL(x)))×fx(x)×σL(x)$
(20)
The expected utility (EU) function [49,50]
$EU(x)=fx(x)αtσL(x)11+αt2exp(−L̂2(x)2σL(x)1+αt2)$
(21)

where $αt$ is the exploration–exploitation tradeoff coefficient that measures the uncertainty in the prediction.

The above learning functions are derived from different perspectives of reliability analysis. For example, the EFF function evaluates how close is a new point to the failure boundary; the U function determines the probability of making a classification error; the H function is defined according to the Shannon information entropy theory; the ERF function measures the expected risk/probability of making a mistake on the prediction, which is similar to the U function; the MCE function accounts for the probability density function in the training points selection; and the EU function includes the exploration and exploitation into the learning function. In addition to these learning functions, efforts have been made to integrate adaptive surrogate modeling methods with importance sampling method [16,51], which further improve the efficiency of reliability analysis.

In the field of surrogate modeling-based reliability analysis, it has been shown that the U function performs better than the EFF function [15]. It is also observed that the fundamental idea of the H function is similar to that of the EFF function since they follow a similar derivation procedure [45]. In addition, the ERF, EU, and MCE functions are all similar to U function expect that EU and MCE functions have an additional term to account for the probability density function of input random variables. While EU function has a similar form as MCE, it has more parameters to tune and thus is more complicated for implementation. Based on these observations, the U function and the MCE learning function are employed in this paper as illustrative examples for the collision-avoidance reliability analysis and the convergence criterion given in Ref. [16] is employed to check the convergence. The convergence criterion is defined as $εmax<5%$, where $εmax$ is the maximum potential percentage error of failure probability estimate and is given by [16]
$εmax=maxNf∈[0,Ne]{N2−NfN1+Nf×100%}$
(22)

where $N1$ is the number of samples that are classified as fail and with U values (i.e., Eq. (17)) greater than 2, $N2$ is the number of samples that are classified as fail and with U values less than 2, and $Ne$ is the total number of samples that have U values less than 2.

The overall procedure is summarized as follows:

• Step 1: Generate initial training points for the random variables in the collision-avoidance system using Latin hypercube sampling (LHS) approach [52].

• Step 2: Run the collision-avoidance simulation and obtain the minimum distance ($Lmin$) as the quantity of interest.

• Step 3: Construct a surrogate model using Gaussian process surrogate modeling method.

• Step 4: Check the convergence using the criterion given in Eq. (22) [16]. If the criterion is satisfied, stop and obtain the reliability estimate. Otherwise, go to the next step.

• Step 5: Identify new training point using the U function (i.e., Eq. (17)) or MCE function (i.e., Eq. (20)), and go back to step 3 after performing a new collision-avoidance simulation at the new training point.

In Sec. 4, a case study is used to demonstrate the application of aforementioned uncertainty modeling and reliability analysis framework to collision-avoidance reliability analysis.

## Case Study

A collision-avoidance system equipped with ACC and AEB given in Ref. [53] (as shown in Fig. 4) is adopted as our case study to demonstrate the proposed collision-avoidance reliability analysis framework. The simulation model is a simulink model built according to the control logic presented in Fig. 5. Figure 4 presents an overview of the simulink model. For the purpose of illustration, the simulation model used in this case study is relatively computationally cheap. Each execution of the simulink model [53] takes about 0.2 s. This computationally cheap simulation model allows us to verify the effectiveness of the proposed collision-avoidance reliability analysis approach using the brute-force Monte Carlo simulation. The proposed approach, however, is not limited to the simulation model used in this case study. It is also applicable to actual vehicle tests in the lab or simulation models (e.g., carsim [24], hardware-in-the-loop simulations), which are much more sophisticated and computationally expensive than our in-house built simulation model (i.e., Fig. 4).

In the case study problem, the system has five-level control strategies, which are explained as below and depicted in Fig. 5.

• (1)

Level 1: If $TTC>t4$, the vehicle will drive normally under ACC. No braking action is required.

• (2)

Level 2: If $t3, the system will give alerts to the passengers and let the passenger to make decision.

• (3)

Level 3: If $t2, the vehicle will brake and the acceleration is −0.3g, where g is the gravitational constant.

• (4)

Level 4: If $t1, the vehicle will brake with an acceleration of −0.6g.

• (5)

Level 5: If $0 or the relative distance between the vehicle and the obstacle is less than 2 m, the vehicle will brake with an acceleration of −g.

In the above control strategies, the relationship between $ti$ and $ti+1$ is defined as $ti+1=2ti$. $t1$ is therefore a parameter that will affect the performance of the collision-avoidance system. The random variables in the system include the initial speed of the automated vehicle ($vA$), the speed of the obstacle ($v0$), and the relative distance between the vehicle and the obstacle ($Ldis$). Figure 6 presents the simulation results of one collision-avoidance simulation for a given realization of the random variables. Figure 6(a) gives the position of the vehicle and the obstacle with respect to the initial position over the simulation time period of interest. Figure 6(b) depicts the relative distance between these two objects over the simulation time period under the control strategy given in Fig. 5. It shows that the relative distance has a wavy nature due to the influence of the control logic depicted in Fig. 5. When the relative distance is too small, the vehicle will brake through AEB with a certain deceleration determined by the TTC (see Fig. 5). Due to the deceleration, the relative distance will increase. When the TTC meets the safety requirement, the vehicle will then accelerate to travel at a specific speed through ACC. We then perform reliability analysis based on this collision-avoidance simulation model using the methods presented in Sec. 3.

### Uncertainty Modeling of Traffic Conditions.

In order to model the uncertainty in the traffic conditions, we first extract data of $v0$, $vA$, and $Ldis$ from the Department of Transportation Safety Pilot Model Deployment database [54]. The data were collected in Ann Arbor, MI starting in August 2012 and contain various real-world vehicle operation information, such as vehicle speed, and relative distance between a vehicle and an object in front of the vehicle. It contains the naturalistic driving data of over 2800 vehicles equipped with wireless connected vehicle devices for more than 2 years. As of April 2015, 34.9 × 106 miles were logged [8]. Figure 7 gives a snapshot one of the data sets (i.e., DataFrontTargets) [54] from the Safety Pilot Model Deployment database. Even though the data have been collected over a long time period with a large number of vehicles, it may not fully account for all the uncertainty sources in the real-world driving environment. In this paper, it is assumed that the data presented in this database are representative of the uncertainty sources in the driving environment. Including other uncertainty sources such as weather, road condition, and road profile, in the autonomous vehicle reliability analysis is worth pursuing in future.

Based on the extracted driving data, we then build probability models for the random variables. Figure 8 shows the marginal PDFs of the three random variables in the collision-avoidance simulation model. Following that, Fig. 9 gives the joint PDF of different variables. As shown in Figs. 8 and 9, the distributions of vehicle speed and relative distance present multimodal properties. It implies that it is not appropriate to use parametric approaches to model these uncertainty sources.

### Collision-Avoidance Reliability Analysis.

Based on the uncertainty modeling, we then perform collision-avoidance reliability analysis using the adaptive surrogate modeling method discussed in Sec. 3.2. The Gaussian process surrogate model with zero-order trend function is employed as our surrogate model.

We first generate 200 initial training points for the surrogate modeling using the LHS approach. Using the 200 initial training points, we then perform leave-one-out cross validation for Kriging surrogate models with different types of correlation function. The cross-validation mean square errors for the surrogate models with the squared-exponential function (the most commonly used one), the exponential function, and cubic spline function are, respectively, 68.19, 14.33, and 50.73. In this paper, the exponential correlation function is therefore used in Eq. (14).

The initial surrogate model is then refined for the reliability analysis purpose using the learning functions given in Sec. 3.2. Figure 10 shows the convergence history of the failure probability estimate as a function of the number of collision-avoidance simulations obtained using the MCE learning function and the U function. Along with the results of adaptive surrogate modeling methods, in this figure, we also plot the failure probability estimate obtained by directly constructing Kriging surrogate model using different numbers of LHS samples. It shows that the adaptive surrogate modeling methods (U or MCS learning functions) are much more efficient than the direct surrogate modeling using LHS samples. For this particular example, the U function given in Eq. (17) performs much better than the MCE function given in Eq. (20)). We, therefore, employ the U function as our learning function for the surrogate model refinement in reliability analysis.

In addition, the figure shows that the proposed collision-avoidance reliability analysis framework only needs to evaluate the collision-avoidance simulation model 1037 times to get a very accurate estimate of the failure probability (i.e., 1—reliability). The MCS result in Fig. 10 is based on 2 × 106 samples. This demonstrates the effectiveness of the proposed framework in improving the accuracy and efficiency of collision-avoidance reliability analysis.

Figure 11 presents the collision failure probability as a function of the controller parameter t1. It indicates that the collision failure probability will decrease when t1 increases. Increasing the value of t1, however, will increase the frequency of unnecessary brake actions, which may cause uncomfortable feelings for the passenger and increase the fuel consumption. Based on the curve given in Fig. 11, we can determine an appropriate value for t1 by solving the optimization model given in Eq. (4). For example, a value of 0. 8 should be employed if we have a reliability requirement of 0.998.

### Discussion and Path Forward.

Even though the adaptive Kriging surrogate model drastically reduces the required number of simulations in collision-avoidance reliability analysis, the number of simulations (i.e., 1037) is still quite high. In the reliability analysis literature [1416], it is observed that the adaptive Kriging surrogate modeling-based reliability analysis methods usually can reduce the number of simulations/tests to a number close to or less than 100, for problems with three random variables. It implies that the performance of the adaptive Kriging approach in this paper is not as good as expected, even if it is much more efficient than current available methods for autonomous vehicle collision-avoidance reliability analysis [8,11]. In order to explain this phenomenon and provide guidance for future improvements, we plot the minimum relative distance response of this case study. Figures 1214 show the response function by, respectively, fixing one of the three input variables. We obtain the following two observations from the results of these figures: (1) the response is quite nonsmooth with almost discontinuous response in certain regions (see Fig. 13) and (2) the failure boundary or limit state (i.e., minimum relative distance = 0 m) is quite flat (see Figs. 12 and 14).

The almost discontinuous response in certain regions (i.e., the first observation) poses challenges to the construction of Kriging surrogate model. The reason is that the ordinary Kriging employed in this paper uses a fixed correlation function (i.e., kernel) for the whole design domain. When the response exhibits certain discontinuous characteristics, it is hard to determine an optimal correlation function that is universal to the whole design space. Even though the correlation function is not optimal, we can still construct a Kriging surrogate model. But the required number of function evaluations will be much higher than that for the response without discontinuous regions. The adaptive surrogate modeling approach used in this paper can reduce the influence of the discontinuity on the required effort for Kriging surrogate modeling, since we are only interested in the prediction accuracy near the failure boundary in the adaptive surrogate modeling method. This is also why the adaptive surrogate modeling is much more efficient than the direct surrogate modeling using space-filling LHS samples (as shown in Fig. 10). However, the adaptive surrogate modeling methods cannot completely eliminate the influence of the discontinuity due to the issue of using a fixed correlation function as been discussed previously. This is one of the reasons that the adaptive Kriging surrogate modeling method is not as good as expected. There are several possible ways to address the challenges caused by the discontinuity in the function response.

• First, the nonstationary Kriging surrogate modeling methods [55,56] can be employed to tackle the abrupt changes (i.e., discontinuity) in the function response. In nonstationary Kriging surrogate model, a nonstationary covariance function is used in the surrogate modeling rather than a fixed stationary correlation function. The nonstationary covariance function allows us to determine the best correlation structure for different local regions, and thus better capture the statistical relationship between the responses even if there are some discontinuous regions. This type of Kriging surrogate modeling method has been widely used in the geostatistics communities [57] to handle the discontinuity in the soil properties, and been introduced into engineering design field in recent years [58,59].

• Second, other types of surrogate modeling-based reliability analysis methods, which can directly handle the discontinuity in the function response, can be investigated for the autonomous vehicle reliability analysis. For instance, the support vector machine-based approach suggested in Ref. [60], and the neural network-based method [61].

For the second observation (i.e., the limit state is quite flat in certain regions), it requires a large number of training points to accurately perform reliability analysis, no matter what type of surrogate model is employed. This is also the main reason that the adaptive Kriging-based method is not as good as expected. This challenge can be addressed by combining the adaptive surrogate modeling method with importance sampling approach or subset simulation method. The integration of these methods will enable us to further concentrate on small regions, which are critical for reliability analysis, and thus further improve the efficiency of the collision-avoidance reliability analysis.

Given the previous observations and discussions, possible approaches that can further improve the efficiency of the autonomous vehicle collision-avoidance reliability analysis method presented in this paper are the syntheses of importance sampling approach or subset simulation method with adaptive sampling method and other advanced surrogate modeling techniques, such as nonstationary Kriging, support vector machine, and neural networks. It is also worth investigating which surrogate modeling method is the best among the suggested options. It should be noted that approaches have been investigated for adaptive surrogate modeling using nonstationary Kriging and support vector machine. But the integration of adaptive sampling strategy with neural networks has rarely been studied.

## Conclusions

Assuring the safety and reliability of autonomous vehicles is one of the important steps in the development of AVs. The collision-avoidance simulations or real-world tests need to be evaluated numerous times in the analysis. This brings significant challenges to the reliability assurance of AVs if computationally expensive simulation models are used or actual field vehicle tests are performed.

Using an in-house built collision-avoidance system as an example, this paper develops a collision-avoidance reliability analysis framework by integrating a recently developed uncertainty modeling approach with adaptive surrogate modeling-based reliability analysis method. A Gaussian copula model is employed in the uncertainty modeling step to accurately represent various uncertainty sources based on real-world traffic data. Based on the uncertainty modeling, the adaptive surrogate modeling method is used to improve the efficiency of reliability analysis without sacrificing the accuracy. Results of a case study show that the developed method can efficiently estimate the failure probability of a collision-avoidance system. The developed method focuses on collision-avoidance reliability analysis of car-following scenario. The developed approach can be extended to other types of collision-avoidance reliability analysis (e.g., lane changing or head-on collision) by replacing the simulation models and data with the corresponding models and data.

In addition to the uncertainty in the traffic conditions, there are also many sources of uncertainty in the physical parameters of AVs in the mass production. Considering the physical uncertainty sources in the collision-avoidance reliability analysis will be investigated in our future study. In this paper, the adaptive surrogate modeling with active learning function is employed for reliability analysis. Integrating the adaptive surrogate modeling with importance sampling or subset simulation can further improve the efficiency of collision- avoidance reliability analysis. In addition, the response function as shown in Figs. 1214 is quite highly nonlinear and even discontinuous in certain region. Replacing the ordinary Kriging model used in this paper with more advanced nonstationary Kriging models [55,56] or other types of surrogate models (e.g., support vector machine, neural networks) is also an interesting topic that worth pursuing.

## Acknowledgment

This research was partially supported by the Institute for Advanced Vehicle Systems (IAVS) seed funding at the University of Michigan-Dearborn for the third author, and by the Automotive Research Center (ARC) in accordance with Cooperative Agreement W56HZV-04-2-0001 U.S. Army Tank Automotive Research, Development and Engineering Center (TARDEC), Warren, MI, for the fourth author. The support is gratefully acknowledged.

## Funding Data

• Automotive Research Center (Award No. W56HZV-04-2-0001; Funder ID: 10.13039/100008192).

## References

References
1.
Fagnant
,
D. J.
, and
Kockelman
,
K.
,
2015
, “
Preparing a Nation for Autonomous Vehicles: Opportunities, Barriers and Policy Recommendations
,”
Transp. Res. Part A: Policy Pract.
,
77
, pp.
167
181
.
2.
Ioannou
,
P. A.
, and
Chien
,
C.-C.
,
1993
, “
Autonomous Intelligent Cruise Control
,”
IEEE Trans. Veh. Technol.
,
42
(
4
), pp.
657
672
.
3.
Stern
,
R. E.
,
Cui
,
S.
,
Delle Monache
,
M. L.
,
,
R.
,
Bunting
,
M.
,
Churchill
,
M.
,
Hamilton
,
N.
,
Haulcy
,
R.
,
Pohlmann
,
H.
,
Wu
,
F.
,
Piccoli
,
B.
,
Seibold
,
B.
,
Sprinkle
,
J.
, and
Work
,
D. B.
,
2018
, “
Dissipation of Stop-and-Go Waves Via Control of Autonomous Vehicles: Field Experiments
,”
Transp. Res. Part C: Emerging Technol.
,
89
, pp.
205
221
.
4.
Euro NCAP,
2015
, “
Euro, NCAP and Protocol, AEB VRU Test v1. 0.1
,” Euro, NCAP, Leuven, Belgium.
5.
Wu
,
C.
,
Peng
,
L.
,
Huang
,
Z.
,
Zhong
,
M.
, and
Chu
,
D.
,
2014
, “
A Method of Vehicle Motion Prediction and Collision Risk Assessment With a Simulated Vehicular Cyber Physical System
,”
Transp. Res. Part C: Emerging Technol.
,
47
, pp.
179
191
.
6.
Chang
,
Y.-Y.
,
Yu
,
C.-Y.
, and
Lee
,
L.-T.
,
2016
, “
Research of Vehicle-Mounted Detection and Collision Avoidance System Based on Dedicated Short Range Communications
,”
International Symposium on Computer, Consumer and Control
(
IS3C
), Xi'an, China, July 4–6, pp.
299
302
.
7.
Deng
,
C.
,
Wu
,
C.
,
Cao
,
S.
, and
Lyu
,
N.
, “
Modeling the Effect of Limited Sight Distance Through Fog on Car-Following Performance Using QN-ACTR Cognitive Architecture
,”
Transp. Res. Part F: Traffic Psychol. Behav.
(in press).
8.
Zhao
,
D.
,
Lam
,
H.
,
Peng
,
H.
,
Bao
,
S.
,
LeBlanc
,
D. J.
,
Nobukawa
,
K.
, and
Pan
,
C. S.
,
2017
, “
Accelerated Evaluation of Automated Vehicles Safety in Lane-Change Scenarios Based on Importance Sampling Techniques
,”
IEEE Trans. Intell. Transp. Syst.
,
18
(
3
), pp.
595
607
.
9.
Zhao
,
D.
,
Huang
,
X.
,
Peng
,
H.
,
Lam
,
H.
, and
LeBlanc
,
D. J.
,
2018
, “
Accelerated Evaluation of Automated Vehicles in Car-Following Maneuvers
,”
IEEE Trans. Intell. Transp. Syst.
,
19
(
3
), pp.
733
744
.
10.
Zhao
,
D.
,
Peng
,
H.
,
Bao
,
S.
,
Nobukawa
,
K.
,
LeBlanc
,
D.
, and
Pan
,
C.
,
2016
, “
Accelerated Evaluation of Automated Vehicles Using Extracted Naturalistic Driving Data
,”
,
18
(
3
), pp.
287
296
.
11.
Huang
,
Z.
,
Lam
,
H.
,
LeBlanc
,
D. J.
, and
Zhao
,
D.
,
2017
, “
Accelerated Evaluation of Automated Vehicles Using Piecewise Mixture Models
,”
IEEE Trans. Intell. Transp. Syst.
,
19
(
9
), pp.
2845
2855
.
12.
Xu
,
Y.
,
Wang
,
J.
,
Zou
,
Y.
, and
Sun
,
J.
,
2018
, “
Accelerated Testing for Automated Vehicles Safety Evaluation in Cut-In Scenarios Based on Importance Sampling, Genetic Algorithm and Simulation Applications
,”
J. Intell. Connected Veh.
,
1
(
1
), pp.
28
38
.
13.
Huang
,
Z.
,
Guo
,
Y.
,
Arief
,
M.
,
Lam
,
H.
, and
Zhao
,
D.
,
2018
, “
A Versatile Approach to Evaluating and Testing Automated Vehicles Based on Kernel Methods
,”
Annual American Control Conference
(
ACC
), Milwaukee, WI, June 27–29, pp.
4796
4802
.https://www.researchgate.net/publication/320180255_A_Versatile_Approach_to_Evaluating_and_Testing_Automated_Vehicles_based_on_Kernel_Methods
14.
Bichon
,
B. J.
,
Eldred
,
M. S.
,
Swiler
,
L. P.
,
,
S.
, and
McFarland
,
J. M.
,
2008
, “
Efficient Global Reliability Analysis for Nonlinear Implicit Performance Functions
,”
AIAA J.
,
46
(
10
), pp.
2459
2468
.
15.
Echard
,
B.
,
Gayton
,
N.
, and
Lemaire
,
M.
,
2011
, “
AK-MCS: An Active Learning Reliability Method Combining Kriging and Monte Carlo Simulation
,”
Struct. Saf.
,
33
(
2
), pp.
145
154
.
16.
Hu
,
Z.
, and
,
S.
,
2016
, “
Global Sensitivity Analysis-Enhanced Surrogate (GSAS) Modeling for Reliability Analysis
,”
Struct. Multidiscip. Optim.
,
53
(
3
), pp.
501
521
.
17.
Fauriat
,
W.
, and
Gayton
,
N.
,
2014
, “
AK-SYS: An Adaptation of the AK-MCS Method for System Reliability
,”
Reliab. Eng. Syst. Saf.
,
123
, pp.
137
144
.
18.
Hu
,
Z.
, and
,
S.
,
2016
, “
A Single-Loop Kriging Surrogate Modeling for Time-Dependent Reliability Analysis
,”
ASME J. Mech. Des.
,
138
(
6
), p.
061406
.
19.
Levinson
,
J.
,
,
J.
,
Becker
,
J.
,
Dolson
,
J.
,
Held
,
D.
,
Kammel
,
S.
,
Kolter
,
J. Z.
,
Langer
,
D.
,
Pink
,
O.
,
Pratt
,
V.
, and
Sokolsky
,
M.
,
2011
, “
Towards Fully Autonomous Driving: Systems and Algorithms
,”
Intelligent Vehicles Symposium
(
IV
163
168
.
20.
Lee
,
D. N.
,
1976
, “
A Theory of Visual Control of Braking Based on Information About Time-to-Collision
,”
Perception
,
5
(
4
), pp.
437
459
.
21.
Hu
,
Z.
, and
Du
,
X.
,
2013
, “
Time-Dependent Reliability Analysis With Joint Upcrossing Rates
,”
Struct. Multidiscip. Optim.
,
48
(
5
), pp.
893
907
.
22.
Hu
,
Z.
, and
Du
,
X.
,
2015
, “
Mixed Efficient Global Optimization for Time-Dependent Reliability Analysis
,”
ASME J. Mech. Des.
,
137
(
5
), p.
051401
.
23.
Kaempchen
,
N.
,
Schiele
,
B.
, and
Dietmayer
,
K.
,
2009
, “
Situation Assessment of an Autonomous Emergency Brake for Arbitrary Vehicle-to-Vehicle Collision Scenarios
,”
IEEE Trans. Intell. Transp. Syst.
,
10
(
4
), pp.
678
687
.
24.
Benekohal
,
R.
, and
Treiterer
,
J.
,
1988
, “
CarSim: Car-Following Model for Simulation of Traffic in Normal and Stop-and-Go Conditions
,”
Transp. Res. Record
,
1194
, pp.
99
111
.
25.
Seguro
,
J.
, and
Lambert
,
T.
,
2000
, “
Modern Estimation of the Parameters of the Weibull Wind Speed Distribution for Wind Energy Analysis
,”
J. Wind Eng. Ind. Aerodyn.
,
85
(
1
), pp.
75
84
.
26.
Cohen
,
A. C.
,
1965
, “
Maximum Likelihood Estimation in the Weibull Distribution Based on Complete and on Censored Samples
,”
Technometrics
,
7
(
4
), pp.
579
588
.
27.
Lai
,
C.
,
Xie
,
M.
, and
Murthy
,
D.
,
2003
, “
A Modified Weibull Distribution
,”
IEEE Trans. Reliab.
,
52
(
1
), pp.
33
37
.
28.
Coit
,
D. W.
, and
Jin
,
T.
,
2000
, “
Gamma Distribution Parameter Estimation for Field Reliability Data With Missing Failure Times
,”
IIE Trans.
,
32
(
12
), pp.
1161
1166
.
29.
Aban
,
I. B.
,
Meerschaert
,
M. M.
, and
Panorska
,
A. K.
,
2006
, “
Parameter Estimation for the Truncated Pareto Distribution
,”
J. Am. Stat. Assoc.
,
101
(
473
), pp.
270
277
.
30.
Hu
,
Z.
,
,
S.
, and
Du
,
X.
,
2016
, “
Uncertainty Quantification of Time-Dependent Reliability Analysis in the Presence of Parametric Uncertainty
,”
ASCE-ASME J. Risk Uncertainty Part B
,
2
(
3
), p.
031005
.
31.
Wand
,
M. P.
, and
Jones
,
M. C.
,
1993
, “
Comparison of Smoothing Parameterizations in Bivariate Kernel Density Estimation
,”
J. Am. Stat. Assoc.
,
88
(
422
), pp.
520
528
.
32.
Jones
,
M. C.
,
Marron
,
J. S.
, and
Sheather
,
S. J.
,
1996
, “
Progress in Data-Based Bandwidth Selection for Kernel Density Estimation
,”
Comput. Stat.
,
11
(
3
), pp.
337
381
.https://pdfs.semanticscholar.org/e06c/eca548d9984008959afcfc46802a731df472.pdf
33.
Hu
,
Z.
, and
,
S.
,
2018
, “
Bayesian Network Learning for Data-Driven Design
,”
ASCE-ASME J. Risk Uncertainty Part B
,
4
(
4
), p.
041002
.
34.
Hu
,
Z.
, and
,
S.
,
2016
, “
Time-Dependent Reliability Analysis Using a Vine-Arma Load Model
,”
ASCE-ASME J. Risk Uncertainty Part B
,
3
(
1
), p.
011007
.
35.
Hu
,
Z.
, and
,
S.
,
2019
, “
Probability Models for Data-Driven Global Sensitivity Analysis
,”
Reliab. Eng. Syst. Saf.
, (in press).
36.
Sklar
,
A.
,
1973
, “
Random Variables, Joint Distribution Functions, and Copulas
,”
Kybernetika
,
9
(
6
), pp.
449
460
.https://dml.cz/bitstream/handle/10338.dmlcz/125838/Kybernetika_09-1973-6_2.pdf
37.
Embrechts
,
P.
,
Lindskog
,
F.
, and
McNeil
,
A.
,
2001
,
Modelling Dependence With Copulas, Rapport Technique
,
Departement de Mathematiques, Institut Federal de Technologie de Zurich
,
Zurich, Switzerland
.
38.
Zhang
,
W.
,
Wang
,
H.
,
Hartmann
,
C.
,
Weber
,
M.
, and
Schutte
,
C.
,
2014
, “
Applications of the Cross-Entropy Method to Importance Sampling and Optimal Control of Diffusions
,”
SIAM J. Sci. Comput.
,
36
(
6
), pp.
A2654
A2672
.
39.
Hu
,
Z.
, and
Du
,
X.
,
2015
, “
A Random Field Approach to Reliability Analysis With Random and Interval Variables
,”
ASCE-ASME J. Risk Uncertainty Part B
,
1
(
4
), p.
041005
.
40.
Zhang
,
J.
, and
Du
,
X.
,
2010
, “
A Second-Order Reliability Method With First-Order Efficiency
,”
ASME J. Mech. Des.
,
132
(
10
), p.
101006
.
41.
Zhu
,
Z.
, and
Du
,
X.
,
2016
, “
Reliability Analysis With Monte Carlo Simulation and Dependent Kriging Predictions
,”
ASME J. Mech. Des.
,
138
(
12
), p.
121403
.
42.
Basudhar
,
A.
,
Missoum
,
S.
, and
Sanchez
,
A. H.
,
2008
, “
Limit State Function Identification Using Support Vector Machines for Discontinuous Responses and Disjoint Failure Domains
,”
Probab. Eng. Mech.
,
23
(
1
), pp.
1
11
.
43.
Xiu
,
D.
, and
,
G. E.
,
2002
, “
The Wiener-Askey Polynomial Chaos for Stochastic Differential Equations
,”
SIAM J. Sci. Comput.
,
24
(
2
), pp.
619
644
.
44.
Rasmussen
,
C. E.
,
2004
, “
Gaussian Processes in Machine Learning
,”
,
Springer
, Cambridge, MA, pp.
63
71
.
45.
Lv
,
Z.
,
Lu
,
Z.
, and
Wang
,
P.
,
2015
, “
A New Learning Function for Kriging and Its Applications to Solve Reliability Problems in Engineering
,”
Comput. Math. Appl.
,
70
(
5
), pp.
1182
1197
.
46.
Yang
,
X.
,
Liu
,
Y.
,
Gao
,
Y.
,
Zhang
,
Y.
, and
Gao
,
Z.
,
2015
, “
An Active Learning Kriging Model for Hybrid Reliability Analysis With Both Random and Interval Variables
,”
Struct. Multidiscip. Optim.
,
51
(
5
), pp.
1003
1016
.
47.
Yang
,
X.
,
Liu
,
Y.
,
Zhang
,
Y.
, and
Yue
,
Z.
,
2015
, “
Probability and Convex Set Hybrid Reliability Analysis Based on Active Learning Kriging Model
,”
Appl. Math. Modell.
,
39
(
14
), pp.
3954
3971
.
48.
Wang
,
Z.
, and
Wang
,
P.
,
2013
, “
A Maximum Confidence Enhancement Based Sequential Sampling Scheme for Simulation-Based Design
,”
ASME J. Mech. Des.
,
136
(
2
), p.
021006
.
49.
,
M. K.
,
Hu
,
C.
,
MacKenzie
,
C. A.
,
Eshghi
,
A. T.
, and
Lee
,
S.
,
2018
, “
Sequential Exploration-Exploitation With Dynamic Trade-Off for Efficient Reliability Analysis of Complex Engineered Systems
,”
Struct. Multidiscip. Optim.
,
57
(
1
), pp.
235
250
.
50.
,
M. K.
,
Li
,
M.
,
Hu
,
C.
,
MacKenzie
,
C. A.
,
Lee
,
S.
, and
Eshghi
,
A. T.
,
2018
, “
A High-Dimensional Reliability Analysis Method for Simulation-Based Design Under Uncertainty
,”
ASME J. Mech. Des.
,
140
(
7
), p.
071401
.
51.
Dubourg
,
V.
,
Sudret
,
B.
, and
Deheeger
,
F.
,
2013
, “
Metamodel-Based Importance Sampling for Structural Reliability Analysis
,”
Probab. Eng. Mech.
,
33
, pp.
47
57
.
52.
Helton
,
J. C.
, and
Davis
,
F. J.
,
2003
, “
Latin Hypercube Sampling and the Propagation of Uncertainty in Analyses of Complex Systems
,”
Reliab. Eng. Syst. Saf.
,
81
(
1
), pp.
23
69
.
53.
Liu
,
Y.
,
Zhao
,
Y.
,
Dale
,
R.
, and
Li
,
X.
,
2017
, “
Development of Test Bench for Autonomous Emergency Braking System
,” University of Michigan-Dearborn, Dearborn, MI.
54.
Bezzina
,
D.
, and
Sayer
,
J.
,
2014
, “
Safety Pilot Model Deployment: Test Conductor Team Report
,” National Highway Traffic Safety Administration, The Intelligent Transportation Systems Joint Program Office, Washington, DC, Report No. DOT HS 812 171.
55.
Paciorek
,
C. J.
, and
Schervish
,
M. J.
,
2006
, “
Spatial Modelling Using a New Class of Nonstationary Covariance Functions
,”
Environmetrics
,
17
(
5
), pp.
483
506
.
56.
Kim
,
H.-M.
,
Mallick
,
B. K.
, and
Holmes
,
C. C.
,
2005
, “
Analyzing Nonstationary Spatial Data Using Piecewise Gaussian Processes
,”
J. Am. Stat. Assoc.
,
100
(
470
), pp.
653
668
.
57.
Meyer
,
T. H.
,
2004
, “
The Discontinuous Nature of Kriging Interpolation for Digital Terrain Modeling
,”
Cartography Geogr. Inf. Sci.
,
31
(
4
), pp.
209
216
.
58.
Clark
,
D. L.
, Jr.
,
Bae
,
H.-R.
,
Gobal
,
K.
, and
Penmetsa
,
R.
,
2016
, “
Engineering Design Exploration Using Locally Optimized Covariance Kriging
,”
AIAA J.
,
54
(
10
), pp.
3160
3175
.
59.
Xiong
,
Y.
,
Chen
,
W.
,
Apley
,
D.
, and
Ding
,
X.
,
2007
, “
A Non-Stationary Covariance-Based Kriging Method for Metamodelling in Engineering Design
,”
Int. J. Numer. Methods Eng.
,
71
(
6
), pp.
733
756
.
60.
Devathi
,
H.
,
Hu
,
Z.
, and
,
S.
,
2016
, “
Snap-Through Buckling Reliability Analysis Under Spatiotemporal Variability and Epistemic Uncertainty
,”
AIAA J.
,
54
(
12
), pp.
3981
3993
.
61.
,
J. E.
, and
Alvarez
,
D. A.
,
2001
, “
Neural-Network-Based Reliability Analysis: A Comparative Study
,”
Comput. Methods Appl. Mech. Eng.
,
191
(
1–2
), pp.
113
132
.