## Abstract

Model management is an essential component in data-driven surrogate-assisted evolutionary optimization. In model management, the solutions with a large degree of uncertainty in approximation play an important role. They can strengthen the exploration ability of algorithms and improve the accuracy of surrogates. However, there is no theoretical method to measure the uncertainty of prediction of Non-Gaussian process surrogates. To address this issue, this article proposes a method to measure the uncertainty. In this method, a stationary random field with a known zero mean is used to measure the uncertainty of prediction of Non-Gaussian process surrogates. Based on experimental analyses, this method is able to measure the uncertainty of prediction of Non-Gaussian process surrogates. The method's effectiveness is demonstrated on a set of benchmark problems in single surrogate and ensemble surrogates cases.

## 1 Introduction

Data-driven optimization problems usually involve objective and constraint functions that are not available, and the evaluation of these functions is time-consuming and complex. There are only small data from physical experiments, numerical simulations, or daily life, and the evaluation of these functions involves a number of computationally expensive numerical simulations or costly physical experiments (Preen and Bull, 2016; Wang et al., 2016; Jin et al., 2018).

Evolutionary algorithms (EAs) are population-based search methods that mimic natural biological evolution and species' social behavior. They are promising in solving non-convex, constrained, multiobjective, or dynamic problems (Michalewicz and Schoenauer, 1996; Hart et al., 1998; Li et al., 2014; Zhang, Mei et al., 2021). However, most existing research on EAs usually assumes that the analytic objective and constraint functions are available, and evaluating these functions is cheap and simple. Therefore, EAs cannot be directly used to solve the data-driven optimization problems. Surrogate-assisted evolutionary algorithms (SAEAs) are considered to address the limitation of EAs in solving these problems (Jin et al., 2000; Tong et al., 2019; Zhang, Li et al., 2021; Wang et al., 2022). In SAEAs, many machine learning models can be used as surrogates to approximate the exact functions, including polynomial regression (PR), Gaussian process (GP), artificial neural network (ANN), radial basis function network (RBFN), support vector machine (SVM), and the ensemble of these surrogates. A limited number of exact function evaluations are carried out, and a small amount of data is used to train these surrogates (Braun et al., 2009; Jin et al., 2000; Chugh et al., 2019).

For all surrogates mentioned above, GP is usually used (Emmerich et al., 2006; Coelho and Bouillard, 2011; Chugh et al., 2016; Zhan and Xing, 2021). There is provided prediction and uncertainty information by GP, which is important in SAEAs. Then the existing infill sampling criteria can be used to guide the search of EAs and the update of surrogates, such as the lower confidence bound (LCB) (Torczon and Trosset, 1998), the expected improvement (EI) (Jones et al., 1998) and the probability of improvement (PoI) (Ulmer et al., 2003). On the contrary, although many Non-Gaussian process (Non-GP) surrogates can also provide a good prediction, they cannot provide the uncertainty of prediction of surrogates. In this case, these Non-GP surrogates have significant limitations: (1) Because there is no uncertainty information of prediction of surrogates, it is hard to improve the exploration of EAs and the accuracy of surrogates; (2) The existing infill sampling criteria cannot be used to guide the search of EAs.

It should be emphasized that the uncertainty information of prediction of surrogates plays an essential role in model management in SAEAs, because (1) solutions with a large degree of uncertainty indicate that the fitness landscape around them has not been well explored, and therefore the evaluation of these solutions is likely to find a better solution (Branke and Schmidt, 2005); (2) evaluating these solutions can most effectively improve the accuracy of surrogates (Jin, 2011).

Several methods are used to measure the uncertainty of prediction of Non-GP surrogates. For instance, Bayesian neural networks can measure the uncertainty of prediction of neural networks (Gal and Ghahramani, 2015). Cross-validation also can be used to measure the uncertainty of prediction of surrogates (Hutter et al., 2019). However, there is a significant limitation for the two methods: the accuracy of uncertainty highly depends on the size of training data. However, there is not much training data in data-driven optimization progress. Besides, the prior distribution also needs to be known for Bayesian Neural Networks. Based on the limitation, the two methods will not be investigated in the article.

In addition to the above methods, there are also three typical methods: (1) The distance from the solutions to the existing training data has been used as an uncertainty measure in Branke and Schmidt (2005). Since ensemble surrogates have been proven to provide uncertainty information, two methods have been proposed to measure the uncertainty of prediction of ensemble surrogates. (2) The literature (Wang et al., 2017) defined the uncertainty measurement to be the maximum difference between outputs of ensemble members. (3) The variance of predictions output by the base surrogates of ensemble is used to estimate the uncertainty of prediction of ensemble surrogates (Guo et al., 2018).

Among the three methods above, the first method is a qualitative uncertainty measurement method. In theory, it is not able to accurately measure the uncertainty of prediction of surrogates. Instead, it indicates only the crowded degree of the neighborhood of a solution. The second method is the disagreement among the outputs of ensemble members for the prediction of surrogates. This method was proposed based on Query-by-Committee (QBC) in active learning, which shows that the query with the maximum disagreement strategy can efficiently enhance the accuracy of surrogates (Wang et al., 2017). In essence, this method describes the difference of predictions among ensemble members in a solution. In the third method, the uncertainty of prediction of surrogates is defined by the variance of predictions output by the base surrogates of ensemble. It indicates the average squared deviation of the base surrogates about the output of ensemble. In the probability and statistic viewpoint, these methods for measuring the uncertainty of prediction of Non-GP surrogates are not a sound method. These methods mentioned above cannot address one important issue: to measure the uncertainty of prediction of Non-GP surrogates. Therefore, it can be confirmed that there is no a theoretical sound method to measure the uncertainty of prediction of Non-GP surrogates.

To address the issue mentioned above, this article proposes an uncertainty measure for the prediction (UMP) of Non-GP surrogates. This method can be written in the form of a random field model. In detail, it consists of two components: regression function (namely Non-GP surrogate) and residual variation (also known as uncertainty). In this method, two components are uncorrelated. In the first term, Non-GP surrogate as regression function only depends on decision variables, and the second term represents the uncertainty of prediction of Non-GP surrogate based on a stationary random field. Thus, based on the random field model, the uncertainty of prediction of Non-GP surrogate can be measured. Then, the existing infill sampling criteria can be used to guide the search of algorithms and the update of surrogates.

In this article, an uncertainty measure for the prediction of Non-GP Surrogates is proposed to overcome the drawbacks of existing uncertainty methods. The main contribution of this article can be summarized as follows:

(1) An uncertainty measure for the prediction of Non-GP surrogates is proposed, which overcomes the drawbacks of existing uncertainty methods;

(2) The effectiveness of the proposed method is investigated on a set of benchmark problems and analysed on Rastrigin function in both single surrogate and ensemble surrogates cases. The experimental results demonstrate that the proposed method is promising in solving data-driven optimization problems.

The rest of this article is structured as follows. Section 2 presents a brief review of the used surrogates, ensemble surrogates, and infill sampling criterion in this article. Section 3 presents the proposed uncertainty measure for the prediction of Non-GP surrogates. Section 4 demonstrates and discusses experimental results. Finally, Section 5 concludes the article with a summary and looks into the future work.

## 2 Related Work

Surrogates and infill sampling criteria are essential components in online surrogate-assisted evolutionary algorithms. This section presents a brief review of surrogates and the infill sampling criterion involved in this article.

### 2.1 Polynomial Regression

### 2.2 Radial Basis Function Network

### 2.3 Support Vector Machine

### 2.4 Ensemble Surrogates

### 2.5 Lower Confidence Bound

## 3 Uncertainty Measure for Prediction of Non-GP Surrogates

We aim to address the issue that there is no theoretical method to measure the uncertainty of prediction of Non-GP surrogates. Hence, an uncertainty measure for prediction of Non-GP surrogates is proposed in this article. This method can be written in the form of random field model. In detail, it consists of two components: regression function (namely Non-GP surrogate) and residual variation (the uncertainty of prediction of Non-GP surrogate). In this method, the two components are uncorrelated. In the first term, Non-GP surrogate as regression function only depends on decision variables, and the second term represents the uncertainty of prediction of Non-GP surrogate based on a stationary random field.

### 3.1 Formulation for Uncertainty Measure for Prediction

#### 3.1.1 Hyperparameter Estimation

#### 3.1.2 Prediction Distribution

### 3.2 Instantiation of UMP Framework

In UMP, any Non-GP surrogate can be considered to be the first term of the UMP. In this article, the first term will be instantiated with RBFN, QP, and SVM as a surrogate, respectively.

#### 3.2.1 UMP with RBFN

#### 3.2.2 UMP with Quadratic Polynomial

#### 3.2.3 UMP with SVM

### 3.3 Workflow of UMP

The pseudocode for the workflow of UMP is presented in the Algorithm 1. Initially, $11d-1$ samples in the search space are generated using Latin hypercube sampling (LHS) (Stein, 1987) and evaluated by exact functions. Then these samples are archived in an initial database. $\tau $ latest samples in the database are selected as training data to train the Non-GP surrogate. The Non-GP surrogate replace the exact functions in evolving a population of $NP$ individuals for $T$ generations with a DE. Then a potential candidate solution is selected in the population by using LCB and evaluated by exact functions. After that, the solution is added to the database. Finally, when the computational budget is exhausted, the best solution in the database is chosen as the output.

## 4 Results and Discussion

To investigate the performance of the proposed UMP, a set of experiments is carried out in both single and ensemble surrogates by Algorithm 1, respectively. The Non-GP surrogates involving RBFN, QP, and SVM are considered in this article.

For the single surrogate, two experiments are carried out. First, the experiment compares Non-GP surrogates with and without UMP, and they are named UMP/RBFN, UMP/QP, UMP/SVM, RBFN, QP, and SVM, respectively. Second, the UMP compares with the existing uncertainty method in Branke and Schmidt (2005), which is the distance from the solutions to the existing training data (DUM). The three algorithms with the UMP are named UMP/RBFN, UMP/QP, and UMP/SVM, and the compared algorithms are named DUM/RBFN, DUM/QP, and DUM/SVM.

Regarding ensemble surrogates, the proposed UMP compares with method $Uens$ which is the maximum difference between the outputs of the ensemble members (Wang et al., 2017) and VUM which is variance of predictions output by the base surrogates of the ensemble (Guo et al., 2018), respectively. In this article, the ensemble surrogates consist of three surrogates: RBFN, QP, and SVM. These algorithms with the proposed method and two compared methods are named UMP/ensemble, $Uens$/ensemble, and VUM/ensemble, respectively.

### 4.1 Parameter Settings

There are several parameters in experiments. The setting of these parameters is given below.

(1) The computational budget with exact function evaluations $FEs=100$ was performed in this article, based on the assumption that the optimization algorithm is only allowed to evaluate a small number of candidate solutions during optimization. The number of the run was 25.

(2) DE parameters: DE/rand/1/bin was employed in this article. The evolution generations $T=100$, population size $NP=20$, scaling factor $F=0.5$, and the crossover rate $CR=0.9$.

(3) Initial samples $11d-1$ were randomly generated by LHS.

(4) The range of values for parameters **$\theta $** was $[1.0e-6,20]$.

(5) Training data $\tau =50$ for dimension $d=2$, $\tau =11d-1$ for $d=5,10$ was considered. $\tau $ training data in the database was selected under considering both the quality and the computational cost of Non-GP surrogate.

### 4.2 Test Problems

The effectiveness of the proposed method is verified on benchmark problems CEC 2014 (Liu et al., 2014) with 2, 5, and 10 dimensions. The benchmark problems are listed in Table 1.

Problem . | Objective function name . | $f*$ . | Property . |
---|---|---|---|

$F1$ | Shifted Sphere | 0 | Unimodal |

$F2$ | Shifted Ellipsoid | 0 | Unimodal |

$F3$ | Shifted and Rotated Ellipsoid | 0 | Unimodal |

$F4$ | Shifted Step | 0 | Unimodal, Discontinuous |

$F5$ | Shifted Ackley | 0 | Multi-modal |

$F6$ | Shifted Griewank | 0 | Multi-modal |

$F7$ | Shifted and Rotated Rosenbrock | 0 | Multi-modal with very narrow valley |

$F8$ | Shifted and Rotated Rastrigin | 0 | Very complicated multi-modal |

Problem . | Objective function name . | $f*$ . | Property . |
---|---|---|---|

$F1$ | Shifted Sphere | 0 | Unimodal |

$F2$ | Shifted Ellipsoid | 0 | Unimodal |

$F3$ | Shifted and Rotated Ellipsoid | 0 | Unimodal |

$F4$ | Shifted Step | 0 | Unimodal, Discontinuous |

$F5$ | Shifted Ackley | 0 | Multi-modal |

$F6$ | Shifted Griewank | 0 | Multi-modal |

$F7$ | Shifted and Rotated Rosenbrock | 0 | Multi-modal with very narrow valley |

$F8$ | Shifted and Rotated Rastrigin | 0 | Very complicated multi-modal |

### 4.3 Comparison on Single Non-GP Surrogate

#### 4.3.1 Effect of the UMP

Problem . | d . | UMP/RBFN . | RBFN . | UMP/QP . | QP . | UMP/SVM . | SVM . |
---|---|---|---|---|---|---|---|

F1 | 2 | 9.8806E-3$\xb1$ 0.0156 | 0.0642 $\xb1$ 0.1018 | 1.4462E-4$\xb1$ 2.4305E-4 | 2.8772 $\xb1$ 2.8031 | 6.0986E-4$\xb1$ 7.6748E-4 | 0.8707 $\xb1$ 1.5511 |

5 | 0.0331$\xb1$ 0.0328 | 0.2288 $\xb1$ 0.1966 | 0.7640$\xb1$ 0.7112 | 54.0371 $\xb1$ 23.7337 | 0.0264$\xb1$ 0.0169 | 25.1719 $\xb1$ 16.9316 | |

10 | 1.9803$\xb1$ 1.0066 | 9.6927 $\xb1$ 5.0127 | 67.9611$\xb1$ 21.5325 | 287.0700 $\xb1$ 76.2774 | 21.5962$\xb1$ 11.4967 | 87.2006 $\xb1$ 37.7374 | |

F2 | 2 | 0.0182$\xb1$ 0.0342 | 0.2342 $\xb1$ 0.5029 | 3.0383E-4$\xb1$ 2.5810E-4 | 2.8759 $\xb1$ 2.6865 | 1.0445E-3$\xb1$ 1.3433E-3 | 5.2747 $\xb1$ 8.8904 |

5 | 0.4057$\xb1$ 0.3093 | 2.3933 $\xb1$ 1.7200 | 14.0614$\xb1$ 19.1254 | 155.4160 $\xb1$ 68.8183 | 0.0799$\xb1$ 0.0468 | 173.4310 $\xb1$ 82.1052 | |

10 | 32.4434$\xb1$ 16.5484 | 129.7015 $\xb1$ 51.3120 | 307.0399$\xb1$ 97.6482 | 1425.5204 $\xb1$ 393.2456 | 109.8823$\xb1$ 53.2732 | 555.1530 $\xb1$ 268.1075 | |

F3 | 2 | 0.0437$\xb1$ 0.0915 | 1.2991 $\xb1$ 1.8210 | 9.9812E-3$\xb1$ 0.0320 | 1.3025 $\xb1$ 1.2417 | 0.0011$\xb1$ 0.0010 | 5.4055 $\xb1$ 7.0946 |

5 | 0.4611$\xb1$ 0.4911 | 3.2056 $\xb1$ 4.6624 | 5.6759$\xb1$ 5.0457 | 39.0094 $\xb1$ 21.5417 | 0.0307$\xb1$ 0.0185 | 38.9898 $\xb1$ 25.2550 | |

10 | 243.0518$\xb1$ 84.7628 | 823.7360 $\xb1$ 378.8218 | 477.9377$\xb1$ 182.8839 | 1921.1620 $\xb1$ 503.9823 | 559.6709$\xb1$ 316.1527 | 2451.7201 $\xb1$ 795.8515 | |

F4 | 2 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 2.2400 $\xb1$ 2.1029 | 0.0$\xb1$ 0.0 | 0.6800 $\xb1$ 0.6144 |

5 | 0.1200$\xb1$ 0.3249 | 0.2000 $\xb1$ 0.4000 | 4.4000$\xb1$ 4.9799 | 49.0000 $\xb1$ 24.5715 | 0.7600$\xb1$ 1.0688 | 26.2000 $\xb1$ 24.6398 | |

10 | 2.5200$\xb1$ 1.0998 | 9.5200 $\xb1$ 4.3185 | 65.2000$\xb1$ 22.7385 | 230.8000 $\xb1$ 77.1305 | 23.5200$\xb1$ 10.9986 | 76.1600 $\xb1$ 34.7754 | |

F5 | 2 | 2.7644$\xb1$ 1.0851 | 10.4969 $\xb1$ 4.7438 | 2.7476$\xb1$ 0.8620 | 8.0343 $\xb1$ 2.2641 | 1.8426$\xb1$ 0.6829 | 2.2079 $\xb1$ 1.1024 |

5 | 3.6208$\xb1$ 0.7392 | 12.9774 $\xb1$ 1.5847 | 4.0928$\xb1$ 0.912 | 16.1825 $\xb1$ 1.9481 | 3.5112$\xb1$ 0.5640 | 6.3787 $\xb1$ 2.4737 | |

10 | 4.2216$\xb1$ 0.3694 | 13.2189 $\xb1$ 1.1633 | 10.6836$\xb1$ 6.2570 | 18.4682 $\xb1$ 1.0169 | 3.7534$\xb1$ 0.3806 | 9.8341 $\xb1$ 1.4496 | |

F6 | 2 | 0.1410$\xb1$ 0.0035 | 0.1600 $\xb1$ 0.3545 | 0.4851$\xb1$ 0.2165 | 1.1618 $\xb1$ 0.7925 | 0.0602$\xb1$ 0.0401 | 0.1971 $\xb1$ 0.1836 |

5 | 0.6092$\xb1$ 0.2302 | 0.7121 $\xb1$ 0.8439 | 1.1118$\xb1$ 0.2986 | 16.7889 $\xb1$ 7.6244 | 0.5397$\xb1$ 0.1432 | 3.9278 $\xb1$ 3.3254 | |

10 | 1.1185$\xb1$ 0.0837 | 1.2103 $\xb1$ 0.1364 | 14.2539$\xb1$ 4.4617 | 75.7570 $\xb1$ 21.1887 | 3.7237$\xb1$ 1.0159 | 18.4265 $\xb1$ 6.9328 | |

F7 | 2 | 0.1219$\xb1$ 0.1055 | 0.2861 $\xb1$ 0.2670 | 0.0732$\xb1$ 0.0671 | 0.0773 $\xb1$ 0.0712 | 0.0131$\xb1$ 0.0142 | 0.2308 $\xb1$ 0.2212 |

5 | 43.3089$\xb1$ 30.4430 | 70.2126 $\xb1$ 47.2680 | 66.0930$\xb1$ 34.6149 | 72.0384 $\xb1$ 30.1393 | 67.1543$\xb1$ 27.5345 | 126.1766 $\xb1$ 79.2998 | |

10 | 204.2603$\xb1$ 65.2703 | 316.0039 $\xb1$ 125.3541 | 353.6906$\xb1$ 159.2818 | 728.8562 $\xb1$ 328.2364 | 708.1847$\xb1$ 356.1181 | 719.1093 $\xb1$ 365.0644 | |

F8 | 2 | 3.1274$\xb1$ 1.4961 | 3.3412 $\xb1$ 3.0295 | 3.7549$\xb1$ 2.0459 | 3.8671 $\xb1$ 1.3504 | 2.3542$\xb1$ 1.9835 | 3.1703 $\xb1$ 2.2549 |

5 | 18.5786$\xb1$ 3.9201 | 19.3769 $\xb1$ 4.4019 | 22.5314$\xb1$ 7.7902 | 29.2176 $\xb1$ 5.6230 | 21.0797$\xb1$ 4.3804 | 22.6443 $\xb1$ 7.0308 | |

10 | 59.4989$\xb1$ 11.4535 | 59.9236 $\xb1$ 6.4440 | 76.0060$\xb1$ 16.9585 | 96.6026 $\xb1$ 16.0349 | 72.4144$\xb1$ 13.7050 | 90.2328 $\xb1$ 12.0011 | |

Average rank | 1.98 | 3.35 | 3.35 | 5.46 | 2.02 | 4.83 |

Problem . | d . | UMP/RBFN . | RBFN . | UMP/QP . | QP . | UMP/SVM . | SVM . |
---|---|---|---|---|---|---|---|

F1 | 2 | 9.8806E-3$\xb1$ 0.0156 | 0.0642 $\xb1$ 0.1018 | 1.4462E-4$\xb1$ 2.4305E-4 | 2.8772 $\xb1$ 2.8031 | 6.0986E-4$\xb1$ 7.6748E-4 | 0.8707 $\xb1$ 1.5511 |

5 | 0.0331$\xb1$ 0.0328 | 0.2288 $\xb1$ 0.1966 | 0.7640$\xb1$ 0.7112 | 54.0371 $\xb1$ 23.7337 | 0.0264$\xb1$ 0.0169 | 25.1719 $\xb1$ 16.9316 | |

10 | 1.9803$\xb1$ 1.0066 | 9.6927 $\xb1$ 5.0127 | 67.9611$\xb1$ 21.5325 | 287.0700 $\xb1$ 76.2774 | 21.5962$\xb1$ 11.4967 | 87.2006 $\xb1$ 37.7374 | |

F2 | 2 | 0.0182$\xb1$ 0.0342 | 0.2342 $\xb1$ 0.5029 | 3.0383E-4$\xb1$ 2.5810E-4 | 2.8759 $\xb1$ 2.6865 | 1.0445E-3$\xb1$ 1.3433E-3 | 5.2747 $\xb1$ 8.8904 |

5 | 0.4057$\xb1$ 0.3093 | 2.3933 $\xb1$ 1.7200 | 14.0614$\xb1$ 19.1254 | 155.4160 $\xb1$ 68.8183 | 0.0799$\xb1$ 0.0468 | 173.4310 $\xb1$ 82.1052 | |

10 | 32.4434$\xb1$ 16.5484 | 129.7015 $\xb1$ 51.3120 | 307.0399$\xb1$ 97.6482 | 1425.5204 $\xb1$ 393.2456 | 109.8823$\xb1$ 53.2732 | 555.1530 $\xb1$ 268.1075 | |

F3 | 2 | 0.0437$\xb1$ 0.0915 | 1.2991 $\xb1$ 1.8210 | 9.9812E-3$\xb1$ 0.0320 | 1.3025 $\xb1$ 1.2417 | 0.0011$\xb1$ 0.0010 | 5.4055 $\xb1$ 7.0946 |

5 | 0.4611$\xb1$ 0.4911 | 3.2056 $\xb1$ 4.6624 | 5.6759$\xb1$ 5.0457 | 39.0094 $\xb1$ 21.5417 | 0.0307$\xb1$ 0.0185 | 38.9898 $\xb1$ 25.2550 | |

10 | 243.0518$\xb1$ 84.7628 | 823.7360 $\xb1$ 378.8218 | 477.9377$\xb1$ 182.8839 | 1921.1620 $\xb1$ 503.9823 | 559.6709$\xb1$ 316.1527 | 2451.7201 $\xb1$ 795.8515 | |

F4 | 2 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 2.2400 $\xb1$ 2.1029 | 0.0$\xb1$ 0.0 | 0.6800 $\xb1$ 0.6144 |

5 | 0.1200$\xb1$ 0.3249 | 0.2000 $\xb1$ 0.4000 | 4.4000$\xb1$ 4.9799 | 49.0000 $\xb1$ 24.5715 | 0.7600$\xb1$ 1.0688 | 26.2000 $\xb1$ 24.6398 | |

10 | 2.5200$\xb1$ 1.0998 | 9.5200 $\xb1$ 4.3185 | 65.2000$\xb1$ 22.7385 | 230.8000 $\xb1$ 77.1305 | 23.5200$\xb1$ 10.9986 | 76.1600 $\xb1$ 34.7754 | |

F5 | 2 | 2.7644$\xb1$ 1.0851 | 10.4969 $\xb1$ 4.7438 | 2.7476$\xb1$ 0.8620 | 8.0343 $\xb1$ 2.2641 | 1.8426$\xb1$ 0.6829 | 2.2079 $\xb1$ 1.1024 |

5 | 3.6208$\xb1$ 0.7392 | 12.9774 $\xb1$ 1.5847 | 4.0928$\xb1$ 0.912 | 16.1825 $\xb1$ 1.9481 | 3.5112$\xb1$ 0.5640 | 6.3787 $\xb1$ 2.4737 | |

10 | 4.2216$\xb1$ 0.3694 | 13.2189 $\xb1$ 1.1633 | 10.6836$\xb1$ 6.2570 | 18.4682 $\xb1$ 1.0169 | 3.7534$\xb1$ 0.3806 | 9.8341 $\xb1$ 1.4496 | |

F6 | 2 | 0.1410$\xb1$ 0.0035 | 0.1600 $\xb1$ 0.3545 | 0.4851$\xb1$ 0.2165 | 1.1618 $\xb1$ 0.7925 | 0.0602$\xb1$ 0.0401 | 0.1971 $\xb1$ 0.1836 |

5 | 0.6092$\xb1$ 0.2302 | 0.7121 $\xb1$ 0.8439 | 1.1118$\xb1$ 0.2986 | 16.7889 $\xb1$ 7.6244 | 0.5397$\xb1$ 0.1432 | 3.9278 $\xb1$ 3.3254 | |

10 | 1.1185$\xb1$ 0.0837 | 1.2103 $\xb1$ 0.1364 | 14.2539$\xb1$ 4.4617 | 75.7570 $\xb1$ 21.1887 | 3.7237$\xb1$ 1.0159 | 18.4265 $\xb1$ 6.9328 | |

F7 | 2 | 0.1219$\xb1$ 0.1055 | 0.2861 $\xb1$ 0.2670 | 0.0732$\xb1$ 0.0671 | 0.0773 $\xb1$ 0.0712 | 0.0131$\xb1$ 0.0142 | 0.2308 $\xb1$ 0.2212 |

5 | 43.3089$\xb1$ 30.4430 | 70.2126 $\xb1$ 47.2680 | 66.0930$\xb1$ 34.6149 | 72.0384 $\xb1$ 30.1393 | 67.1543$\xb1$ 27.5345 | 126.1766 $\xb1$ 79.2998 | |

10 | 204.2603$\xb1$ 65.2703 | 316.0039 $\xb1$ 125.3541 | 353.6906$\xb1$ 159.2818 | 728.8562 $\xb1$ 328.2364 | 708.1847$\xb1$ 356.1181 | 719.1093 $\xb1$ 365.0644 | |

F8 | 2 | 3.1274$\xb1$ 1.4961 | 3.3412 $\xb1$ 3.0295 | 3.7549$\xb1$ 2.0459 | 3.8671 $\xb1$ 1.3504 | 2.3542$\xb1$ 1.9835 | 3.1703 $\xb1$ 2.2549 |

5 | 18.5786$\xb1$ 3.9201 | 19.3769 $\xb1$ 4.4019 | 22.5314$\xb1$ 7.7902 | 29.2176 $\xb1$ 5.6230 | 21.0797$\xb1$ 4.3804 | 22.6443 $\xb1$ 7.0308 | |

10 | 59.4989$\xb1$ 11.4535 | 59.9236 $\xb1$ 6.4440 | 76.0060$\xb1$ 16.9585 | 96.6026 $\xb1$ 16.0349 | 72.4144$\xb1$ 13.7050 | 90.2328 $\xb1$ 12.0011 | |

Average rank | 1.98 | 3.35 | 3.35 | 5.46 | 2.02 | 4.83 |

From Table 2 and Figures 1–3, the results of UMP/RBFN, UMP/QP, and UMP/SVM are significantly better than those of the other algorithms in most test problems. The performance of the three algorithms (UMP/RBFN, UMP/QP, and UMP/SVM) is always better than the other on both unimodal or multimodal problems, especially in 5- and 10-dimensional test problems. The results are mainly attributed to the fact that there is uncertainty information in the three algorithms that can be used to guide the search of algorithms and the update of surrogates. The test problems are more complex as the number of dimensions increases, and there are many local optima for multimodal problems. The uncertainty information is able to strengthen the exploration ability of algorithms and improve the accuracy of the surrogates.

#### 4.3.2 Comparison with Peer Algorithms

Problem . | d . | UMP/RBFN . | DUM/RBFN . | UMP/QP . | DUM/QP . | UMP/SVM . | DUM/SVM . |
---|---|---|---|---|---|---|---|

F1 | 2 | 9.8806E-3$\xb1$ 0.0156 | 0.0502 $\xb1$ 0.0802 | 1.4462E-4$\xb1$ 2.4305E-4 | 4.1442 $\xb1$ 3.3661 | 6.0986E-4$\xb1$ 7.6748E-4 | 0.0180 $\xb1$ 0.0232 |

5 | 0.0331$\xb1$ 0.0328 | 0.1686 $\xb1$ 0.1147 | 0.7640$\xb1$ 0.7112 | 140.5686 $\xb1$ 65.2623 | 0.0264$\xb1$ 0.0169 | 26.6025 $\xb1$ 25.5855 | |

10 | 1.9803$\xb1$ 1.0066 | 9.4756 $\xb1$ 4.5445 | 67.9611$\xb1$ 21.5325 | 477.5787 $\xb1$ 130.4803 | 21.5962$\xb1$ 11.4967 | 84.9443 $\xb1$ 29.7331 | |

F2 | 2 | 0.0182$\xb1$ 0.0342 | 0.2449 $\xb1$ 0.3725 | 3.0383E-4$\xb1$ 2.5810E-4 | 6.7903 $\xb1$ 6.7277 | 1.1499E-3$\xb1$ 1.0445E-3 | 0.3620 $\xb1$ 0.7392 |

5 | 0.4057$\xb1$ 0.3093 | 4.1801 $\xb1$ 3.7846 | 14.0614$\xb1$ 19.1254 | 365.6379 $\xb1$ 146.3157 | 0.0799$\xb1$ 0.0468 | 148.4953 $\xb1$ 124.1047 | |

10 | 32.4434$\xb1$ 16.5484 | 119.4572 $\xb1$ 49.2795 | 307.0399$\xb1$ 97.6482 | 2484.1205 $\xb1$ 510.5520 | 109.8823$\xb1$ 53.2732 | 638.3514 $\xb1$ 180.0408 | |

F3 | 2 | 0.0437$\xb1$ 0.0915 | 0.5505 $\xb1$ 0.6879 | 9.9812E-3$\xb1$ 0.0320 | 4.7287 $\xb1$ 2.9526 | 0.0053$\xb1$ 0.0069 | 0.2930 $\xb1$ 0.5269 |

5 | 0.4611$\xb1$ 0.4911 | 2.0510 $\xb1$ 1.1517 | 5.6759$\xb1$ 5.0457 | 110.6858 $\xb1$ 48.6745 | 0.0307$\xb1$ 0.0185 | 35.7253 $\xb1$ 33.3522 | |

10 | 243.0518$\xb1$ 84.7628 | 848.8346 $\xb1$ 268.9680 | 477.9377$\xb1$ 182.8839 | 3141.0693 $\xb1$ 956.7205 | 559.6709$\xb1$ 316.1527 | 2204.2445 $\xb1$ 1091.0657 | |

F4 | 2 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 4.0800 $\xb1$ 3.6212 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 |

5 | 0.1200$\xb1$ 0.3249 | 0.1600 $\xb1$ 0.3666 | 4.4000$\xb1$ 4.9799 | 141.1600 $\xb1$ 55.9783 | 0.7600$\xb1$ 1.0688 | 31.6800 $\xb1$ 26.8264 | |

10 | 2.5200$\xb1$ 1.0998 | 8.6000 $\xb1$ 3.1874 | 65.2000$\xb1$ 22.7385 | 439.3200 $\xb1$ 112.3308 | 23.5200$\xb1$ 10.9986 | 76.6000 $\xb1$ 30.6111 | |

F5 | 2 | 2.7644$\xb1$ 1.0851 | 9.7490 $\xb1$ 3.8671 | 2.7476$\xb1$ 0.8620 | 8.1565 $\xb1$ 2.7408 | 1.8426 $\xb1$ 0.6829 | 0.9289$\xb1$ 0.5018 |

5 | 3.6208$\xb1$ 0.7392 | 11.8476 $\xb1$ 2.1805 | 4.0928$\xb1$ 0.9124 | 17.0522 $\xb1$ 1.5073 | 3.5112$\xb1$ 0.5640 | 6.0516 $\xb1$ 1.4182 | |

10 | 4.2216$\xb1$ 0.3694 | 12.7883 $\xb1$ 1.5767 | 10.6836$\xb1$ 6.2570 | 19.2076 $\xb1$ 0.7207 | 3.7534$\xb1$ 0.3806 | 9.1504 $\xb1$ 1.2157 | |

F6 | 2 | 0.1410$\xb1$ 0.0035 | 0.1494 $\xb1$ 0.0676 | 0.4851$\xb1$ 0.216 | 2.0738 $\xb1$ 1.0517 | 0.0602$\xb1$ 0.0401 | 0.1313 $\xb1$ 0.1128 |

5 | 0.6092$\xb1$ 0.2302 | 0.6143 $\xb1$ 0.1708 | 1.1118$\xb1$ 0.2986 | 33.2851 $\xb1$ 11.7161 | 0.5397$\xb1$ 0.1432 | 5.2599 $\xb1$ 4.2937 | |

10 | 1.1185$\xb1$ 0.0837 | 1.1323 $\xb1$ 0.0466 | 14.2539$\xb1$ 4.4617 | 117.7120 $\xb1$ 20.8241 | 3.7237$\xb1$ 1.0159 | 17.1081 $\xb1$ 6.6650 | |

F7 | 2 | 0.1219$\xb1$ 0.1055 | 0.2439 $\xb1$ 0.2661 | 0.0732$\xb1$ 0.0671 | 0.1221 $\xb1$ 0.1160 | 0.0131$\xb1$ 0.0142 | 0.0974 $\xb1$ 0.0516 |

5 | 43.3089$\xb1$ 30.4430 | 88.9265 $\xb1$ 47.3513 | 66.0930$\xb1$ 34.6149 | 113.1713 $\xb1$ 55.7217 | 67.1543$\xb1$ 27.5345 | 104.1294 $\xb1$ 65.0487 | |

10 | 204.2603$\xb1$ 65.2703 | 311.2954 $\xb1$ 159.9181 | 353.6906$\xb1$ 159.2818 | 1876.2670 $\xb1$ 884.2830 | 708.1847$\xb1$ 356.1181 | 719.7931 $\xb1$ 303.3686 | |

F8 | 2 | 3.1274$\xb1$ 1.4961 | 3.3755 $\xb1$ 2.4286 | 3.7549$\xb1$ 2.0459 | 4.1802 $\xb1$ 2.0134 | 2.3542$\xb1$ 1.9835 | 2.4814 $\xb1$ 2.0600 |

5 | 18.5786$\xb1$ 3.9201 | 18.6270 $\xb1$ 5.6959 | 22.5314$\xb1$ 7.7902 | 31.8924 $\xb1$ 8.3335 | 21.0797$\xb1$ 4.3804 | 21.1876 $\xb1$ 6.2968 | |

10 | 59.4989$\xb1$ 11.4535 | 60.8467 $\xb1$ 6.7949 | 76.0060$\xb1$ 16.9585 | 123.9444 $\xb1$ 13.2013 | 72.4144$\xb1$ 13.7050 | 83.7737 $\xb1$ 13.1376 | |

Average rank | 2.08 | 3.42 | 3.38 | 5.92 | 2.04 | 4.17 |

Problem . | d . | UMP/RBFN . | DUM/RBFN . | UMP/QP . | DUM/QP . | UMP/SVM . | DUM/SVM . |
---|---|---|---|---|---|---|---|

F1 | 2 | 9.8806E-3$\xb1$ 0.0156 | 0.0502 $\xb1$ 0.0802 | 1.4462E-4$\xb1$ 2.4305E-4 | 4.1442 $\xb1$ 3.3661 | 6.0986E-4$\xb1$ 7.6748E-4 | 0.0180 $\xb1$ 0.0232 |

5 | 0.0331$\xb1$ 0.0328 | 0.1686 $\xb1$ 0.1147 | 0.7640$\xb1$ 0.7112 | 140.5686 $\xb1$ 65.2623 | 0.0264$\xb1$ 0.0169 | 26.6025 $\xb1$ 25.5855 | |

10 | 1.9803$\xb1$ 1.0066 | 9.4756 $\xb1$ 4.5445 | 67.9611$\xb1$ 21.5325 | 477.5787 $\xb1$ 130.4803 | 21.5962$\xb1$ 11.4967 | 84.9443 $\xb1$ 29.7331 | |

F2 | 2 | 0.0182$\xb1$ 0.0342 | 0.2449 $\xb1$ 0.3725 | 3.0383E-4$\xb1$ 2.5810E-4 | 6.7903 $\xb1$ 6.7277 | 1.1499E-3$\xb1$ 1.0445E-3 | 0.3620 $\xb1$ 0.7392 |

5 | 0.4057$\xb1$ 0.3093 | 4.1801 $\xb1$ 3.7846 | 14.0614$\xb1$ 19.1254 | 365.6379 $\xb1$ 146.3157 | 0.0799$\xb1$ 0.0468 | 148.4953 $\xb1$ 124.1047 | |

10 | 32.4434$\xb1$ 16.5484 | 119.4572 $\xb1$ 49.2795 | 307.0399$\xb1$ 97.6482 | 2484.1205 $\xb1$ 510.5520 | 109.8823$\xb1$ 53.2732 | 638.3514 $\xb1$ 180.0408 | |

F3 | 2 | 0.0437$\xb1$ 0.0915 | 0.5505 $\xb1$ 0.6879 | 9.9812E-3$\xb1$ 0.0320 | 4.7287 $\xb1$ 2.9526 | 0.0053$\xb1$ 0.0069 | 0.2930 $\xb1$ 0.5269 |

5 | 0.4611$\xb1$ 0.4911 | 2.0510 $\xb1$ 1.1517 | 5.6759$\xb1$ 5.0457 | 110.6858 $\xb1$ 48.6745 | 0.0307$\xb1$ 0.0185 | 35.7253 $\xb1$ 33.3522 | |

10 | 243.0518$\xb1$ 84.7628 | 848.8346 $\xb1$ 268.9680 | 477.9377$\xb1$ 182.8839 | 3141.0693 $\xb1$ 956.7205 | 559.6709$\xb1$ 316.1527 | 2204.2445 $\xb1$ 1091.0657 | |

F4 | 2 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 | 4.0800 $\xb1$ 3.6212 | 0.0$\xb1$ 0.0 | 0.0$\xb1$ 0.0 |

5 | 0.1200$\xb1$ 0.3249 | 0.1600 $\xb1$ 0.3666 | 4.4000$\xb1$ 4.9799 | 141.1600 $\xb1$ 55.9783 | 0.7600$\xb1$ 1.0688 | 31.6800 $\xb1$ 26.8264 | |

10 | 2.5200$\xb1$ 1.0998 | 8.6000 $\xb1$ 3.1874 | 65.2000$\xb1$ 22.7385 | 439.3200 $\xb1$ 112.3308 | 23.5200$\xb1$ 10.9986 | 76.6000 $\xb1$ 30.6111 | |

F5 | 2 | 2.7644$\xb1$ 1.0851 | 9.7490 $\xb1$ 3.8671 | 2.7476$\xb1$ 0.8620 | 8.1565 $\xb1$ 2.7408 | 1.8426 $\xb1$ 0.6829 | 0.9289$\xb1$ 0.5018 |

5 | 3.6208$\xb1$ 0.7392 | 11.8476 $\xb1$ 2.1805 | 4.0928$\xb1$ 0.9124 | 17.0522 $\xb1$ 1.5073 | 3.5112$\xb1$ 0.5640 | 6.0516 $\xb1$ 1.4182 | |

10 | 4.2216$\xb1$ 0.3694 | 12.7883 $\xb1$ 1.5767 | 10.6836$\xb1$ 6.2570 | 19.2076 $\xb1$ 0.7207 | 3.7534$\xb1$ 0.3806 | 9.1504 $\xb1$ 1.2157 | |

F6 | 2 | 0.1410$\xb1$ 0.0035 | 0.1494 $\xb1$ 0.0676 | 0.4851$\xb1$ 0.216 | 2.0738 $\xb1$ 1.0517 | 0.0602$\xb1$ 0.0401 | 0.1313 $\xb1$ 0.1128 |

5 | 0.6092$\xb1$ 0.2302 | 0.6143 $\xb1$ 0.1708 | 1.1118$\xb1$ 0.2986 | 33.2851 $\xb1$ 11.7161 | 0.5397$\xb1$ 0.1432 | 5.2599 $\xb1$ 4.2937 | |

10 | 1.1185$\xb1$ 0.0837 | 1.1323 $\xb1$ 0.0466 | 14.2539$\xb1$ 4.4617 | 117.7120 $\xb1$ 20.8241 | 3.7237$\xb1$ 1.0159 | 17.1081 $\xb1$ 6.6650 | |

F7 | 2 | 0.1219$\xb1$ 0.1055 | 0.2439 $\xb1$ 0.2661 | 0.0732$\xb1$ 0.0671 | 0.1221 $\xb1$ 0.1160 | 0.0131$\xb1$ 0.0142 | 0.0974 $\xb1$ 0.0516 |

5 | 43.3089$\xb1$ 30.4430 | 88.9265 $\xb1$ 47.3513 | 66.0930$\xb1$ 34.6149 | 113.1713 $\xb1$ 55.7217 | 67.1543$\xb1$ 27.5345 | 104.1294 $\xb1$ 65.0487 | |

10 | 204.2603$\xb1$ 65.2703 | 311.2954 $\xb1$ 159.9181 | 353.6906$\xb1$ 159.2818 | 1876.2670 $\xb1$ 884.2830 | 708.1847$\xb1$ 356.1181 | 719.7931 $\xb1$ 303.3686 | |

F8 | 2 | 3.1274$\xb1$ 1.4961 | 3.3755 $\xb1$ 2.4286 | 3.7549$\xb1$ 2.0459 | 4.1802 $\xb1$ 2.0134 | 2.3542$\xb1$ 1.9835 | 2.4814 $\xb1$ 2.0600 |

5 | 18.5786$\xb1$ 3.9201 | 18.6270 $\xb1$ 5.6959 | 22.5314$\xb1$ 7.7902 | 31.8924 $\xb1$ 8.3335 | 21.0797$\xb1$ 4.3804 | 21.1876 $\xb1$ 6.2968 | |

10 | 59.4989$\xb1$ 11.4535 | 60.8467 $\xb1$ 6.7949 | 76.0060$\xb1$ 16.9585 | 123.9444 $\xb1$ 13.2013 | 72.4144$\xb1$ 13.7050 | 83.7737 $\xb1$ 13.1376 | |

Average rank | 2.08 | 3.42 | 3.38 | 5.92 | 2.04 | 4.17 |

The Table 3 and Figures 4–6 show that UMP/RBFN, UMP/QP, and UMP/SVM achieve significantly better performance than the other algorithms on most test problems. There is a significant difference in 10 dimensional test problems, especially for $F3$, $F5$, and $F7$, which maybe due to the accuracy of DUM that is unreliable. Thus, the DUM cannot efficiently guide the search of algorithms when there are many local optima on $F5$ and $F7$ in 10-dimensional test problems, and it cannot accurately fit the fitness landscape of $F3$, $F5$, and $F7$ in 10-dimensional test problems. Instead, the proposed UMP still has good performance in these problems with 10 dimension.

### 4.4 Comparison on Ensemble Surrogates

Problem . | d . | UMP/ensemble . | VUM/ensemble . | $Uens$/ensemble . |
---|---|---|---|---|

F1 | 2 | 7.7161E-5$\xb1$ 1.5035E-4 | 1.3604 $\xb1$ 1.2453 | 9.3382E-3 $\xb1$ 0.0197 |

5 | 0.0011$\xb1$ 0.0012 | 12.1513 $\xb1$ 8.6543 | 0.1555 $\xb1$ 0.0832 | |

10 | 1.0012$\xb1$ 0.7830 | 23.5616 $\xb1$ 9.8029 | 9.3609 $\xb1$ 3.5025 | |

F2 | 2 | 2.0820E-4$\xb1$ 2.2511E-4 | 3.7257 $\xb1$ 2.9982 | 0.0810 $\xb1$ 0.1031 |

5 | 0.0526$\xb1$ 0.0614 | 77.1360 $\xb1$ 38.0107 | 2.6106 $\xb1$ 2.0893 | |

10 | 33.0773$\xb1$ 25.3703 | 284.9453 $\xb1$ 95.0346 | 141.5993 $\xb1$ 40.6508 | |

F3 | 2 | 4.1281E-4$\xb1$ 5.1014E-4 | 3.4762 $\xb1$ 2.0919 | 0.6851 $\xb1$ 1.3881 |

5 | 0.1005$\xb1$ 0.1455 | 28.3199 $\xb1$ 11.8856 | 2.5193 $\xb1$ 1.5458 | |

10 | 537.8770$\xb1$ 150.8156 | 1204.2153 $\xb1$ 381.1870 | 889.4860 $\xb1$ 271.9350 | |

F4 | 2 | 0.0$\xb1$ 0.0 | 1.3200 $\xb1$ 1.3181 | 0.0$\xb1$ 0.0 |

5 | 0.1200$\xb1$ 0.3249 | 12.600 $\xb1$ 11.0381 | 0.1200$\xb1$ 0.3249 | |

10 | 3.2400$\xb1$ 1.9448 | 21.0000 $\xb1$ 8.4332 | 10.2000 $\xb1$ 4.8249 | |

F5 | 2 | 2.7002$\xb1$ 1.1667 | 10.3965 $\xb1$ 4.9964 | 3.8192 $\xb1$ 1.8471 |

5 | 3.5092$\xb1$ 0.7333 | 13.8540 $\xb1$ 2.8998 | 8.4735 $\xb1$ 1.5095 | |

10 | 5.5079$\xb1$ 0.5751 | 13.1730 $\xb1$ 0.9270 | 10.8165 $\xb1$ 1.1127 | |

F6 | 2 | 0.0872$\xb1$ 0.0544 | 0.6791 $\xb1$ 0.3800 | 0.1351 $\xb1$ 0.0789 |

5 | 0.6518 $\xb1$ 0.2003 | 1.3107 $\xb1$ 0.3293 | 0.6159$\xb1$ 0.1419 | |

10 | 1.0563$\xb1$ 0.1378 | 1.3724 $\xb1$ 0.3217 | 1.4022 $\xb1$ 0.3057 | |

F7 | 2 | 0.0076$\xb1$ 0.0090 | 0.1469 $\xb1$ 0.1503 | 0.2916 $\xb1$ 0.2351 |

5 | 63.8693 $\xb1$ 30.8029 | 81.6944 $\xb1$ 35.7565 | 59.3588$\xb1$ 38.5356 | |

10 | 235.0830$\xb1$ 119.8799 | 585.8805 $\xb1$ 231.1875 | 334.1423 $\xb1$ 157.4397 | |

F8 | 2 | 1.7132$\xb1$ 1.1547 | 3.3034 $\xb1$ 1.8626 | 2.7242 $\xb1$ 1.3798 |

5 | 19.9698 $\xb1$ 5.5287 | 24.4794 $\xb1$ 5.4224 | 17.1605$\xb1$ 4.8970 | |

10 | 69.4359 $\xb1$ 7.8951 | 74.3970 $\xb1$ 11.7897 | 58.7404$\xb1$ 8.5256 | |

Average rank | 1.27 | 2.83 | 1.90 |

Problem . | d . | UMP/ensemble . | VUM/ensemble . | $Uens$/ensemble . |
---|---|---|---|---|

F1 | 2 | 7.7161E-5$\xb1$ 1.5035E-4 | 1.3604 $\xb1$ 1.2453 | 9.3382E-3 $\xb1$ 0.0197 |

5 | 0.0011$\xb1$ 0.0012 | 12.1513 $\xb1$ 8.6543 | 0.1555 $\xb1$ 0.0832 | |

10 | 1.0012$\xb1$ 0.7830 | 23.5616 $\xb1$ 9.8029 | 9.3609 $\xb1$ 3.5025 | |

F2 | 2 | 2.0820E-4$\xb1$ 2.2511E-4 | 3.7257 $\xb1$ 2.9982 | 0.0810 $\xb1$ 0.1031 |

5 | 0.0526$\xb1$ 0.0614 | 77.1360 $\xb1$ 38.0107 | 2.6106 $\xb1$ 2.0893 | |

10 | 33.0773$\xb1$ 25.3703 | 284.9453 $\xb1$ 95.0346 | 141.5993 $\xb1$ 40.6508 | |

F3 | 2 | 4.1281E-4$\xb1$ 5.1014E-4 | 3.4762 $\xb1$ 2.0919 | 0.6851 $\xb1$ 1.3881 |

5 | 0.1005$\xb1$ 0.1455 | 28.3199 $\xb1$ 11.8856 | 2.5193 $\xb1$ 1.5458 | |

10 | 537.8770$\xb1$ 150.8156 | 1204.2153 $\xb1$ 381.1870 | 889.4860 $\xb1$ 271.9350 | |

F4 | 2 | 0.0$\xb1$ 0.0 | 1.3200 $\xb1$ 1.3181 | 0.0$\xb1$ 0.0 |

5 | 0.1200$\xb1$ 0.3249 | 12.600 $\xb1$ 11.0381 | 0.1200$\xb1$ 0.3249 | |

10 | 3.2400$\xb1$ 1.9448 | 21.0000 $\xb1$ 8.4332 | 10.2000 $\xb1$ 4.8249 | |

F5 | 2 | 2.7002$\xb1$ 1.1667 | 10.3965 $\xb1$ 4.9964 | 3.8192 $\xb1$ 1.8471 |

5 | 3.5092$\xb1$ 0.7333 | 13.8540 $\xb1$ 2.8998 | 8.4735 $\xb1$ 1.5095 | |

10 | 5.5079$\xb1$ 0.5751 | 13.1730 $\xb1$ 0.9270 | 10.8165 $\xb1$ 1.1127 | |

F6 | 2 | 0.0872$\xb1$ 0.0544 | 0.6791 $\xb1$ 0.3800 | 0.1351 $\xb1$ 0.0789 |

5 | 0.6518 $\xb1$ 0.2003 | 1.3107 $\xb1$ 0.3293 | 0.6159$\xb1$ 0.1419 | |

10 | 1.0563$\xb1$ 0.1378 | 1.3724 $\xb1$ 0.3217 | 1.4022 $\xb1$ 0.3057 | |

F7 | 2 | 0.0076$\xb1$ 0.0090 | 0.1469 $\xb1$ 0.1503 | 0.2916 $\xb1$ 0.2351 |

5 | 63.8693 $\xb1$ 30.8029 | 81.6944 $\xb1$ 35.7565 | 59.3588$\xb1$ 38.5356 | |

10 | 235.0830$\xb1$ 119.8799 | 585.8805 $\xb1$ 231.1875 | 334.1423 $\xb1$ 157.4397 | |

F8 | 2 | 1.7132$\xb1$ 1.1547 | 3.3034 $\xb1$ 1.8626 | 2.7242 $\xb1$ 1.3798 |

5 | 19.9698 $\xb1$ 5.5287 | 24.4794 $\xb1$ 5.4224 | 17.1605$\xb1$ 4.8970 | |

10 | 69.4359 $\xb1$ 7.8951 | 74.3970 $\xb1$ 11.7897 | 58.7404$\xb1$ 8.5256 | |

Average rank | 1.27 | 2.83 | 1.90 |

From Table 4 and Figures 8–10, the UMP/ensemble significantly outperforms $Uens$/ensemble and VUM/ensemble on most test problems. In the results, the performance of VUM/ensemble is worst. The performance of UMP/ensemble and $Uens$/ensemble is similar on $F4$ in 2 and 5 dimensions, and $Uens$/ensemble is slightly better than UMP/ensemble on $F6$, $F7$, and $F8$ in 5 dimension. However, UMP/ensemble is worse than $Uens$/ensemble on $F8$ in 10 dimension, probably because the number of computational budget $FEs=100$ is too small. Most of the reevaluation solutions are consumed due to a large degree of uncertainty, resulting in less opportunity for UMP/ensemble to exploit the search space sufficiently.

Regarding the uncertainty method $Uens$, it is maximum difference among the outputs of the ensemble members for the prediction of a solution. In essence, this method describes the prediction difference among ensemble members in a solution. Likewise, for the method VUM, the uncertainty of prediction of a solution is measured by the variance of the prediction output by the base surrogates of the ensemble. This method indicates the average squared deviation of the base surrogates about the output of the ensemble surrogates. From the probability and statistic viewpoint, these methods measuring the uncertainty of prediction can be insufficient. Therefore, these compared methods are potentially unreliable in measuring the uncertainty of the prediction of surrogates.

## 5 Conclusion

This article mainly addresses the issue that there is no theoretical method to measure the uncertainty of prediction of Non-GP surrogates. This article proposes a theoretical method to measure the uncertainty. In this method, a stationary random field with a known zero mean is used to measure the uncertainty of prediction of Non-KGP surrogates. The method's effectiveness has been verified based on some experiments in single and ensemble surrogates cases. The experimental results demonstrate that the proposed method is more promising than other methods on a set of test problems.

Although the performance of the proposed method is competitive, the computational cost of this method is higher than others. The computational cost of UMP, DUM, $Uens$ and VUM is $O(N3)$, $O(N)$, $O(1)$, and $O(1)$, respectively. $N$ is the number of training samples. Thus, the main limitation of the method is that it needs more computational cost in measuring the uncertainty for the prediction of Non-GP surrogates. Therefore, our future work is to solve this drawback using transfer learning.

## Acknowledgments

This work was supported in part by the National Natural Science Foundation of China under Grants 62076226, in part by the Fundamental Research Funds for the Central Universities China University of Geosciences (Wuhan) under Grant $CUGGC02$, and in part by the 111 project under Grant $B17040$.