Abstract

The Euclidean minimum spanning tree (EMST), widely used in a variety of domains, is a minimum spanning tree of a set of points in space where the edge weight between each pair of points is their Euclidean distance. Since the generation of an EMST is entirely determined by the Euclidean distance between solutions (points), the properties of EMSTs have a close relation with the distribution and position information of solutions. This paper explores the properties of EMSTs and proposes an EMST-based evolutionary algorithm (ETEA) to solve multi-objective optimization problems (MOPs). Unlike most EMO algorithms that focus on the Pareto dominance relation, the proposed algorithm mainly considers distance-based measures to evaluate and compare individuals during the evolutionary search. Specifically, in ETEA, four strategies are introduced: (1) An EMST-based crowding distance (ETCD) is presented to estimate the density of individuals in the population; (2) A distance comparison approach incorporating ETCD is used to assign the fitness value for individuals; (3) A fitness adjustment technique is designed to avoid the partial overcrowding in environmental selection; (4) Three diversity indicators—the minimum edge, degree, and ETCD—with regard to EMSTs are applied to determine the survival of individuals in archive truncation. From a series of extensive experiments on 32 test instances with different characteristics, ETEA is found to be competitive against five state-of-the-art algorithms and its predecessor in providing a good balance among convergence, uniformity, and spread.

1  Introduction

Many real-world problems involve simultaneous optimization of several competing objectives. In these multi-objective optimization problems (MOPs), there is usually no single optimal solution, but rather a set of alternative solutions, called the Pareto set, due to the conflicting nature of the objectives. In the absence of any further information, the decision-makers usually require an approximation of the Pareto set for making their final choice.

Over the past few years, evolutionary algorithms (EAs) have been gaining increasing attention among researchers and practitioners to solve MOPs (Coello et al., 2007; Deb, 2001; Branke et al., 2008). One main advantage of EAs is that they have low requirements on the problem characteristics (e.g., nonconvexity, discontinuity, nonlinear constraint, and multimodality), and objectives can be easily added, removed, or modified. Moreover, due to the fact that they act on a set of candidates, EAs are suitable for generating a Pareto set approximation in a single run.

As a consequence, numerous effective evolutionary multi-objective optimization (EMO) algorithms have been proposed, such as the non-dominated sorting genetic algorithm II (NSGA-II; Deb et al., 2002), strength Pareto evolutionary algorithm 2 (SPEA2; Zitzler et al., 2002), Pareto-based evolution strategy (PAES; Knowles and Corne, 2000), indicator-based evolutionary algorithm (IBEA; Zitzler and Künzli, 2004), ε-dominance (Laumanns et al., 2002) based multi-objective evolutionary algorithm (ε-MOEA; Deb, Mohan, et al., 2005), multi-objective covariance matrix adaptation evolution strategy (MO-CMA-ES; Igel, Hansen, et al., 2007), S metric selection evolutionary multi-objective optimization algorithm (SMS-EMOA; Beume et al., 2007), and decomposition-based multi-objective evolutionary algorithm (MOEA/D; Zhang and Li, 2007), some of which are applied to various problem domains (see Fonseca and Fleming, 1995; Coello and Lamont, 2004; Tan et al., 2005; Abraham et al., 2005; Jin, 2006; Bui and Alam, 2008; Wang et al., 2010; Teo and Abbass, 2004; Friedrich et al., 2010). Generally speaking, these algorithms share the three common goals—minimizing the distance to the optimal front, maintaining the uniform distribution, and extending the distribution range along the optimal front.

In general, EMO algorithms, based on their selection mechanisms, can be divided into three groups: Pareto-based algorithms, aggregation-based algorithms, and indicator-based algorithms (Coello, 2011; Wagner et al., 2007).

The main idea of Pareto-based algorithms is to compare individuals of a population based on their Pareto dominance relation and distribution. The Pareto dominance relation is used to distinguish individuals in terms of convergence, and the distribution is used to maintain the diversity of individuals in the population. Many effective EMO algorithms belong to this group. Among them, NSGA-II (Deb et al., 2002) and SPEA2 (Zitzler et al., 2002) are two representative algorithms.

In aggregation-based algorithms, the objectives are normally aggregated in some form (using either linear or nonlinear schemes), such that a single scalar value is generated. This scalar value is used as the fitness of the algorithm. In comparison with the algorithms in other groups, aggregation-based algorithms require a priori definition of relations among objective functions. As the earliest multi-objective optimization method that can be traced back to the middle of the last century (Kuhn and Tucker, 1951), the aggregation-based approach has become popular again in recent years, partially due to the appearance of an effective algorithm, MOEA/D (Zhang and Li, 2007).

The basic idea behind indicator-based algorithms is to employ a performance indicator to select individuals. One important characteristic of indicator-based algorithms is that in contrast to Pareto-based algorithms which compare individuals using two criteria (i.e., Pareto dominance relation and distribution), these algorithms adopt a single indicator to optimize a desired property of the evolutionary population. The algorithm IBEA (Zitzler and Künzli, 2004) is a pioneer in this group. Recently, some algorithms in this group, such as SMS-EMOA (Beume et al., 2007) and HypE (Bader and Zitzler, 2011), have been found to be promising in solving many-objective optimization problems (Wagner et al., 2007; Bader and Zitzler, 2011; Li et al., 2013).

This paper focuses on Pareto-based EMO algorithms. In these algorithms, the convergence of individuals in the population is estimated according to the Pareto dominance relation based fitness strategies, such as the dominance count (Fonseca and Fleming, 1995), strength (Zitzler et al., 2002), and dominance rank (Deb et al., 2002). However, such estimation depending fully on the Pareto dominance relation may lead to the existence of a large amount of incomparable individuals in the population due to the lack of the quantitative measure (see Farina and Amato, 2003; Ishibuchi et al., 2008; Yang et al., 2013). On the other hand, with respect to diversity, most algorithms only consider the crowding degree of individuals, but ignore the position of individuals in the population. In fact, the position of individuals also has an important influence on diversity since the uniformity and spread of the entire population need to be maintained (a detailed explanation is given in the latter part of Section 3.1).

In this paper, we develop a Euclidean minimum spanning tree (EMST) based EA (ETEA) to address the above issues. The aim of the paper is to employ the characteristics of EMSTs and the distance relation among individuals to balance the convergence, uniformity, and spread of the population during the evolutionary search. To this end, firstly, an EMST-based density estimator is proposed to measure the crowding degree and position of individuals in the population. Secondly, two distance-based measures incorporating the Pareto dominance relation are used to compare individuals in fitness assignment and environmental selection. Finally, three EMST-related indicators are applied to maintain the archive set when the number of non-dominated individuals exceeds the size of the set.

The EMST is a minimum spanning tree of a set of points in the space, where the weight of the edge between each pair of points is their Euclidean distance. In other words, an EMST connects a set of points in the space using lines in order to obtain the minimized total length of all the lines and reach any point from any others through the exclusive lines. EMSTs can be applied in a wide variety of domains, such as the network, piping, Euclidean traveling salesman problems, among others (Lee, 1999; Bansal and Ghanshani, 2006; Wieland et al., 2007; Šeda, 2008).

Since the generation of an EMST is entirely determined by the Euclidean distance between solutions (points), some properties in EMSTs generally have a close relationship with the distribution and position information of the solutions. For example,

  • Solutions which are distributed in more crowded regions have shorter edges;

  • The boundary solutions are often of low node degrees, yet some bridge-like solutions have high node degrees;

  • The line between an individual and its neighbor whose orientation is different from others may have a higher likelihood of becoming an edge of the EMST;

  • The EMST which is constructed by a non-dominated set in the 2-dimensional space degenerates into linear structure.

In this paper, we will employ these properties to deal with MOPs.

As a first attempt to capture and utilize the properties of EMSTs in EMO, we have recently developed a fitness assignment strategy and a diversity maintenance approach in Li et al. (2008). In view of encouraging experimental results of these preliminary studies, this paper conducts a further and thorough investigation along this line. In comparison with the previous work, the main contributions of this paper are summarized as follows.

  1. An elaborate fitness assignment scheme is designed, which takes a distance comparison relation between non-dominated individuals and dominated ones into account, instead of the simple distance evaluation in Li et al. (2008).

  2. A fitness adjustment technique is introduced to avoid partial overcrowding by penalizing the individuals once their neighbors have been picked out during the environmental selection process.

  3. An improved population truncation method is proposed to preserve the boundary solutions as well as to eliminate crowded solutions in the archive.

  4. Systematic experiments are carried out to compare ETEA with five state-of-the-art algorithms on 32 test problems; only NSGA-II and SPEA2 were used to validate the proposed algorithm on a few problems in Li et al. (2008). In addition, this paper also contains a comparative study between ETEA and its predecessor, an analytical and empirical study of computational cost, and an investigation of different parts of the proposed algorithm.

The rest of this paper is organized as follows. In Section 2, relevant notation and definitions are reviewed. Section 3 is devoted to the description of the proposed algorithm. Section 4 presents the algorithm settings, test functions, and performance metrics. Experimental results are presented and analyzed in Section 5. Finally, Section 6 concludes the paper and presents future work.

2  Definitions and Terminology

The concepts of Pareto optimality have been well understood in the literature. This section will introduce notation closely related to our work, such as extreme solutions and boundary solutions.

Without loss of generality, we suppose that an arbitrary MOP consists of m objectives, which are all to be minimized and equally preferable. A solution to this MOP can be described in terms of a decision vector in the decision space . A function evaluates the quality of a specific solution by assigning it an objective vector [] in the objective space .

Pareto optimality is defined by using the concept of dominance. Given two decision vectors a and b, a is said to dominate b (denoted as ), iff a is at least as good as b in all objectives and better in at least one objective. Accordingly, those decision vectors that are not dominated by any other vectors are denoted as Pareto optimal solutions. In general, the set of optimal solutions in the decision space is denoted as the Pareto set, and the corresponding set of objective vectors as the Pareto front. Unfortunately, it is often infeasible to obtain the Pareto set, and it is only hoped to find a good approximation of the set. Usually, we consider the nondominated set found in one run as the approximation.

Although the solutions in a non-dominated set are incomparable with each other on the basis of the Pareto dominance concept, their positions that affect the distribution range of the set can be well distinguished. Several concepts about the range of a non-dominated set are introduced as follows.

Definition 1 (Extreme Solutions):

The solutions in a non-dominated set have the maximum value for at least one objective.

The extreme solutions, which are used in numerous diversity maintenance strategies and performance assessment techniques, can partly reflect the extent of a non-dominated set. Especially for bi-objective problems, the extreme solutions play a decisive role in the distribution range. The greater the distance between two extreme solutions, the wider the distribution range of the non-dominated set. However, the extreme solutions fail to provide enough information to report the range of solutions for problems with more than two objectives. To this end, a concept of boundary solutions (Li and Zheng, 2009) has been presented to overcome this shortcoming. In order to define the boundary solutions, a comparison relation between individuals, called beyond, is first introduced as follows.

Definition 2 (Beyond):

A vector a is said to a vector b in the objective space , if for all and fj(a)>fj(b) for some .

Note that the definition of beyond is equal to that of the Pareto dominance relation regarding a maximization MOP. In the following, the definition of the boundary solutions in a non-dominated set is given according to the beyond relation between solutions in the set.

Definition 3 (Boundary Solutions in the Objective Space

and Boundary Solutions): A vector a in a non-dominated set S is considered as a boundary solution in the objective space (denoted as BSi), if a is not beyond by any member of S for the subset of all the objectives. A vector a is said to be a boundary solution of S if a is one of the vectors in .

The boundary solutions of a non-dominated set entirely determine its range. They are significantly different from extreme solutions, despite the fact that boundary solutions in general include extreme solutions and are even equal to extreme solutions on bi-objective problems. Figure 1 gives an example of boundary solutions and extreme solutions. A more detailed description and analysis can be found in Li and Zheng (2009).

Figure 1:

A tri-objective example of boundary solutions and extreme solutions of a Pareto front: (a) Pareto front, (b) boundary solutions, and (c) extreme solutions.

Figure 1:

A tri-objective example of boundary solutions and extreme solutions of a Pareto front: (a) Pareto front, (b) boundary solutions, and (c) extreme solutions.

3  Description of the Proposed Algorithm

ETEA is an EMO algorithm which utilizes the properties of EMSTs to solve MOPs. In this section, we first present the main loop of ETEA and a density estimator based on EMSTs. Then, we describe the fitness assignment process. Next, we introduce the fitness adjustment technique in environmental selection. Finally, a truncation strategy is given to maintain diversity in the archive.

3.1  Main Loop and Density Estimation

The main loop of ETEA is given in Algorithm 1. Clearly, the basic procedure of the algorithm is similar to general generational EMO algorithms except that a fitness adjustment strategy is added in environmental selection (shown in line 7). Most of the generational EMO algorithms (such as NSGA-II and SPEA2) directly select the best dominated individuals according to fitness information when the non-dominated individuals are not enough to fill the archive. A shortcoming of this strategy is that it may lead to the loss of diversity since neighboring individuals often have similar fitness values. The specific process of fitness adjustment will be described in Section 3.3. In addition, it should be pointed out that this paper only focuses on fitness assignment (line 4) and environmental selection which consists of elitism selection, fitness adjustment, and archive truncation (lines 5–10). In other words, mating selection and variation schemes in ETEA are not determined and can be freely selected by users. In the following, we present a density estimator which guides the search at different parts of the algorithm.
formula

Most EMO algorithms try to maintain diversity by incorporating density information into the selection process (see Horoba and Neumann, 2010): the higher the density of the surrounding area of an individual in a population, the lower the chance of the individual being selected. In other words, density estimation is needed in EMO algorithms to encourage uniform distribution of individuals over the current trade-off surface. In this paper, we employ the edges of an individual (node) in the EMST to estimate its distribution. An estimator, called the Euclidean minimum spanning tree crowding distance, is given here.

Definition 4 (Euclidean Minimum Spanning Tree Crowding Distance):
Let T be a Euclidean minimum spanning tree of a solution set P. For an individual of P, let denote the individuals sharing an edge with , where d is the number of edges attached to (i.e., the degree of node in the EMST; e.g., for node D in Figure 2, d = 3), and denote the length (weight) of the edge , i.e., the Euclidean distance between individuals and . The Euclidean minimum spanning tree crowding distance (ETCD) of is defined as follows:
formula
1
Figure 2:

An EMST of the set , where denotes the length of the edge between solutions and .

Figure 2:

An EMST of the set , where denotes the length of the edge between solutions and .

Clearly, the ETCD of an individual is the kth power mean of the length of all its edges, where k is equal to 0.5. For instance, in Figure 2, the density estimator of individual F is determined by LEF and LFG, and its ETCD is the 0.5th power mean of them. Here, assigning k the value 0.5 is a rough setting in order to obtain a tradeoff among the effects of the neighbors of with different distances. If k is set to 1.0 (i.e., ETCD is the arithmetic mean of edge weights), all neighbors of will have the same contribution to the density of no matter how far they are from , which partly hinders the development of uniformity of the population (see the example in the third observation of ETCD in the list following the next paragraph). Therefore, a value of k lower than 1.0 may be suitable for emphasizing the effect of the closer neighbors. However, when k approximates 0, almost only the closest neighbor will contribute to the density of , which apparently ignores the effects of the other neighbors. Therefore, we simply set k to the middle value between 0 and 1. In fact, other values between 0 and 1 can also be adopted as long as they are away from the boundaries 0 and 1.

Similar to other density estimators, the effectiveness and characteristics of ETCD rely heavily on the properties of the assessment technique, since different techniques will lead to different judgments on density estimation. From the calculation of the proposed estimator, we can draw some observations as follows.

  1. In accordance with the greediness of the procedure of constructing an EMST, the edge between an individual and its closest neighbor (i.e., the individual which has the shortest Euclidean distance to it) belongs to the EMST. Accordingly, from the second shortest edge to others, they generally have a decreasing chance to become a component of the EMST.

  2. According to the connectivity of an EMST, the line between an individual and its neighbor whose orientation is different from others may have a higher likelihood of becoming an edge of the EMST. For example, in Figure 2, for individual B and its neighbors A and C, the line between B and A belongs to EMST in contrast to the line between B and C, although the former is longer than the latter. This is because relative to B, A has a different orientation against other neighbors around B; yet there exists a closer neighbor (D) who has a similar orientation to C with regard to B. Moreover, the second behavior derived from the connectivity of an EMST is that some bridge-like individuals that connect two clusters of individuals have higher ETCD values. For example, individuals D and E in Figure 2 may be regarded as intermediate individuals joining two clusters ( and ). For individual D, clearly, a relatively higher ETCD value is obtained since LDE is included in the calculation of the estimator. In summary, from the above discussion, it becomes clear that the proposed estimator prefers the individuals which can be regarded as an intermediate connection for other members in the population. This phenomenon seems to be consistent with the target of advancing the uniformity of distribution. This is because these intermediates, in contrast to their neighbors, are often located closer to other individuals (or clusters), and thus their offspring have a higher likelihood of filling the empty areas between them and those individuals (or clusters).

  3. Note that the definition of ETCD is slightly different from that of the density estimator in Li et al. (2008). In Li et al. (2008), the density estimator was defined by calculating the arithmetic mean of edge weights. Here, the 0.5th power mean is used to replace the arithmetic mean for improving uniformity. For example, consider individuals B and F in Figure 2 regarding the two density estimators, and assume LAB=9.0, LBD=1.0, LEF=5.0, and LFG=5.0. Clearly, according to Li et al. (2008), the estimation value of B (5.0) is equal to that of F (5.0); yet for ETCD, B performs worse than F (4.0 against 5.0).

The main difference between ETCD and other density estimators is that ETCD not only reflects the crowding degree but partly implies the relative orientation and position information of individuals. Yet most of the existing density estimators (such as the niche techniques, Horn et al., 1994; Tan et al., 2001; Shir et al., 2010; crowding distance, Deb et al., 2002; Nebro et al., 2008; kth nearest neighbor, Zitzler et al., 2002; Elhossini et al., 2010; and grid crowding degree, Corne et al., 2001; Yen and Lu, 2003; Li et al., 2010) only evaluate the density information of individuals. Although these strategies seem to be reasonable, they may be imprecise due to the influence of individuals’ position: the individuals located on or near the border of a population usually have a lower crowding degree; some bridge-like individuals, which are of great service to uniformity, may be distributed in the region with a high crowding degree. For example, considering individual D in Figure 2, it may be assigned a high density value by some estimators (e.g., the niche techniques, crowding distance, kth nearest neighbor, and grid degree), thus being eliminated early. However, as previously discussed, individual D is important in the context of maintaining uniformity and can be regarded as an intermediate individual connecting two clusters and .

3.2  Fitness Assignment

In order to evolve a population toward the optimum as well as to diversify its individuals uniformly along the obtained trade-off surface, the fitness value of individuals should be assigned to reflect both convergence and diversity accordingly. At present, most studies on fitness assignment mainly focus on the issue of the Pareto dominance relation, such as the dominance count, strength, dominance rank, and others (Bosman and Thierens, 2003; Li, 2003; Gong et al., 2008). In this paper, we prefer the distance from individuals to the obtained trade-off surface. We consider the distance differences among some specific individuals and record the successful counts of them (called the distance count here). In detail, with respect to the distance count, we distinguish between non-dominated individuals and dominated ones. For a non-dominated individual, its distance count is assigned to zero. For a dominated individual, denoted as individual i, the non-dominated individual j which dominates and is the closest to i is first selected. Then, the distance count of i is determined by the total number of the non-dominated individuals whose distance from j is shorter than the distance between i and j:
formula
2
where
formula
3
where denotes the cardinality of a set, Lij implies the distance from i to j, and DS and NDS represent the set of dominated and non-dominated solutions, respectively. Here, the distance count is minimized, and for dominated individuals, it is penalized by adding one in order to guarantee that they have a larger value than non-dominated individuals. For example, let us consider dominated individual A in Figure 3. First, individual F is selected since it is the non-dominated individual which dominates and is the closest to A. Then, we look for non-dominated individuals which can contribute to the distance count of A. Here, only individual G is qualified, considering that its distance from F is shorter than the distance between A and F. Thus, the distance count of A is . To better understand the characteristics of this scheme, an example of the distance count in comparison with two well-known strategies (the dominance rank and strength) in NSGA-II and SPEA2 is illustrated in Figure 3.
Figure 3:

Comparison of fitness assignment strategies for a minimization bi-objective problem. The numbers in the parentheses associated with the dominated solutions correspond to the distance count, dominance rank, and strength in ETEA, NSGA-II, and SPEA2, respectively. The dashed lines connect the dominated solutions to their corresponding non-dominated solutions in the distance count calculation.

Figure 3:

Comparison of fitness assignment strategies for a minimization bi-objective problem. The numbers in the parentheses associated with the dominated solutions correspond to the distance count, dominance rank, and strength in ETEA, NSGA-II, and SPEA2, respectively. The dashed lines connect the dominated solutions to their corresponding non-dominated solutions in the distance count calculation.

Clearly, the distance count of dominated individuals is mainly determined by two factors: (1) their distance from the non-dominated front, and (2) the distance between the corresponding non-dominated individual and other ones. An individual with a poor distance count means that it is far away from the non-dominated front, or the non-dominated individual in the population who is the closest to and dominates it is located in a crowded region. In Figure 3, individual D illustrates the first factor: it is located far away from the non-dominated front, thereby obtaining a high distance count; on the other hand, individual C provides an example for the second factor: since its corresponding non-dominated individual is located in a crowded region, C is assigned a relatively high distance count value even if it approximates the non-dominated front. However, the other two strategies (depending on dominance information) are not able to effectively distinguish this case.

It is worthwhile to mention that a significant difference between ETEA and other strategies is that ETEA places more emphasis on the distribution of non-dominated individuals, since its fitness strategy takes into account the distance measurement among individuals. Actually, non-dominated individuals play a crucial role in the selection process of EMO. The non-dominated front that is composed of these individuals can largely determine the search direction and reflect the evolution bias in distinct areas. Therefore, a non-dominated front with uniformly and widely distributed individuals is considerably important and able to drive the whole population toward the desired direction. Naturally, some dominated individuals who have a high likelihood of achieving this target (i.e., they are located near the sparse regions of the non-dominated front) should be assigned better fitness values even if they are dominated by some individuals, such as individual A in Figure 3.

Although the distance count provides elaborate preference information for dominated individuals, it fails when most individuals in the population do not dominate each other because it is equal to zero for all non-dominated individuals. In addition, the density information of each individual in the population cannot also be directly reflected according to the distance count. Therefore, we incorporate ETCD into the fitness in order to discriminate the individuals who have identical distance count as well as to provide a density indicator for each individual. Here, we take the inverse of ETCD in accordance with the minimization of the distance count value. Accordingly, the fitness of individual i is defined as follows:
formula
4

In the fraction of Equation (4), one is added to the denominator to ensure that its value is greater than zero and smaller than or equal to one. As a result, the fitness for non-dominated individuals is within the range of (0, 1], and for dominated individuals larger than one.

3.3  Fitness Adjustment

Mating selection and environmental selection are two indispensable parts of an EMO algorithm. Although both of them are based on fitness information of individuals, they are, in principle, fully independent of each other. Mating selection aims at picking promising individuals for variation and is usually performed in a random way. In contrast, environmental selection determines which of the previously stored individuals and the newly created ones are kept in the archive (Zitzler et al., 2004).

Unfortunately, most current EMO algorithms, such as NSGA-II and SPEA2, do not distinguish this difference and often directly perform the selection operation according to the straightforward fitness rank of individuals. In fact, in contrast to mating selection, where the directly-selected way seems to be reasonable due to the randomness of the selection, the environmental selection based on the straightforward fitness rank may reduce the diversity of the archive because of the deterministic way in which individuals move into the archive, ordered by their level of fitness. Since the fitness value of individuals depends on their position compared with other individuals in the population, those individuals that are closely located often have similar values. Therefore, it is very likely that they are eliminated or preserved simultaneously, which may bring about individuals crowded in some regions yet produce vacancies in other regions.

In this study, we propose a fitness adjustment strategy in environmental selection. The individuals are penalized once their neighbors have been selected into the archive. Specifically, we consider the circle centered at the selected individual as its neighborhood whose range is determined by the distance between it and the non-dominated front. For individuals in the neighborhood, a hierarchical fitness penalty is executed according to their distance from the center individual. It should be noted that this adjustment occurs when the non-dominated individuals are not enough to fill the archive, and it only aims at the dominated individuals. Algorithm 2 gives the detailed procedure of this fitness adjustment strategy.
formula

In Algorithm 2, Function Findout_neighbor(R, p) (line 4) is designed to find out the neighbors of the current best dominated individual p in population R. The neighborhood radius is defined by the distance from the center individual (i.e., the selected individual) to its nearest non-dominated individual who dominates it. Lines 6–8 of the algorithm inflict a fitness penalty on the neighbors of the selected individual. The penalty degree of individuals relies on the crowding degree reflected by the total number of individuals in the neighborhood as well as on the distance between them and the center. Therefore, a more crowded neighborhood leads to a higher overall penalty; and for each individual, the further it is from the center, the milder the penalty.

An example of fitness adjustment is illustrated in Figure 4. It is clear that the penalty mechanism in ETEA largely avoids crowding in the archive, because once an individual is picked out, its neighbors will be penalized (see Figure 4(a)–(d)). However, the selection strategies in NSGA-II and SPEA2, which are directly performed according to the fitness of individuals, reduce the diversity to some extent. Specifically, for NSGA-II, since individuals AF have the same dominance rank, the three most crowded individuals C, D, and E will be eliminated. As to SPEA2, since the calculation of fitness of an individual is based on the strength of the individuals that dominate it, individuals B, C, and F, which are dominated by the individuals that have larger strength values, will be eliminated.

Figure 4:

A scenario of the fitness adjustment procedure in ETEA and its result compared with that of NSGA-II and SPEA2. (a) Original set R. (b) D-eliminated set R. (c) A-eliminated set R. (d) Final archive of ETEA. (e) Final archive of NSGA-II. (f) Final archive of SPEA2. Where Select_num=3. A, B, C, D, E, and F are the candidate dominated individuals. (a)–(c) are the fitness adjustment procedure in environmental selection of ETEA; (d)–(f) are the final archive results by the environmental selection process of the three algorithms. The number in the parentheses associated with each candidate individual means the integral part of the fitness value (i.e., the distance count) in ETEA. The circle corresponds to the neighborhood of the current best dominated individual.

Figure 4:

A scenario of the fitness adjustment procedure in ETEA and its result compared with that of NSGA-II and SPEA2. (a) Original set R. (b) D-eliminated set R. (c) A-eliminated set R. (d) Final archive of ETEA. (e) Final archive of NSGA-II. (f) Final archive of SPEA2. Where Select_num=3. A, B, C, D, E, and F are the candidate dominated individuals. (a)–(c) are the fitness adjustment procedure in environmental selection of ETEA; (d)–(f) are the final archive results by the environmental selection process of the three algorithms. The number in the parentheses associated with each candidate individual means the integral part of the fitness value (i.e., the distance count) in ETEA. The circle corresponds to the neighborhood of the current best dominated individual.

3.4  Archive Truncation

As described in Algorithm 1, the first step in environmental selection is to copy all non-dominated individuals into the archive. If there are still a certain number of available slots in the archive, some best dominated individuals will fill the archive according to the fitness adjustment strategy in the previous section. If the size of these non-dominated individuals exceeds the upper bound of the archive, an archive truncation procedure is activated to remove some individuals for obtaining a representative archive. However, obtaining a representative archive is not a trivial task, since both properties of distribution (i.e., uniformity and spread) are supposed to be taken into account. On the one hand, as a non-dominated front can be a convex, non-convex, disconnected, or piecewise continuous hypersurface, the difficulty may arise regarding how to maintain its proper distribution shape. On the other hand, the boundary effect will emerge when the uniformity of a non-dominated set is considered (Farhang-Mehr and Azarm, 2002). The number of the neighbors of outer individuals is generally less than that of inner ones, even if they have a higher crowding degree. This may result in a misleading estimation of individuals’ density. In addition, the reasonable integration of both properties (uniformity and spread) into one truncation method is also a noticeable issue. The improvement of performance at one point should not cause a simultaneous degradation at the other point.
formula

In this study, we propose an archive truncation strategy by employing the EMST to maintain uniformity and spread. The pseudocode is given in Algorithm 3. The main procedure of the truncation includes three steps. Firstly, an edge with the minimum weight is found in the EMST (line 4), and the two endpoints of the edge are regarded as the candidate individuals to be considered. Secondly, the degree property is introduced to determine their survival (lines 5–10). If the degree value of one candidate is equal to one, the other candidate is eliminated (according to the connectivity of an EMST, there should not be two candidates whose degree is one unless the size of the set is equal to two). Finally, if the degree values of both candidates are larger than one, the candidate with a higher ETCD value is preferable (lines 11–18). Note that the original ETCD of candidates has been slightly modified here: the edge with the minimum weight is removed in the calculation of ETCD. A detailed analysis with regard to this modification will be presented in the last part of this section.

Figure 5 shows an illustration of the truncation procedure for a tri-objective non-dominated set. Firstly, an EMST of the original non-dominated set is generated, and then the shortest edge LAB is found. Individual B is eliminated since the degree of A is equal to one. And again a new EMST of the remaining individuals is generated, and similarly candidates C and E are found. E is eliminated because (1) the degrees of both candidates are greater than one, and (2) the modified ETCD of C is larger than that of E (i.e., the length of edge LCA is larger than the 0.5th power mean of the length of edges LEG and LED). This procedure is repeated until a predefined size is achieved. The final individuals in the archive are A, G, and H. Clearly, by continuous truncation in the archive, the two properties of distribution can be reasonably tuned, and a well-extended and uniformly-distributed non-dominated front will be obtained. More specifically, from the algorithm and illustration of archive truncation, we can draw some in-depth observations as follows.

  1. Duplicate individuals, if they exist, will first be eliminated. This is because the edge weight between them is equal to zero in an EMST, and thus they would be selected to become the candidates first.

  2. The comparison of degree information in the truncation strategy can be considered as a reasonable integration of the two properties of distribution, since it not only reflects the density of individuals but partly implies their position in the population. On the one hand, an individual of degree one, in general, means that it has a loose relationship with the surrounding individuals according to the property of EMSTs. Thus, its crowding extent is generally lower than that of the other individual sharing an edge. Obviously, preserving these individuals may be beneficial to the uniformity of distribution, in comparison with preserving their corresponding opponent. On the other hand, the boundary solutions (defined in Section 2) have a high likelihood of being preserved according to the degree comparison scheme. This is because they are located in the outer part of the population, and not all the orientations around them are with individuals, that is, only part of orientations may affect their degree. Therefore, for them, the probability of the degree equal to one is higher than that for the inner individuals. Figure 6 makes a statistical comparison between the boundary solutions and non-boundary solutions regarding the case that their degree is equal to one, considering 100 randomly generated non-dominated vectors (solutions) in the multidimensional space. It is clear that the probability (>70%) of the case that the individual of degree one belongs to the boundary solutions is significantly larger than the probability (<30%) of the case that it belongs to the non-boundary solutions, especially in a lower dimension space. It is interesting to note that the probability reaches 100% in the two-dimensional space. This is because the EMST generated by two-dimensional non-dominated solutions is linear, and thus only two boundary solutions whose degree is equal to one exist.

  3. When the degree of both candidates is larger than one, the modified ETCD, which takes into account their non-sharing edges, is introduced to determine their survival. In other words, we estimate the density of the two candidates by considering all individuals, except the closest one, connecting the candidates. This modification seems to be reasonable. In fact, there is always one candidate to be eliminated no matter how close the two candidates are, that is, the edge formed by them will not appear in the next round of truncation. Therefore, considering the effects of this edge is meaningless and may even lead to some erroneous judgments on their distributions. As the edge LCD in Figure 5(d), the original ETCD of individual C is larger than that of individual D, and thus D will be eliminated. Obviously, this operation decreases the uniformity of solutions in the archive, compared to the result in Figure 5(e) obtained by the modified ETCD.

Figure 5:

An example of the archive truncation process on a tri-objective non-dominated set, where the archive size is set to 3. (a) Original non-dominated set, (b) B-eliminated set, (c) E-eliminated set, (d) F-eliminated set, (e) C-eliminated set, and (f) D-eliminated set (i.e., the final individuals in the archive).

Figure 5:

An example of the archive truncation process on a tri-objective non-dominated set, where the archive size is set to 3. (a) Original non-dominated set, (b) B-eliminated set, (c) E-eliminated set, (d) F-eliminated set, (e) C-eliminated set, and (f) D-eliminated set (i.e., the final individuals in the archive).

Figure 6:

The percentage of the case that an individual of degree one is the boundary solution (BS), where the total case satisfies that, for a pair of individuals (i.e., two individuals sharing an edge in the EMST), one and only one belongs to BS, and one of them has the degree equal to one. The EMST is constructed by 100 non-dominated vectors which are randomly generated in the k-dimensional unit hypercube [0, 1]k, where k=2, 3, 4, 5.

Figure 6:

The percentage of the case that an individual of degree one is the boundary solution (BS), where the total case satisfies that, for a pair of individuals (i.e., two individuals sharing an edge in the EMST), one and only one belongs to BS, and one of them has the degree equal to one. The EMST is constructed by 100 non-dominated vectors which are randomly generated in the k-dimensional unit hypercube [0, 1]k, where k=2, 3, 4, 5.

4  Experimental Design

This section is devoted to designing an experiment scheme for performance validation of ETEA. First, we briefly introduce the set of MOPs which will be used as the benchmark for this experiment. Then, two popular metrics are described to give an appropriate performance evaluation for algorithms. Finally, a general experimental setting is presented for the comparison between ETEA and the other six EMO algorithms.

4.1  Test Problems

In this section, we describe different sets of test problems according to the number of objectives. These problems have been commonly used in the literature.

For the bi-objective problem set, we firstly choose problems from Van Veldhuizen's studies (Van Veldhuizen, 1999), including Schaffer1, Schaffer2, Fonseca, Kursawe, and Poloni. Then, the ZDT problem family, including ZDT1, ZDT2, ZDT3, ZDT4, and ZDT6 (Zitzler et al., 2000), is considered. Finally, the WFG problem family (WFG1 to WFG9) (Huband et al., 2006), based on variable linkages, is included. For the tri-objective problem set, three Viennet problems (Viennet1, Viennet2, and Viennet3; Van Veldhuizen, 1999) and the DTLZ problem family (DTLZ1 to DTLZ7; Deb, Thiele, et al., 2005) are chosen. Moreover, three recent tri-objective problems (called the UF problems; Zhang et al., 2009) which emphasize the complexity of the shapes of the Pareto set are taken into account as well. All the problems have been configured as in the original papers where they were described.

4.2  Performance Metrics

To compare the performance of the selected algorithms, we introduce two widely-used quality metrics, hypervolume (HV; Zitzler and Thiele, 1999) and inverted generational distance (IGD; Zhang et al., 2008), which can give a comprehensive assessment in terms of convergence, uniformity, and spread. The HV metric is a very popular quality metric due to its compliance with the Pareto dominance relation (see Zitzler et al., 2003). HV calculates the volume of the objective space between the obtained solution set and a reference point, and a larger value is preferable. On the other hand, IGD measures the average distance from the points in the Pareto front to their closest solution in the obtained set. A low IGD value indicates that the obtained solution set is close to the Pareto front and also has good distribution uniformity and range.

The main difference between IGD and HV is that, for the former, the Pareto front of problems must be known, and yet for the latter, a reference point that may bring about some effects on the performance judgment has to be chosen appropriately. In addition, the preference between uniformity and spread for the two metrics is also distinct. The IGD metric, which is based on uniformly-distributed points along the whole Pareto front, prefers the uniformity of the obtained solution set; while the HV metric, with significant contributions from the boundary solutions, has a bias toward the extent of the set.

4.3  General Experimental Setting

In order to validate the performance of ETEA, we compare it with six EMO algorithms: NSGA-II (Deb et al., 2002), SPEA2 (Zitzler et al., 2002), IBEA (Zitzler and Künzli, 2004), ε-MOEA (Deb, Mohan, et al., 2005), TDEA (Karahan and Köksalan, 2010), and MST-MOEA (i.e., the predecessor of ETEA; Li et al., 2008). NSGA-II1 is one of the most popular EMO algorithms. The main characteristic of NSGA-II is its fast non-dominated sorting and crowding distance-based density estimation. SPEA22 is also a prevalent EMO algorithm, which borrows a so-called fitness strength value and the kth nearest neighbor to select individuals into the next population. In recent years, some indicator-based EMO algorithms have also found to be competitive in balancing convergence and diversity. Here, we select a representative indicator-based algorithm IBEA to make a comparative study. IBEA3 aims to integrate the preference information of the decision-maker into multi-objective search. The main idea is to define the optimization goal in terms of a binary performance measure and then to directly use this measure in the mating and environmental selection processes. ε-MOEA4 is a steady-state algorithm that typically creates only one new member that is tested to enter the population at each step of the algorithm (see Kumar and Rockett, 2002; Igel, Suttorp, et al., 2007; Durillo et al., 2009). ε-MOEA uses a grid-based strategy and divides the objective space into hyperboxes by the size of ε. Each hyperbox can contain at most a single individual, thus preventing crowding. However, due to the feature of ε-dominance, the boundary solutions may be lost in the evolutionary process (Hernández-Díaz et al., 2007; Karahan and Köksalan, 2010). Similar to ε-MOEA, TDEA5 is also a grid-based steady-state algorithm. It defines a territory around an individual to maintain diversity. Its main difference against ε-MOEA is that the hyperboxes of TDEA are based on individuals rather than independent of them. The comparative study in Karahan and Köksalan (2010) shows its competitiveness in comparison with some state-of-the-art EMO algorithms. MST-MOEA is the first EMO algorithm that is designed based on the EMST. Although both MST-MOEA and ETEA algorithms employ the properties of EMSTs to enhance the performance of algorithms, they are of great difference in fitness assignment, environmental selection, and archive truncation. In the following, the experimental setting for the comparative study of these algorithms is listed.

  • Parameter Setting for Crossover and Mutation. All selected EMO algorithms are given real-valued decision variables. Two widely-used crossover and mutation operators, simulated binary crossover (SBX) and polynomial mutation (Deb, 2001), are chosen. Following the practice in Deb et al. (2002), the distribution indexes in both SBX and the polynomial mutation are set to 20. A crossover probability pc=1.0 and a mutation probability pm=1/n (where n is the number of decision variables) are used according to Deb (2001).

  • Population and Archive Size. Like most of the studies of EMO algorithms, for generational algorithms the population size is set to 100, and the archive is also maintained at the same size if it exists (Coello et al., 2007). For steady-state algorithms, the regular population size is set to 100 according to Deb, Mohan, et al. (2005).

  • Number of Runs and Stopping Condition. We independently run each algorithm 50 times for each test problem. The termination criterion of the algorithms is a predefined number of evaluations. Here, we set the evaluation number to different values for problems with different numbers of objectives, since the difficulty of problems generally increases with the number of objectives (Brockhoff et al., 2009; Schütze et al., 2011). Similar to the experimental studies in Deb, Mohan, et al. (2005) and Beume et al. (2007), the algorithms are assigned a larger number of evaluations for tri-objective problems than for bi-objective ones, that is, 30,000 against 25,000.

  • Parameter Settings in IBEA, ε-MOEA, and TDEA. In IBEA, the parameter is set to 0.05 as recommended in Zitzler and Künzli (2004). ε-MOEA and TDEA require the user to set the size of hyperboxes in grid (i.e., ε and ). In order to guarantee a fair comparison, we set them so that the archive of the two algorithms is approximately of the same size as that of the other algorithms (given in Table 1).

  • Reference Point Setting in HV. In the calculation of the HV metric for a solution set, choosing a reference point that is slightly larger than the worst value of each objective on the Pareto front is found to be suitable, since the effects of convergence and diversity of the set can be well balanced (Knowles, 2006; Auger et al., 2009). Here, as suggested in Kukkonen and Deb (2006), we select the integer point slightly larger than the worst value of each objective on the Pareto front of a problem as its reference point. As a consequence, the reference points for SCH1, SCH2, FON, KUR, and POL is (5, 5), (2, 17), (2, 2), (−14, 1), and (0, 1), respectively. The reference points used in all the ZDT and WFG problems is (2, 2) and (3, 5), respectively, and for VNT1, VNT2, and VNT3 is (5, 6, 5), (5, −16, −12), and (9, 18, 1), respectively. The reference point for the tri-objective DTLZ and UF problems is (2, 2, 2), except (1, 1, 1) for DTLZ1 and (2, 2, 7) for DTLZ7. Note that the solutions that do not dominate the reference point are discarded in the HV calculation (i.e., the solutions that are worse than the reference point in at least one objective will contribute zero to HV).

  • Substitution of the Pareto Front for IGD. For the IGD metric, it is necessary to know the Pareto front of test problems. In most of the test problems used in this study, their Pareto fronts are known (families ZDT, DTLZ, WFG, and UF). For them we select 10,000 evenly-distributed points along the Pareto front as its substitution in the calculation of IGD since they can accurately represent the true Pareto front (Sen and Yang, 1998). For other test problems, the substitution of their Pareto fronts is available at the website http://www.cs.cinvestav.mx/emoobook/.

  • Operating Environment. The hardware used in the comparison experiments is a PC with 2.8 GHz Pentium 4 CPU with a memory of 1 GB, and the operating system is Windows XP. The code of ETEA and MST-MOEA is written in C.

Table 1:
Parameter settings of ε-MOEA and TDEA.
SCH1SCH2FONKURPOLZDT1ZDT2ZDT3
ε 0.0200 0.0180 0.0028 0.0350 0.0400 0.0076 0.0076 0.0030 
 0.0110 0.0075 0.0130 0.0080 0.0080 0.0090 0.0090 0.0070 
 ZDT4 ZDT6 WFG1 WFG2 WFG3 WFG4 WFG5 WFG6 
ε 0.0075 0.0065 0.0070 0.0040 0.0200 0.0160 0.0160 0.0160 
 0.0075 0.0060 0.0030 0.0070 0.0076 0.0100 0.0100 0.0100 
 WFG7 WFG8 WFG9 VNT1 VNT2 VNT3 DTLZ1 DTLZ2 
ε 0.0160 0.0110 0.0160 0.1000 0.0070 0.0110 0.0340 0.0630 
 0.0100 0.0070 0.0100 0.0800 0.0260 0.0200 0.0600 0.1050 
 DTLZ3 DTLZ4 DTLZ5 DTLZ6 DTLZ7 UF8 UF9 UF10 
ε 0.0630 0.0150 0.0050 0.0300 0.0500 0.0150 0.0200 0.0050 
 0.0200 0.0400 0.0110 0.0250 0.0600 0.0850 0.0700 0.0100 
SCH1SCH2FONKURPOLZDT1ZDT2ZDT3
ε 0.0200 0.0180 0.0028 0.0350 0.0400 0.0076 0.0076 0.0030 
 0.0110 0.0075 0.0130 0.0080 0.0080 0.0090 0.0090 0.0070 
 ZDT4 ZDT6 WFG1 WFG2 WFG3 WFG4 WFG5 WFG6 
ε 0.0075 0.0065 0.0070 0.0040 0.0200 0.0160 0.0160 0.0160 
 0.0075 0.0060 0.0030 0.0070 0.0076 0.0100 0.0100 0.0100 
 WFG7 WFG8 WFG9 VNT1 VNT2 VNT3 DTLZ1 DTLZ2 
ε 0.0160 0.0110 0.0160 0.1000 0.0070 0.0110 0.0340 0.0630 
 0.0100 0.0070 0.0100 0.0800 0.0260 0.0200 0.0600 0.1050 
 DTLZ3 DTLZ4 DTLZ5 DTLZ6 DTLZ7 UF8 UF9 UF10 
ε 0.0630 0.0150 0.0050 0.0300 0.0500 0.0150 0.0200 0.0050 
 0.0200 0.0400 0.0110 0.0250 0.0600 0.0850 0.0700 0.0100 

5  Results and Discussion

This section validates the performance of ETEA according to the experimental design in the previous section. Firstly, we evaluate the proposed algorithm and compare it with five state-of-the-art EMO algorithms: NSGA-II, SPEA2, IBEA, ε-MOEA, and TDEA. Secondly, we analyze the time complexity of the proposed algorithm and show the computational cost of all the considered algorithms. Then, a comparative study between ETEA and its predecessor (MST-MOEA) is presented. Finally, we investigate the different parts of the proposed algorithm and identify their contribution to the performance of the algorithm.

5.1  Performance Comparison

In order to systematically present the results, the test problems have been grouped into two categories according to the number of their objectives. For each problem, we executed 50 independent runs. The values included in the tables of results are mean and standard deviation. The best mean for each problem has a gray background, as shown in Table 2. In addition, a t-test at a .05 significance level has been used to compare ETEA with its competitors. Symbols and indicate that the p value of 98 DOF is significant at a .05 level by a two-tailed t-test. The symbol indicates that ETEA is better than its competitor, and means the opposite.

Table 2:
The HV comparison of the six EMO algorithms on bi-objective problems.
ProblemETEANSGA-IISPEA2IBEAε-MOEATDEA
SCH1 2.2275e+1 (6.47e−4) 2.2271e+1 (1.57e−3) 2.2274e+1 (7.38e−4) 2.2272e+1 (1.07e−3) 2.2229e+1 (1.07e−3) 2.2270e+1 (2.02e−3) 
SCH2 3.8259e+1 (2.12e−3) 3.8246e+1 (3.84e−3) 3.8258e+1 (2.41e−3) 3.7981e+1 (1.32e−1) 3.8121e+1 (1.34e−3) 3.8219e+1 (2.98e−2) 
FON 3.0621e+0 (1.80e−4) 3.0618e+0 (1.84e−4) 3.0620e+0 (1.16e−4) 3.0608e+0 (1.52e−4) 3.0595e+0 (6.72e−4) 3.0553e+0 (5.03e−3) 
KUR 3.7072e+1 (1.05e−2) 3.7005e+1 (1.42e−2) 3.7064e+1 (1.15e−2) 3.6662e+1 (5.63e−2) 3.7068e+1 (1.46e−2) 3.7050e+1 (2.13e−2) 
POL 7.5316e+1 (4.33e−2) 7.5267e+1 (5.59e−2) 7.5326e+1 (9.72e−2) 6.0192e+1 (1.20e+0) 7.0977e+1 (7.32e−2) 7.4259e+1 (6.56e−1) 
ZDT1 3.6601e+0 (3.92e−4) 3.6591e+0 (4.10e−4) 3.6594e+0 (4.72e−4) 3.6590e+0 (8.02e−4) 3.6476e+0 (1.73e−3) 3.6566e+0 (1.76e−3) 
ZDT2 3.3260e+0 (6.68e−4) 3.3250e+0 (5.79e−4) 3.3248e+0 (8.92e−4) 3.3239e+0 (2.75e−4) 3.3230e+0 (1.60e−3) 3.3191e+0 (3.05e−3) 
ZDT3 4.8131e+0 (4.47e−4) 4.8124e+0 (4.66e−4) 4.8118e+0 (5.16e−4) 4.8062e+0 (2.11e−4) 4.8094e+0 (1.11e−3) 4.8035e+0 (3.61e−1) 
ZDT4 3.6514e+0 (7.73e−3) 3.6506e+0 (7.75e−3) 3.6500e+0 (8.68e−3) 2.4820e+0 (2.09e−1) 3.6350e+0 (2.16e−2) 3.6307e+0 (4.11e−2) 
ZDT6 3.0242e+0 (2.58e−3) 3.0219e+0 (2.72e−3) 3.0230e+0 (2.10e−3) 3.0365e+0 (5.70e−4) 3.0281e+0 (2.22e−3) 3.0236e+0 (2.68e−3) 
WFG1 7.2435e+0 (1.26e+0) 7.6348e+0 (9.96e−1) 7.5547e+0 (9.22e−1) 7.0554e+0 (9.81e−1) 5.6509e+0 (7.41e−1) 5.4301e+0 (8.11e−1) 
WFG2 1.1151e+1 (4.17e−1) 1.1001e+1 (4.16e−1) 1.0952e+1 (4.09e−1) 1.0947e+1 (4.08e−1) 1.0914e+1 (4.02e−1) 1.0858e+1 (3.83e−1) 
WFG3 1.0944e+1 (5.11e−3) 1.0934e+1 (7.03e−3) 1.0940e+1 (5.52e−3) 1.0941e+1 (3.09e−3) 1.0926e+1 (8.49e−3) 1.0917e+1 (1.35e−2) 
WFG4 8.6679e+0 (7.34e−3) 8.6676e+0 (4.01e−3) 8.6674e+0 (4.93e−3) 8.6671e+0 (1.64e−3) 8.6574e+0 (1.16e−2) 8.6507e+0 (1.50e−2) 
WFG5 8.1575e+0 (3.00e−2) 8.1586e+0 (3.45e−2) 8.1531e+0 (3.49e−2) 8.1953e+0 (4.80e−2) 8.1283e+0 (2.12e−2) 8.1219e+0 (2.74e−2) 
WFG6 8.5708e+0 (1.07e−1) 8.5381e+0 (1.46e−1) 8.5088e+0 (1.57e−1) 8.4984e+0 (1.97e−1) 8.4775e+0 (1.71e−1) 8.4326e+0 (2.24e−1) 
WFG7 8.6703e+0 (6.64e−3) 8.6701e+0 (3.02e−3) 8.6689e+0 (6.54e−3) 8.6675e+0 (1.50e−3) 8.6612e+0 (1.05e−2) 8.6488e+0 (1.44e−2) 
WFG8 7.0008e+0 (3.61e−1) 7.1049e+0 (4.50e−1) 6.9988e+0 (4.42e−1) 6.9244e+0 (4.03e−1) 6.8374e+0 (3.25e−1) 6.7856e+0 (2.60e−1) 
WFG9 8.4377e+0 (1.57e−2) 8.4327e+0 (1.71e−2) 8.4328e+0 (1.48e−2) 8.4435e+0 (2.12e−2) 8.4143e+0 (2.29e−2) 8.4065e+0 (2.27e−2) 
ProblemETEANSGA-IISPEA2IBEAε-MOEATDEA
SCH1 2.2275e+1 (6.47e−4) 2.2271e+1 (1.57e−3) 2.2274e+1 (7.38e−4) 2.2272e+1 (1.07e−3) 2.2229e+1 (1.07e−3) 2.2270e+1 (2.02e−3) 
SCH2 3.8259e+1 (2.12e−3) 3.8246e+1 (3.84e−3) 3.8258e+1 (2.41e−3) 3.7981e+1 (1.32e−1) 3.8121e+1 (1.34e−3) 3.8219e+1 (2.98e−2) 
FON 3.0621e+0 (1.80e−4) 3.0618e+0 (1.84e−4) 3.0620e+0 (1.16e−4) 3.0608e+0 (1.52e−4) 3.0595e+0 (6.72e−4) 3.0553e+0 (5.03e−3) 
KUR 3.7072e+1 (1.05e−2) 3.7005e+1 (1.42e−2) 3.7064e+1 (1.15e−2) 3.6662e+1 (5.63e−2) 3.7068e+1 (1.46e−2) 3.7050e+1 (2.13e−2) 
POL 7.5316e+1 (4.33e−2) 7.5267e+1 (5.59e−2) 7.5326e+1 (9.72e−2) 6.0192e+1 (1.20e+0) 7.0977e+1 (7.32e−2) 7.4259e+1 (6.56e−1) 
ZDT1 3.6601e+0 (3.92e−4) 3.6591e+0 (4.10e−4) 3.6594e+0 (4.72e−4) 3.6590e+0 (8.02e−4) 3.6476e+0 (1.73e−3) 3.6566e+0 (1.76e−3) 
ZDT2 3.3260e+0 (6.68e−4) 3.3250e+0 (5.79e−4) 3.3248e+0 (8.92e−4) 3.3239e+0 (2.75e−4) 3.3230e+0 (1.60e−3) 3.3191e+0 (3.05e−3) 
ZDT3 4.8131e+0 (4.47e−4) 4.8124e+0 (4.66e−4) 4.8118e+0 (5.16e−4) 4.8062e+0 (2.11e−4) 4.8094e+0 (1.11e−3) 4.8035e+0 (3.61e−1) 
ZDT4 3.6514e+0 (7.73e−3) 3.6506e+0 (7.75e−3) 3.6500e+0 (8.68e−3) 2.4820e+0 (2.09e−1) 3.6350e+0 (2.16e−2) 3.6307e+0 (4.11e−2) 
ZDT6 3.0242e+0 (2.58e−3) 3.0219e+0 (2.72e−3) 3.0230e+0 (2.10e−3) 3.0365e+0 (5.70e−4) 3.0281e+0 (2.22e−3) 3.0236e+0 (2.68e−3) 
WFG1 7.2435e+0 (1.26e+0) 7.6348e+0 (9.96e−1) 7.5547e+0 (9.22e−1) 7.0554e+0 (9.81e−1) 5.6509e+0 (7.41e−1) 5.4301e+0 (8.11e−1) 
WFG2 1.1151e+1 (4.17e−1) 1.1001e+1 (4.16e−1) 1.0952e+1 (4.09e−1) 1.0947e+1 (4.08e−1) 1.0914e+1 (4.02e−1) 1.0858e+1 (3.83e−1) 
WFG3 1.0944e+1 (5.11e−3) 1.0934e+1 (7.03e−3) 1.0940e+1 (5.52e−3) 1.0941e+1 (3.09e−3) 1.0926e+1 (8.49e−3) 1.0917e+1 (1.35e−2) 
WFG4 8.6679e+0 (7.34e−3) 8.6676e+0 (4.01e−3) 8.6674e+0 (4.93e−3) 8.6671e+0 (1.64e−3) 8.6574e+0 (1.16e−2) 8.6507e+0 (1.50e−2) 
WFG5 8.1575e+0 (3.00e−2) 8.1586e+0 (3.45e−2) 8.1531e+0 (3.49e−2) 8.1953e+0 (4.80e−2) 8.1283e+0 (2.12e−2) 8.1219e+0 (2.74e−2) 
WFG6 8.5708e+0 (1.07e−1) 8.5381e+0 (1.46e−1) 8.5088e+0 (1.57e−1) 8.4984e+0 (1.97e−1) 8.4775e+0 (1.71e−1) 8.4326e+0 (2.24e−1) 
WFG7 8.6703e+0 (6.64e−3) 8.6701e+0 (3.02e−3) 8.6689e+0 (6.54e−3) 8.6675e+0 (1.50e−3) 8.6612e+0 (1.05e−2) 8.6488e+0 (1.44e−2) 
WFG8 7.0008e+0 (3.61e−1) 7.1049e+0 (4.50e−1) 6.9988e+0 (4.42e−1) 6.9244e+0 (4.03e−1) 6.8374e+0 (3.25e−1) 6.7856e+0 (2.60e−1) 
WFG9 8.4377e+0 (1.57e−2) 8.4327e+0 (1.71e−2) 8.4328e+0 (1.48e−2) 8.4435e+0 (2.12e−2) 8.4143e+0 (2.29e−2) 8.4065e+0 (2.27e−2) 

The p value of 98 DOF is significant at a .05 level of significance by two-tailed t-test. ETEA is better than its competitor.

The p value of 98 DOF is significant at a .05 level of significance by two-tailed t-test. ETEA is worse than its competitor.

Tables 2 and 3 show the results of the bi-objective problems in terms of HV and IGD, respectively. It is clear that ETEA performs significantly better than the other five EMO algorithms. For HV, the proposed algorithm obtains the best value in 13 out of the 19 test problems, and IBEA, NSGA-II, and SPEA2 perform the best in 3, 2, and 1 out of all the problems, respectively. Moreover, for the majority of the problems where the proposed algorithm outperforms the other algorithms, the results have statistical significance (12, 10, 13, 17, and 18 out of all the 19 problems for NSGA-II, SPEA2, IBEA, ε-MOEA, and TDEA, respectively). To graphically illustrate the work of these algorithms, we show typical distributions of the final solutions obtained by the six algorithms on ZDT4 and WFG6 in Figures 7 and 8, respectively. Clearly, the solutions of ETEA are located uniformly along the whole Pareto front of the problems, which means that ETEA can provide a good trade-off among convergence, uniformity, and spread.

Table 3:
IGD comparison of the six EMO algorithms on bi-objective problems.
ProblemETEANSGA-IISPEA2IBEAε-MOEATDEA
SCH1 1.6600e−2 (9.31e−5) 1.8770e−2 (4.11e−4) 1.6607e−2 (1.04e−4) 1.9027e−2 (4.10e−4) 5.5664e−2 (6.14e−4) 1.7382e−2 (4.66e−4) 
SCH2 2.2344e−2 (4.41e−4) 2.3655e−2 (9.19e−4) 2.2645e−2 (5.44e−4) 1.2724e−1 (5.64e−2) 2.3706e−2 (1.23e−5) 2.2486e−2 (3.26e−4) 
FON 4.6455e−3 (7.14e−5) 5.5651e−3 (1.94e−4) 4.6601e−3 (7.31e−5) 2.2760e−2 (2.46e−3) 1.6571e−2 (1.07e−4) 4.7148e−3 (1.37e−4) 
KUR 3.3764e−2 (6.57e−4) 4.2330e−2 (2.03e−3) 3.4165e−2 (8.00e−4) 2.0370e−1 (2.30e−2) 3.5053e−2 (1.92e−4) 3.4003e−2 (1.10e−3) 
POL 5.3160e−2 (1.36e−3) 6.9675e−2 (4.57e−3) 5.3148e−2 (1.17e−3) 4.6631e−1 (1.32e−1) 1.9646e−1 (1.18e−3) 6.1638e−2 (2.38e−3) 
ZDT1 4.0241e−3 (6.94e−5) 4.8165e−3 (2.27e−4) 4.1792e−3 (9.21e−5) 4.1447e−3 (6.10e−5) 4.2747e−3 (5.43e−5) 4.2314e−3 (1.43e−4) 
ZDT2 4.0065e−3 (7.01e−5) 4.8254e−3 (1.63e−4) 4.1685e−3 (1.06e−4) 9.2889e−3 (4.18e−4) 5.7034e−3 (1.75e−4) 4.3642e−3 (1.59e−4) 
ZDT3 4.9152e−3 (1.07e−4) 5.6877e−3 (2.93e−3) 5.5663e−3 (4.13e−3) 3.1754e−2 (3.40e−3) 8.3480e−3 (9.02e−3) 5.1557e−3 (7.05e−3) 
ZDT4 6.0413e−3 (2.05e−3) 6.5646e−3 (1.70e−3) 6.5017e−3 (2.09e−3) 6.1194e−1 (1.14e−1) 6.9474e−3 (3.77e−3) 7.5864e−3 (1.10e−2) 
ZDT6 6.8528e−3 (6.51e−4) 7.6825e−3 (7.41e−4) 7.2374e−3 (5.67e−4) 5.5267e−3 (1.59e−4) 5.1994e−3 (3.01e−4) 6.2396e−3 (5.51e−4) 
WFG1 7.2727e−1 (2.09e−1) 6.1325e−1 (1.71e−1) 6.6093e−1 (1.58e−1) 8.6697e−1 (1.51e−1) 1.0243e+0 (1.35e−1) 1.0668e+0 (1.68e−1) 
WFG2 1.2190e−2 (1.82e−3) 1.4041e−2 (1.76e−3) 1.2986e−2 (1.81e−3) 7.3349e−2 (1.00e−2) 1.3516e−2 (2.72e−3) 1.4253e−2 (2.66e−3) 
WFG3 1.2146e−2 (3.77e−4) 1.4915e−2 (8.48e−4) 1.2383e−2 (3.61e−4) 1.2924e−2 (2.03e−4) 1.1827e−2 (2.94e−4) 1.1923e−2 (4.24e−4) 
WFG4 1.2945e−2 (2.50e−4) 1.3439e−2 (7.46e−4) 1.2913e−2 (3.72e−4) 1.8421e−2 (1.06e−3) 1.0121e−2 (9.64e−5) 1.1390e−2 (3.76e−4) 
WFG5 6.6740e−2 (2.14e−4) 6.7911e−2 (1.60e−3) 6.6761e−2 (1.13e−3) 7.1200e−2 (2.96e−4) 6.8338e−2 (3.63e−5) 6.6783e−2 (9.40e−5) 
WFG6 2.6514e−2 (1.44e−2) 3.0866e−2 (2.04e−2) 3.3049e−2 (2.28e−2) 4.1843e−2 (2.82e−2) 3.8578e−2 (2.46e−2) 4.1992e−2 (3.36e−2) 
WFG7 1.3016e−2 (2.72e−4) 1.6155e−2 (8.32e−4) 1.3064e−2 (2.95e−4) 2.1030e−2 (9.61e−4) 1.6374e−2 (1.20e−4) 1.3889e−2 (4.07e−4) 
WFG8 1.7005e−1 (3.28e−2) 1.6018e−1 (4.32e−2) 1.6954e−1 (3.99e−2) 1.9361e−1 (2.75e−2) 1.7821e−1 (3.07e−2) 1.8732e−1 (2.12e−2) 
WFG9 1.3875e−2 (1.18e−3) 1.7041e−2 (1.68e−3) 1.4064e−2 (1.06e−3) 1.9743e−2 (1.65e−3) 1.6912e−2 (1.95e−3) 1.5129e−2 (1.52e−3) 
ProblemETEANSGA-IISPEA2IBEAε-MOEATDEA
SCH1 1.6600e−2 (9.31e−5) 1.8770e−2 (4.11e−4) 1.6607e−2 (1.04e−4) 1.9027e−2 (4.10e−4) 5.5664e−2 (6.14e−4) 1.7382e−2 (4.66e−4) 
SCH2 2.2344e−2 (4.41e−4) 2.3655e−2 (9.19e−4) 2.2645e−2 (5.44e−4) 1.2724e−1 (5.64e−2) 2.3706e−2 (1.23e−5) 2.2486e−2 (3.26e−4) 
FON 4.6455e−3 (7.14e−5) 5.5651e−3 (1.94e−4) 4.6601e−3 (7.31e−5) 2.2760e−2 (2.46e−3) 1.6571e−2 (1.07e−4) 4.7148e−3 (1.37e−4) 
KUR 3.3764e−2 (6.57e−4) 4.2330e−2 (2.03e−3) 3.4165e−2 (8.00e−4) 2.0370e−1 (2.30e−2) 3.5053e−2 (1.92e−4) 3.4003e−2 (1.10e−3) 
POL 5.3160e−2 (1.36e−3) 6.9675e−2 (4.57e−3) 5.3148e−2 (1.17e−3) 4.6631e−1 (1.32e−1) 1.9646e−1 (1.18e−3) 6.1638e−2 (2.38e−3) 
ZDT1 4.0241e−3 (6.94e−5) 4.8165e−3 (2.27e−4) 4.1792e−3 (9.21e−5) 4.1447e−3 (6.10e−5) 4.2747e−3 (5.43e−5) 4.2314e−3 (1.43e−4) 
ZDT2 4.0065e−3 (7.01e−5) 4.8254e−3 (1.63e−4) 4.1685e−3 (1.06e−4) 9.2889e−3 (4.18e−4) 5.7034e−3 (1.75e−4) 4.3642e−3 (1.59e−4) 
ZDT3 4.9152e−3 (1.07e−4) 5.6877e−3 (2.93e−3) 5.5663e−3 (4.13e−3) 3.1754e−2 (3.40e−3) 8.3480e−3 (9.02e−3) 5.1557e−3 (7.05e−3) 
ZDT4 6.0413e−3 (2.05e−3) 6.5646e−3 (1.70e−3) 6.5017e−3 (2.09e−3) 6.1194e−1 (1.14e−1) 6.9474e−3 (3.77e−3) 7.5864e−3 (1.10e−2) 
ZDT6 6.8528e−3 (6.51e−4) 7.6825e−3 (7.41e−4) 7.2374e−3 (5.67e−4) 5.5267e−3 (1.59e−4) 5.1994e−3 (3.01e−4) 6.2396e−3 (5.51e−4) 
WFG1 7.2727e−1 (2.09e−1) 6.1325e−1 (1.71e−1) 6.6093e−1 (1.58e−1) 8.6697e−1 (1.51e−1) 1.0243e+0 (1.35e−1) 1.0668e+0 (1.68e−1) 
WFG2 1.2190e−2 (1.82e−3) 1.4041e−2 (1.76e−3) 1.2986e−2 (1.81e−3) 7.3349e−2 (1.00e−2) 1.3516e−2 (2.72e−3) 1.4253e−2 (2.66e−3) 
WFG3 1.2146e−2 (3.77e−4) 1.4915e−2 (8.48e−4) 1.2383e−2 (3.61e−4) 1.2924e−2 (2.03e−4) 1.1827e−2 (2.94e−4) 1.1923e−2 (4.24e−4) 
WFG4 1.2945e−2 (2.50e−4) 1.3439e−2 (7.46e−4) 1.2913e−2 (3.72e−4) 1.8421e−2 (1.06e−3) 1.0121e−2 (9.64e−5) 1.1390e−2 (3.76e−4) 
WFG5 6.6740e−2 (2.14e−4) 6.7911e−2 (1.60e−3) 6.6761e−2 (1.13e−3) 7.1200e−2 (2.96e−4) 6.8338e−2 (3.63e−5) 6.6783e−2 (9.40e−5) 
WFG6 2.6514e−2 (1.44e−2) 3.0866e−2 (2.04e−2) 3.3049e−2 (2.28e−2) 4.1843e−2 (2.82e−2) 3.8578e−2 (2.46e−2) 4.1992e−2 (3.36e−2) 
WFG7 1.3016e−2 (2.72e−4) 1.6155e−2 (8.32e−4) 1.3064e−2 (2.95e−4) 2.1030e−2 (9.61e−4) 1.6374e−2 (1.20e−4) 1.3889e−2 (4.07e−4) 
WFG8 1.7005e−1 (3.28e−2) 1.6018e−1 (4.32e−2) 1.6954e−1 (3.99e−2) 1.9361e−1 (2.75e−2) 1.7821e−1 (3.07e−2) 1.8732e−1 (2.12e−2) 
WFG9 1.3875e−2 (1.18e−3) 1.7041e−2 (1.68e−3) 1.4064e−2 (1.06e−3) 1.9743e−2 (1.65e−3) 1.6912e−2 (1.95e−3) 1.5129e−2 (1.52e−3) 

The p value of 98 DOF is significant at a .05 level of significance by two-tailed t-test. ETEA is better than its competitor.

The p value of 98 DOF is significant at a .05 level of significance by two-tailed t-test. ETEA is worse than its competitor.

Figure 7:

The final solutions obtained by the six algorithms on ZDT4.

Figure 7:

The final solutions obtained by the six algorithms on ZDT4.

Figure 8:

The final solutions obtained by the six algorithms on WFG6.

Figure 8:

The final solutions obtained by the six algorithms on WFG6.

Similar to HV, the results of IGD in Table 3 show that the proposed algorithm has a clear advantage over the other five algorithms for the majority of the problems. It obtains the best value in 13 out of the 19 problems, and most of the differences of the results between ETEA and the other algorithms have statistical significance. Specifically, the number of the problems where ETEA outperforms NSGA-II, SPEA2, IBEA, ε-MOEA, and TDEA with statistical significance is 16, 12, 17, 15, and 14, respectively. Interestingly, these algorithms sometimes obtain different and contradictory comparison results regarding different quality metrics (i.e., HV and IGD), although both metrics involve comprehensive performance of convergence, uniformity, and spread; for example, for WFG3, ETEA performs the best in terms of HV, but obtains a worse IGD value than ε-MOEA, whereas for WFG5, ETEA performs worse than IBEA with regard to HV but obtains the best IGD value of all.

In order to investigate such a contradictory observation, we introduce three widely-used performance metrics to separately assess the convergence, uniformity, and spread of the solution sets. They are generational distance (GD; Van Veldhuizen and Lamont, 1998), spacing6 (SP; Schott, 1995), and maximum spread7 (MS; Zitzler et al., 2000). The GD metric evaluates the convergence of a solution set by measuring the average distance from the solutions in the set to their closest point in the Pareto front; SP evaluates the uniformity of a solution set by calculating the standard deviation of the distance from each solution to its closest neighbor in the set; and MS evaluates the spread of a solution set by measuring the length of the diagonal of a minimal hyperbox that encloses the set. For the former two metrics, a smaller value is preferable, and as to the last metric, a larger value is better. More details of these metrics can be found in Van Veldhuizen and Lamont (1998), Schott (1995), and Zitzler et al. (2000).

Here, the WFG problem family is selected for investigation since the contradictory phenomenon on it appears to be the most obvious. Table 4 gives the results of all the algorithms on the WFG problems in terms of GD, SP, and MS. Additionally, for a clearer comparison, the table shows the rank of the six algorithms for each problem according to their average value.

Table 4:
Comparison of the GD, SP, and MS results of the six EMO algorithms on WFG problems. The number in the upper right quadrant associated with an algorithm indicates its rank among the six algorithms for a test problem. The value in the last row of each metric corresponds to the sum rank for all the problems.
Metric problemETEArankNSGA-IIrankSPEA2rankIBEArankε-MOEA rankTDEArank
GD WFG1 5.263e−2 (1.5e−2)1 5.349e−2 (1.7e−2)2 6.778e−2 (1.8e−2)4 6.076e−2 (1.9e−2)3 7.104e−2 (1.6e−2)5 9.505e−2 (2.0e−2)6 
 WFG2 8.723e−4 (8.0e−4)3 1.027e−3 (8.8e−4)5 8.442e−4 (7.8e−4)2 1.577e−3 (1.6e−3)6 9.646e−4 (9.6e−4)4 6.492e−4 (5.3e−4)1 
 WFG3 5.167e−4 (7.3e−5)3 6.261e−4 (6.7e−5)5 5.388e−4 (5.7e−5)4 6.963e−4 (1.3e−4)6 4.528e−4 (5.5e−5)2 3.519e−4 (9.7e−5)1 
 WFG4 1.470e−3 (6.3e−5)5 1.404e−3 (1.5e−4)4 1.529e−3 (8.0e−5)6 9.975e−4 (1.5e−4)2 9.955e−4 (2.0e−4)1 1.301e−3 (1.2e−4)3 
 WFG5 6.280e−3 (3.1e−5)1 6.440e−3 (3.8e−5)4 6.296e−3 (2.6e−5)2 6.963e−3 (3.6e−4)6 6.714e−3 (4.5e−4)5 6.316e−3 (7.5e−5)3 
 WFG6 1.830e−3 (2.5e−3)1 1.989e−3 (1.7e−3)2 2.966e−3 (2.5e−3)4 3.221e−3 (3.7e−3)6 2.570e−3 (2.0e−3)3 3.168e−3 (2.6e−3)5 
 WFG7 7.394e−4 (3.9e−5)3 8.650e−4 (7.1e−5)5 7.564e−4 (4.4e−5)4 9.937e−4 (3.3e−4)6 5.347e−4 (5.7e−5)1 7.048e−4 (5.4e−5)2 
 WFG8 2.772e−2 (5.6e−3)5 2.566e−2 (6.9e−3)2 2.970e−2 (7.6e−3)6 2.714e−2 (5.0e−3)3 2.394e−2 (3.7e−3)1 2.769e−2 (7.8e−3)4 
 WFG9 7.261e−4 (2.9e−4)3 1.020e−3 (2.5e−4)5 7.475e−4 (1.4e−4)4 1.258e−3 (4.8e−4)6 5.997e−4 (1.6e−4)1 6.635e−4 (2.0e−4)2 
 Sum rank 25 34 36 44 23 27 
SP WFG1 5.001e−3 (2.5e−3)1 1.289e−2 (6.1e−3)3 1.441e−2 (2.9e−2)5 1.826e−2 (1.2e−2)6 1.382e−2 (2.7e−2)4 9.298e−3 (3.9e−3)2 
 WFG2 5.612e−3 (9.3e−4)1 1.403e−2 (1.8e−3)4 5.952e−3 (8.7e−4)2 1.977e−2 (2.0e−3)6 1.754e−2 (5.4e−4)5 6.387e−3 (1.3e−3)3 
 WFG3 6.413e−3 (6.7e−4)3 1.512e−2 (1.7e−3)6 6.794e−3 (7.5e−4)4 1.353e−2 (1.2e−3)5 3.254e−3 (3.6e−4)1 6.238e−3 (9.2e−4)2 
 WFG4 7.166e−3 (9.2e−4)1 1.826e−2 (1.7e−3)4 7.689e−3 (7.5e−4)2 7.484e−2 (2.1e−2)6 3.450e−2 (2.3e−3)5 1.295e−2 (1.4e−3)3 
 WFG5 7.364e−3 (6.5e−4)1 1.854e−2 (2.4e−3)4 7.395e−3 (8.1e−4)2 4.205e−2 (2.2e−2)6 3.564e−2 (6.2e−4)5 1.182e−2 (7.9e−4)3 
 WFG6 7.238e−3 (8.5e−4)1 1.934e−2 (1.6e−3)4 7.482e−3 (9.4e−4)2 4.519e−2 (2.5e−2)5 4.613e−2 (7.8e−4)6 1.277e−2 (1.2e−3)3 
 WFG7 7.408e−3 (8.2e−4)1 1.955e−2 (1.8e−3)4 7.494e−3 (7.2e−4)2 7.001e−2 (2.1e−2)6 3.192e−2 (1.2e−3)5 1.222e−2 (9.8e−4)3 
 WFG8 1.422e−2 (8.0e−3)2 1.891e−2 (6.0e−3)3 1.205e−2 (4.7e−3)1 6.556e−2 (5.5e−2)6 4.531e−2 (2.9e−2)5 2.492e−2 (1.4e−2)4 
 WFG9 7.020e−3 (8.5e−4)1 1.782e−2 (1.9e−3)4 7.615e−3 (8.5e−4)2 2.831e−2 (7.5e−3)5 3.468e−2 (4.7e−3)6 1.210e−2 (1.3e−3)3 
 Sum rank 12 36 22 51 42 26 
MS WFG1 2.822e+0 (7.8e−1)4 3.021e+0 (7.6e−1)1 2.962e+0 (6.3e−1)2 2.833e+0 (6.3e−1)3 2.113e+0 (6.3e−1)5 1.931e+0 (6.8e−1)6 
 WFG2 3.926e+0 (4.9e−1)1 3.920e+0 (5.0e−1)2 3.860e+0 (4.8e−1)3 3.846e+0 (4.8e−1)4 3.797e+0 (4.9e−1)5 3.665e+0 (3.9e−1)6 
 WFG3 4.469e+0 (3.2e−3)4 4.470e+0 (5.0e−4)2 4.470e+0 (2.9e−3)2 4.472e+0 (6.8e−5)1 4.436e+0 (4.3e−3)5 4.424e+0 (1.8e−3)6 
 WFG4 4.471e+0 (2.3e−3)1 4.470e+0 (1.1e−3)2 4.469e+0 (2.4e−3)4 4.470e+0 (2.6e−4)2 4.459e+0 (9.4e−3)5 4.456e+0 (8.9e−3)6 
 WFG5 4.409e+0 (1.9e−2)3 4.410e+0 (1.9e−2)2 4.409e+0 (1.8e−2)3 4.431e+0 (2.1e−2)1 4.386e+0 (1.6e−2)6 4.387e+0 (2.2e−2)5 
 WFG6 4.472e+0 (1.3e−3)1 4.471e+0 (1.2e−3)3 4.471e+0 (1.8e−3)3 4.472e+0 (4.1e−3)1 4.463e+0 (7.0e−3)5 4.455e+0 (6.8e−3)6 
 WFG7 4.471e+0 (3.5e−3)2 4.471e+0 (5.8e−4)2 4.470e+0 (1.5e−3)4 4.472e+0 (1.5e−4)1 4.460e+0 (7.5e−3)5 4.450e+0 (9.7e−3)6 
 WFG8 4.394e+0 (6.3e−2)3 4.416e+0 (6.4e−2)1 4.384e+0 (4.6e−2)4 4.402e+0 (6.5e−2)2 4.362e+0 (7.1e−2)5 4.339e+0 (9.2e−2)6 
 WFG9 4.327e+0 (1.7e−2)2 4.326e+0 (2.0e−2)4 4.327e+0 (1.8e−2)2 4.330e+0 (1.3e−2)1 4.309e+0 (9.6e−3)5 4.305e+0 (1.8e−2)6 
 Sum rank 21 19 27 16 46 53 
Metric problemETEArankNSGA-IIrankSPEA2rankIBEArankε-MOEA rankTDEArank
GD WFG1 5.263e−2 (1.5e−2)1 5.349e−2 (1.7e−2)2 6.778e−2 (1.8e−2)4 6.076e−2 (1.9e−2)3 7.104e−2 (1.6e−2)5 9.505e−2 (2.0e−2)6 
 WFG2 8.723e−4 (8.0e−4)3 1.027e−3 (8.8e−4)5 8.442e−4 (7.8e−4)2 1.577e−3 (1.6e−3)6 9.646e−4 (9.6e−4)4 6.492e−4 (5.3e−4)1 
 WFG3 5.167e−4 (7.3e−5)3 6.261e−4 (6.7e−5)5 5.388e−4 (5.7e−5)4 6.963e−4 (1.3e−4)6 4.528e−4 (5.5e−5)2 3.519e−4 (9.7e−5)1 
 WFG4 1.470e−3 (6.3e−5)5 1.404e−3 (1.5e−4)4 1.529e−3 (8.0e−5)6 9.975e−4 (1.5e−4)2 9.955e−4 (2.0e−4)1 1.301e−3 (1.2e−4)3 
 WFG5 6.280e−3 (3.1e−5)1 6.440e−3 (3.8e−5)4 6.296e−3 (2.6e−5)2 6.963e−3 (3.6e−4)6 6.714e−3 (4.5e−4)5 6.316e−3 (7.5e−5)3 
 WFG6 1.830e−3 (2.5e−3)1 1.989e−3 (1.7e−3)2 2.966e−3 (2.5e−3)4 3.221e−3 (3.7e−3)6 2.570e−3 (2.0e−3)3 3.168e−3 (2.6e−3)5 
 WFG7 7.394e−4 (3.9e−5)3 8.650e−4 (7.1e−5)5 7.564e−4 (4.4e−5)4 9.937e−4 (3.3e−4)6 5.347e−4 (5.7e−5)1 7.048e−4 (5.4e−5)2 
 WFG8 2.772e−2 (5.6e−3)5 2.566e−2 (6.9e−3)2 2.970e−2 (7.6e−3)6 2.714e−2 (5.0e−3)3 2.394e−2 (3.7e−3)1 2.769e−2 (7.8e−3)4 
 WFG9 7.261e−4 (2.9e−4)3 1.020e−3 (2.5e−4)5 7.475e−4 (1.4e−4)4 1.258e−3 (4.8e−4)6 5.997e−4 (1.6e−4)1 6.635e−4 (2.0e−4)2 
 Sum rank 25 34 36 44 23 27 
SP WFG1 5.001e−3 (2.5e−3)1 1.289e−2 (6.1e−3)3 1.441e−2 (2.9e−2)5 1.826e−2 (1.2e−2)6 1.382e−2 (2.7e−2)4 9.298e−3 (3.9e−3)2 
 WFG2 5.612e−3 (9.3e−4)1 1.403e−2 (1.8e−3)4 5.952e−3 (8.7e−4)2 1.977e−2 (2.0e−3)6 1.754e−2 (5.4e−4)5 6.387e−3 (1.3e−3)3 
 WFG3 6.413e−3 (6.7e−4)3 1.512e−2 (1.7e−3)6 6.794e−3 (7.5e−4)4 1.353e−2 (1.2e−3)5 3.254e−3 (3.6e−4)1 6.238e−3 (9.2e−4)2 
 WFG4 7.166e−3 (9.2e−4)1 1.826e−2 (1.7e−3)4 7.689e−3 (7.5e−4)2 7.484e−2 (2.1e−2)6 3.450e−2 (2.3e−3)5 1.295e−2 (1.4e−3)3 
 WFG5 7.364e−3 (6.5e−4)1 1.854e−2 (2.4e−3)4 7.395e−3 (8.1e−4)2 4.205e−2 (2.2e−2)6 3.564e−2 (6.2e−4)5 1.182e−2 (7.9e−4)3 
 WFG6 7.238e−3 (8.5e−4)1 1.934e−2 (1.6e−3)4 7.482e−3 (9.4e−4)2 4.519e−2 (2.5e−2)5 4.613e−2 (7.8e−4)6 1.277e−2 (1.2e−3)3 
 WFG7 7.408e−3 (8.2e−4)1 1.955e−2 (1.8e−3)4 7.494e−3 (7.2e−4)2 7.001e−2 (2.1e−2)6 3.192e−2 (1.2e−3)5 1.222e−2 (9.8e−4)3 
 WFG8 1.422e−2 (8.0e−3)2 1.891e−2 (6.0e−3)3 1.205e−2 (4.7e−3)1 6.556e−2 (5.5e−2)6 4.531e−2 (2.9e−2)5 2.492e−2 (1.4e−2)4 
 WFG9 7.020e−3 (8.5e−4)1 1.782e−2 (1.9e−3)4 7.615e−3 (8.5e−4)2 2.831e−2 (7.5e−3)5 3.468e−2 (4.7e−3)6 1.210e−2 (1.3e−3)3 
 Sum rank 12 36 22 51 42 26 
MS WFG1 2.822e+0 (7.8e−1)4 3.021e+0 (7.6e−1)1 2.962e+0 (6.3e−1)2 2.833e+0 (6.3e−1)3 2.113e+0 (6.3e−1)5 1.931e+0 (6.8e−1)6 
 WFG2 3.926e+0 (4.9e−1)1 3.920e+0 (5.0e−1)2 3.860e+0 (4.8e−1)3 3.846e+0 (4.8e−1)4 3.797e+0 (4.9e−1)5 3.665e+0 (3.9e−1)6 
 WFG3 4.469e+0 (3.2e−3)4 4.470e+0 (5.0e−4)2 4.470e+0 (2.9e−3)2 4.472e+0 (6.8e−5)1 4.436e+0 (4.3e−3)5 4.424e+0 (1.8e−3)6 
 WFG4 4.471e+0 (2.3e−3)1 4.470e+0 (1.1e−3)2 4.469e+0 (2.4e−3)4 4.470e+0 (2.6e−4)2 4.459e+0 (9.4e−3)5 4.456e+0 (8.9e−3)6 
 WFG5 4.409e+0 (1.9e−2)3 4.410e+0 (1.9e−2)2 4.409e+0 (1.8e−2)3 4.431e+0 (2.1e−2)1 4.386e+0 (1.6e−2)6 4.387e+0 (2.2e−2)5 
 WFG6 4.472e+0 (1.3e−3)1 4.471e+0 (1.2e−3)3 4.471e+0 (1.8e−3)3 4.472e+0 (4.1e−3)1 4.463e+0 (7.0e−3)5 4.455e+0 (6.8e−3)6 
 WFG7 4.471e+0 (3.5e−3)2 4.471e+0 (5.8e−4)2 4.470e+0 (1.5e−3)4 4.472e+0 (1.5e−4)1 4.460e+0 (7.5e−3)5 4.450e+0 (9.7e−3)6 
 WFG8 4.394e+0 (6.3e−2)3 4.416e+0 (6.4e−2)1 4.384e+0 (4.6e−2)4 4.402e+0 (6.5e−2)2 4.362e+0 (7.1e−2)5 4.339e+0 (9.2e−2)6 
 WFG9 4.327e+0 (1.7e−2)2 4.326e+0 (2.0e−2)4 4.327e+0 (1.8e−2)2 4.330e+0 (1.3e−2)1 4.309e+0 (9.6e−3)5 4.305e+0 (1.8e−2)6 
 Sum rank 21 19 27 16 46 53 

It can be seen from the table that in contrast to ETEA, NSGA-II, and SPEA2, the algorithms IBEA, ε-MOEA, and TDEA show clear differences among convergence, uniformity, and spread. IBEA performs the best in terms of MS but obtains the worst results with respect to GD and SP. ε-MOEA and TDEA perform well in terms of GD but have the worst values for the MS metric.

From the above observations, the contradictory results of the different algorithms on the comprehensive performance metrics HV and IGD can be reasonably explained, considering that HV prefers extensity and IGD has a bias toward distribution uniformity. IBEA, which directly selects individuals according to their fitness based on a binary performance measure, may fail to maintain the uniformity of a solution set, thus obtaining a relatively poor IGD result; ε-MOEA and TDEA, which lack extensity-preserving mechanisms, may lose the boundary solutions of the Pareto front, thus providing a relatively low HV value.

On the other hand, the results in Table 4 also show that the proposed algorithm ETEA is competitive for all the considered metrics. It takes the second, first, and third places in terms of GD, SP, and MS, respectively, regarding the sum rank on all WFG problems. This indicates that the solution set obtained by ETEA has a good balance among convergence, uniformity, and spread.

Tables 5 and 6 show the results for the tri-objective problems in terms of HV and IGD. The advantage of ETEA over the other algorithms on tri-objective problems seems to be not as clear as that for bi-objective problems. Nevertheless, ETEA performs better than the other five algorithms on more than half of all the problems. It is able to obtain the best values in 8 and 7 out of the 13 problems for HV and IGD, respectively, and with statistical significance in most of the cases. The number of the problems where ETEA outperforms NSGA-II, SPEA2, IBEA, ε-MOEA, and TDEA with statistical significance is 13, 10, 11, 8, and 8, respectively, for HV, and 13, 8, 10, 10, and 7, respectively, for IGD. Figure 9 gives a typical distribution of the final solutions obtained by the six algorithms on DTLZ1. For a better observation, the boundary of the Pareto front of the problem is also plotted in the figure.

Table 5:
HV comparison of the six EMO algorithms on tri-objective problems.
Problem ETEA NSGA-II SPEA2 IBEA ε-MOEA TDEA 
VNT1 6.1582e+1 (4.15e−2) 6.1184e+1 (1.07e−1) 6.1577e+1 (3.35e−2) 5.9847e+1 (7.20e−2) 6.0071e+1 (3.49e−2) 6.1522e+1 (7.36e−2) 
VNT2 1.9146e+0 (3.90e−4) 1.9094e+0 (1.99e−3) 1.9145e+0 (5.62e−4) 1.8743e+0 (1.60e−2) 1.9069e+0 (1.84e−4) 1.9159e+0 (3.16e−4) 
VNT3 2.8303e+1 (1.09e−2) 2.8271e+1 (4.89e−3) 2.8300e+1 (1.01e−2) 2.8124e+1 (2.57e−2) 2.8365e+1 (7.09e−3) 2.8353e+1 (3.47e−3) 
DTLZ1 9.7286e−1 (3.30e−4) 9.6730e−1 (5.35e−4) 9.7212e−1 (1.08e−3) 8.9767e−1 (1.01e−2) 9.5612e−1 (1.26e−2) 9.7072e−1 (5.13e−4) 
DTLZ2 7.3948e+0 (6.57e−3) 7.3512e+0 (1.98e−2) 7.3912e+0 (6.93e−3) 5.7020e+0 (2.92e+0) 7.3890e+0 (9.06e−3) 7.4050e+0 (1.18e−2) 
DTLZ3 2.3436e+0 (2.75e+0) 6.9185e−1 (2.33e+0) 6.3694e−1 (1.79e+0) 6.2280e+0 (2.11e+0) 6.3262e+0 (2.09e+0 9.8252e−1 (2.17e+0) 
DTLZ4 7.2158e+0 (3.19e−1) 6.8874e+0 (5.94e−1) 6.9116e+0 (5.53e−1) 6.4116e+0 (1.19e+0) 7.0132e+0 (4.04e−1) 6.9417e+0 (4.54e−1) 
DTLZ5 6.1015e+0 (6.15e−4) 6.0993e+0 (6.83e−4) 6.1011e+0 (7.26e−4) 6.0884e+0 (1.06e−3) 6.1000e+0 (2.11e−3) 6.1013e+0 (1.23e−3) 
DTLZ6 4.6522e+0 (1.66e−1) 4.0978e+0 (2.60e−1) 4.0443e+0 (2.66e−1) 5.9742e+0 (9.68e−2) 5.2916e+0 (1.27e−1) 5.2551e+0 (1.12e−1) 
DTLZ7 1.3400e+1 (2.60e−2) 1.3258e+1 (5.74e−2) 1.3331e+1 (2.06e−1) 1.0257e+1 (2.73e+0) 1.3120e+1 (2.04e−2) 1.3275e+1 (4.80e−2) 
UF8 6.9641e+0 (3.83e−1) 6.4670e+0 (2.62e−1) 6.7052e+0 (3.58e−1) 6.6493e+0 (3.36e−1) 6.8654e+0 (3.09e−1) 6.6603e+0 (2.44e−1) 
UF9 7.1990e+0 (3.41e−1) 6.0702e+0 (6.33e−1) 6.8345e+0 (3.97e−1) 6.3068e+0 (8.74e−2) 7.1966e+0 (2.46e−1) 7.0881e+0 (2.55e−1) 
UF10 5.2946e+0 (7.31e−1) 3.7563e+0 (1.56e+0) 5.2645e+0 (6.98e−1) 4.8410e+0 (8.12e−1) 5.0434e+0 (9.63e−1) 3.8190e+0 (8.95e−1) 
Problem ETEA NSGA-II SPEA2 IBEA ε-MOEA TDEA 
VNT1 6.1582e+1 (4.15e−2) 6.1184e+1 (1.07e−1) 6.1577e+1 (3.35e−2) 5.9847e+1 (7.20e−2) 6.0071e+1 (3.49e−2) 6.1522e+1 (7.36e−2) 
VNT2 1.9146e+0 (3.90e−4) 1.9094e+0 (1.99e−3) 1.9145e+0 (5.62e−4) 1.8743e+0 (1.60e−2) 1.9069e+0 (1.84e−4) 1.9159e+0 (3.16e−4) 
VNT3 2.8303e+1 (1.09e−2) 2.8271e+1 (4.89e−3) 2.8300e+1 (1.01e−2) 2.8124e+1 (2.57e−2) 2.8365e+1 (7.09e−3) 2.8353e+1 (3.47e−3) 
DTLZ1 9.7286e−1 (3.30e−4) 9.6730e−1 (5.35e−4) 9.7212e−1 (1.08e−3) 8.9767e−1 (1.01e−2) 9.5612e−1 (1.26e−2) 9.7072e−1 (5.13e−4) 
DTLZ2 7.3948e+0 (6.57e−3) 7.3512e+0 (1.98e−2) 7.3912e+0 (6.93e−3) 5.7020e+0 (2.92e+0) 7.3890e+0 (9.06e−3) 7.4050e+0 (1.18e−2) 
DTLZ3 2.3436e+0 (2.75e+0) 6.9185e−1 (2.33e+0) 6.3694e−1 (1.79e+0) 6.2280e+0 (2.11e+0) 6.3262e+0 (2.09e+0 9.8252e−1 (2.17e+0) 
DTLZ4 7.2158e+0 (3.19e−1) 6.8874e+0 (5.94e−1) 6.9116e+0 (5.53e−1) 6.4116e+0 (1.19e+0) 7.0132e+0 (4.04e−1) 6.9417e+0 (4.54e−1) 
DTLZ5 6.1015e+0 (6.15e−4) 6.0993e+0 (6.83e−4) 6.1011e+0 (7.26e−4) 6.0884e+0 (1.06e−3) 6.1000e+0 (2.11e−3) 6.1013e+0 (1.23e−3) 
DTLZ6 4.6522e+0 (1.66e−1) 4.0978e+0 (2.60e−1) 4.0443e+0 (2.66e−1) 5.9742e+0 (9.68e−2) 5.2916e+0 (1.27e−1) 5.2551e+0 (1.12e−1) 
DTLZ7 1.3400e+1 (2.60e−2) 1.3258e+1 (5.74e−2) 1.3331e+1 (2.06e−1) 1.0257e+1 (2.73e+0) 1.3120e+1 (2.04e−2) 1.3275e+1 (4.80e−2) 
UF8 6.9641e+0 (3.83e−1) 6.4670e+0 (2.62e−1) 6.7052e+0 (3.58e−1) 6.6493e+0 (3.36e−1) 6.8654e+0 (3.09e−1) 6.6603e+0 (2.44e−1) 
UF9 7.1990e+0 (3.41e−1) 6.0702e+0 (6.33e−1) 6.8345e+0 (3.97e−1) 6.3068e+0 (8.74e−2) 7.1966e+0 (2.46e−1) 7.0881e+0 (2.55e−1) 
UF10 5.2946e+0 (7.31e−1) 3.7563e+0 (1.56e+0) 5.2645e+0 (6.98e−1) 4.8410e+0 (8.12e−1) 5.0434e+0 (9.63e−1) 3.8190e+0 (8.95e−1) 

The p value of 98 DOF is significant at a .05 level of significance by two-tailed t-test. ETEA is better than its competitor.

The p value of 98 DOF is significant at a .05 level of significance by two-tailed t-test. ETEA is worse than its competitor.

Table 6:
IGD comparison of the six EMO algorithms on tri-objective problems.
ProblemETEANSGA-IISPEA2IBEAε-MOEATDEA
VNT1 1.2664e−1 (2.38e−3) 1.5865e−1 (7.63e−3) 1.2714e−1 (2.44e−3) 1.6788e−1 (2.66e−2) 1.4595e−1 (9.38e−4) 1.2965e−1 (2.89e−3) 
VNT2 1.2305e−2 (2.54e−4) 2.3127e−2 (2.22e−3) 1.2311e−2 (3.39e−4) 4.4864e−2 (9.86e−3) 1.6541e−2 (3.20e−4) 1.1023e−2 (2.40e−4) 
VNT3 3.2065e−2 (1.09e−3) 4.9850e−2 (2.98e−3) 3.2478e−2 (1.27e−3) 2.5818e+0 (9.51e−2) 2.9258e−1 (3.20e−1) 3.8531e−2 (3.17e−3) 
DTLZ1 2.0657e−2 (5.21e−4) 3.3593e−2 (3.12e−2) 2.2111e−2 (2.38e−3) 1.8333e−1 (1.51e−2) 2.3695e−2 (2.26e−3) 4.6974e−2 (9.50e−2) 
DTLZ2 5.4021e−2 (9.72e−4) 6.8904e−2 (3.10e−3) 5.4357e−2 (1.30e−3) 5.1119e−1 (5.53e−1) 6.8102e−2 (9.32e−4) 5.3236e−2 (1.58e−3) 
DTLZ3 1.4106e+0 (1.10e+0) 2.7742e+0 (2.02e+0) 1.7410e+0 (1.50e+0) 5.5839e−1 (2.46e−1) 3.5472e−1 (6.35e−1) 2.6762e+0(2.14e+0) 
DTLZ4 1.5341e−1 (1.46e−1) 2.3645e−1 (1.66e−1) 1.9619e−1 (1.73e−1) 4.9543e−1 (3.69e−1) 2.3741e−1 (1.62e−1) 2.4994e−1 (1.74e−1) 
DTLZ5 4.2391e−3 (3.82e−4) 5.5382e−3 (3.31e−4) 4.3894e−3 (4.12e−4) 2.5685e−2 (1.32e−3) 6.4302e−3 (1.50e−4) 4.1865e−3 (3.08e−4) 
DTLZ6 4.9256e−1 (4.74e−2) 6.5429e−1 (6.32e−2) 6.5959e−1 (6.47e−2) 6.2603e−2 (3.00e−2) 2.7887e−1 (4.60e−2) 2.8453e−1 (3.52e−2) 
DTLZ7 6.2268e−2 (1.16e−3) 7.7085e−2 (4.22e−3) 6.3198e−2 (3.67e−3) 4.4057e−1 (3.16e−1) 7.3459e−2 (8.14e−2) 6.4598e−2 (3.08e−2) 
UF8 1.3654e−1 (4.47e−2) 2.3119e−1 (4.54e−2) 1.4552e−1 (4.05e−2) 3.3842e−1 (2.31e−2) 2.4193e−1 (8.10e−2) 1.4845e−1 (2.56e−2) 
UF9 1.6542e−1 (5.21e−2) 3.7618e−1 (9.30e−2) 1.9727e−1 (6.24e−2) 1.3921e−1 (4.76e−2) 1.2172e−1 (6.44e−2) 1.3805e−1 (4.50e−2) 
UF10 3.1911e−1 (6.83e−2) 5.8537e−1 (2.82e−1) 3.2101e−1 (5.78e−2) 5.5294e−1 (9.56e−2) 4.3721e−1 (1.10e−1) 4.2203e−1 (9.81e−2) 
ProblemETEANSGA-IISPEA2IBEAε-MOEATDEA
VNT1 1.2664e−1 (2.38e−3) 1.5865e−1 (7.63e−3) 1.2714e−1 (2.44e−3) 1.6788e−1 (2.66e−2) 1.4595e−1 (9.38e−4) 1.2965e−1 (2.89e−3) 
VNT2 1.2305e−2 (2.54e−4) 2.3127e−2 (2.22e−3) 1.2311e−2 (3.39e−4) 4.4864e−2 (9.86e−3) 1.6541e−2 (3.20e−4) 1.1023e−2 (2.40e−4) 
VNT3 3.2065e−2 (1.09e−3) 4.9850e−2 (2.98e−3) 3.2478e−2 (1.27e−3) 2.5818e+0 (9.51e−2) 2.9258e−1 (3.20e−1) 3.8531e−2 (3.17e−3) 
DTLZ1 2.0657e−2 (5.21e−4) 3.3593e−2 (3.12e−2) 2.2111e−2 (2.38e−3) 1.8333e−1 (1.51e−2) 2.3695e−2 (2.26e−3) 4.6974e−2 (9.50e−2) 
DTLZ2 5.4021e−2 (9.72e−4) 6.8904e−2 (3.10e−3) 5.4357e−2 (1.30e−3) 5.1119e−1 (5.53e−1) 6.8102e−2 (9.32e−4) 5.3236e−2 (1.58e−3) 
DTLZ3 1.4106e+0 (1.10e+0) 2.7742e+0 (2.02e+0) 1.7410e+0 (1.50e+0) 5.5839e−1 (2.46e−1) 3.5472e−1 (6.35e−1) 2.6762e+0(2.14e+0) 
DTLZ4 1.5341e−1 (1.46e−1) 2.3645e−1 (1.66e−1) 1.9619e−1 (1.73e−1) 4.9543e−1 (3.69e−1) 2.3741e−1 (1.62e−1) 2.4994e−1 (1.74e−1) 
DTLZ5 4.2391e−3 (3.82e−4) 5.5382e−3 (3.31e−4) 4.3894e−3 (4.12e−4) 2.5685e−2 (1.32e−3) 6.4302e−3 (1.50e−4) 4.1865e−3 (3.08e−4) 
DTLZ6 4.9256e−1 (4.74e−2) 6.5429e−1 (6.32e−2) 6.5959e−1 (6.47e−2) 6.2603e−2 (3.00e−2) 2.7887e−1 (4.60e−2) 2.8453e−1 (3.52e−2) 
DTLZ7 6.2268e−2 (1.16e−3) 7.7085e−2 (4.22e−3) 6.3198e−2 (3.67e−3) 4.4057e−1 (3.16e−1) 7.3459e−2 (8.14e−2) 6.4598e−2 (3.08e−2) 
UF8 1.3654e−1 (4.47e−2) 2.3119e−1 (4.54e−2) 1.4552e−1 (4.05e−2) 3.3842e−1 (2.31e−2) 2.4193e−1 (8.10e−2) 1.4845e−1 (2.56e−2) 
UF9 1.6542e−1 (5.21e−2) 3.7618e−1 (9.30e−2)