Offline Learning with a Selection Hyper-Heuristic: An Application to Water Distribution Network Optimisation

Optimising the design and rehabilitation of water distribution networks (WDN) is an important real-world problem. A WDN delivers water from reservoirs, tanks, and water treatment facilities to consumers via a network of pipes and makes use of pumps and valves to meet consumer demand. The WDN optimisation problem is characterised as a discrete NP-hard combinatorial optimisation problem with large-scale multimodal search landscapes (see Kheiri et al., 2015).

A variety of metaheuristics (see Blum and Roli, 2003) have been successfully applied to this problem such as simulated annealing (Cunha and Sousa, 1999), shuffled frog leaping algorithms (see Eusuff and Lansey, 2003), harmony search (Geem, 2006), honey-bee mating optimisation (Mohan and Babu, 2010), differential evolution (Zheng et al., 2013), particle swarm optimisation (Ezzeldin et al., 2014), multiobjective evolutionary algorithms (Wang et al., 2015), and ant colony optimisation (Shirzad, 2017). To date, hyper-heuristics have received relatively little attention in the WDN optimisation literature (see Kheiri et al., 2015).

Hyper-heuristics are general purpose heuristic methods. They differ from metaheuristics in that most metaheuristics search the space of problem solutions, whereas hyper-heuristics search the space of heuristics. In practice this means that a metaheuristic can have access to problem specific information, while a hyper-heuristic is subject to the limitations of the domain barrier and is unable to access problem specific information. The domain barrier requires the hyper-heuristic to perform well in the absence of problem specific information and therefore, it is hoped, to be “reuseable” across different problems and problem domains with minimal changes (see Drake et al., 2019).

Hyper-heuristics can be classified as either generation or selection hyper-heuristics. A generation hyper-heuristic generates new heuristics by discovery, or by modifying or combining existing low-level heuristics. Selection hyper-heuristics, such as those developed in this article, must select and apply a heuristic chosen from a set of low-level heuristics. The goal of both types of hyper-heuristics is to improve the search process through learning and/or optimisation. Such methods have proved effective when applied to a number of real-world problems (see Burke et al., 2019).

Many hyper-heuristics employ learning algorithms in order to improve optimisation performance, and this learning may be classified as either online or offline. Online learning is based on the low-level heuristic selections and resulting objective function values computed during the execution of a hyper-heuristic on a single problem. In contrast, offline learning is performed on a database of low-level heuristic selections and objective function values computed by a hyper-heuristic on a set of benchmark problems (see Burke et al., 2019).

This article investigates the optimisation of the 12 WDN problems presented in Wang et al. (2015) with the sequence-based selection hyper-heuristic, SSHH described in Kheiri et al. (2015). The SSHH hyper-heuristic employs online learning, and its performance is compared with the results produced by the five multiobjective evolutionary algorithms (or MOEAs) used in Wang et al. (2015). The SSHH hyper-heuristic can also be trained offline using the Baum--Welch learning algorithm detailed in Rabiner (1989) and an appropriate set of heuristic subsequences. The subsequences are chosen from an offline learning database using the statistical methodology presented in Yates and Keedwell (2019). This statistical methodology is also used to analyse the results of offline learning, which leads to an improved offline learning strategy. Finally, the potential for scalable learning, where knowledge learned from a small problem is usefully transferred to a second larger problem, is explored.

This study presents four results. Specifically,

1. a hyper-heuristic with online learning is competitive with the state-of-the-art across 12 WDN problems of varying size,

2. offline learning can improve on online learning performance,

3. the effectiveness of the heuristics changes markedly during the optimisation process for WDN problems, and

4. knowledge learned from small problems can be transferred to larger ones, raising the potential for high-performance algorithms trained on benchmarks and deployed on large, real-world problems.

The structure of this article is as follows. Section 2 describes the methodology, Section 3 describes the experimental setup, Section 4 presents the experimental results in detail, and Section 5 contains the conclusions of this study.

Section 2.1 contains an overview of the 12 water distribution networks, while Section 2.2 specifies the objective functions to be minimised. Section 2.3 describes the single objective function used in this study, and the solution points that are used to compare optimisation performance. Section 2.4 contains a description of the sequence-based SSHH hyper-heuristic, and the Baum--Welch learning algorithm. Lastly, Section 2.5 presents the elements of the statistical framework introduced in Yates and Keedwell (2019) which are employed in this study.

Water distribution networks are an important element of urban infrastructure as they convey clean water from reservoirs, tanks, and water treatment works to homes and businesses via a set of pipes. The design for a WDN aims to ensure a supply of clean water to the demand nodes at sufficient pressure and for minimum monetary cost. Although cost minimisation is the primary design objective, there are many other objectives that can also be considered such as minimisation of water age, adherence to water pressure and velocity constraints, and increasing the robustness of the network to reduce supply outages.

In this article, a simple WDN optimisation problem is considered. The discrete decision variables are the diameters of the pipes in a network. The objectives are to minimise the network's overall cost and maximise the network's reliability, while meeting all patterns of demand (or loading) conditions, and maintaining the minimum required head (pressure) throughout the network.

The problems differ from one another in a number of other respects. Of the 12 problems, 11 are based on real-world networks, while the TLN network (see Alperovits and Shamir, 1977) is an example of a hypothetical network. The TRN (see Gessler, 1985), BAK (see Lee and Lee, 2001), NYT (see Schaake and Lai, 1969), and EXN (see Farmani et al., 2004) networks are expansion problems, where the task is to extend an existing network by modifying some of the pipes in the network. In addition to selecting pipe diameters such problems can sometimes make use of extra options such pipe cleaning, pipe duplication, or ”leaving alone.” The remaining problems TLN, BLA (see Sherali et al., 2001), HAN (see Fujiwara and Khang, 1990), GOY (see Kim et al., 1994), FOS, PES, MOD (see Bragalli et al., 2008), and BIN (see Reca and Martínez, 2006) are pure design problems where the diameters of any or all of the pipes in a network can be modified.

Each problem has minimum head pressure requirements for the demand nodes. The BLA, FOS, PES, and MOD networks also have maximum pressure requirements, and upper bounds on water velocities in the pipes. The TRN network differs from the other problems in that it has three sets of loading conditions, while the BIN network, unlike a typical WDN, has a fixed level of water consumption across all demand nodes.

Figure 1 shows an example schematic of the Blacksburg distribution network.

In this study, the water distribution network design problem is specified by three objective functions: the cost $C$⁠, the head pressure deficit $H$⁠, and the network's resilience$In$⁠. The cost and head pressure deficit are to be minimised while the resilience is to be maximised.

The solutions produced by the SSHH hyper-heuristic are compared with those of the five multiobjective evolutionary algorithms (or MOEAs) used in Wang et al. (2015). It should be emphasised that the objective of this study is not to demonstrate that SSHH is a “superior” optimiser to the MOEAs. Rather, it is to show that SSHH is a computationally efficient optimisation algorithm capable of producing high-quality solutions comparable to the state-of-the-art, and that these solutions can be improved upon with offline learning.

An MOEA is a multiobjective optimiser that operates on a population of solutions. Such algorithms naturally generate a Pareto front of “best” solutions, which make the trade-offs between the conflicting objectives of cost and resilience explicit. Although SSHH operates on a single solution, it can also generate a Pareto front. However, the use of a single objective value forces the algorithm to explore a smaller region of the solution space; the region of low cost, and therefore lower resilience networks. As a result, comparing the Pareto fronts produced by the two methods is unhelpful. Instead, the SSHH and MOEA optimisers are compared on a single point on the published Pareto front for each problem; the solutions with the cheapest monetary cost for each problem.

Table 2 shows the cheapest viable solutions, their resilience, and the number of objective function evaluations used to generate them, taken from the Pareto fronts of $C$ and $In$ presented² in Wang et al. (2015). The Pareto fronts were generated by executing the five MOEAs 30 times, on each problem for the number of iterations shown.

A selection hyper-heuristic selects heuristics from a given set of low-level heuristics and applies them sequentially to optimise a particular problem. A selection hyper-heuristic can be viewed as an abstraction of an evolutionary algorithm. For example, a genetic algorithm can be represented as a selection hyper-heuristic where the low level heuristics used are crossover and mutate.

The SSHH hyper-heuristic is a sequence-based selection hyper-heuristic with online learning (see Kheiri et al., 2015; Kheiri and Keedwell, 2015, 2017). It uses a hidden Markov model (HMM) (see Rabiner, 1989) to generate sequences of heuristic selections, their parameters, and solution acceptance check decisions.

The HMM consists of a set of hidden states, and four probability matrices:

1. a state transition matrix to determine the probability of moving from one hidden state to another,

2. a low-level heuristic emission matrix to determine which low-level heuristic to select,

3. a parameter emission matrix to determine the parameter for a low-level heuristic, and

4. an acceptance check emission matrix to determine when a solution should be evaluated and checked for acceptance.

In the absence of a priori knowledge regarding a particular problem the number of hidden states is set to be the number of low-level heuristics in the domain, and the state transition, the parameter and acceptance check matrices are set to be equiprobable. The low-level heuristic emission matrix is set to the identity matrix. This ensures that, initially, each equiprobable hidden state emits a single low-level heuristic together with an equiprobable choice of heuristic parameter and acceptance check decision.

At each iteration of the optimisation process, the SSHH hyper-heuristic moves from its current state to a new state according to the probabilities of the transition matrix. Once a new state has been chosen, the emission probability matrices are used to determine a low-level heuristic, its parameters, and whether to check for acceptance or not. The selected parametrised low-level heuristic is applied to the current solution in order to derive a new solution. If the acceptance check decision is true, the objective function is evaluated on the new solution, and a decision is made whether to accept it. Specifically, a new solution is accepted if it improves on the current solution or is within $5%$ of the current best solution; otherwise it is rejected.

The SSHH hyper-heuristic employs online learning. During optimisation, SSHH keeps a history of the heuristic selections, parameters, and acceptance checks produced by the HMM. If, following an acceptance check, a new, best solution is found, the online learning algorithm steps through the history, increasing the probabilities of the accepted state transitions and emissions that led to the new minima. Thus the probability that the SSHH produces the sequence of heuristic selections and emissions contained in the history is now higher. After the acceptance check, the history is erased and the optimisation process is resumed.

The SSHH hyper-heuristic can also be trained offline using the Baum--Welch learning algorithm (see Rabiner, 1989). The Baum--Welch algorithm calculates the maximum likelihood estimates of a HMM's parameters for an arbitrary sequence of observations. In this study, the parameter to be estimated is the state transition probability matrix, and the observations are sequences of heuristics chosen from the offline learning database. The objective is to demonstrate that the Baum--Welch algorithm can learn an effective optimisation strategy for a problem before online learning refines the approach.

The SSHH hyper-heuristic requires a number of low-level heuristics. The five heuristics used in Kheiri et al. (2015) and a two-point crossover heuristic $C5$ are used in these experiments. The low-level heuristics are:

1. $M0$ – change one pipe diameter randomly,

2. $S1$ – swap two pipe diameters at random,

3. $M2$ – increase or decrease a randomly selected pipe diameter by one size,

4. $R3$ – “ruin” several pipes and rebuild randomly where the number of pipes to be changed is a parameter in the range [1,5],

5. $S4$ – shuffle several pipes (i.e., makes several swaps) where the number of pipes to be changed is a parameter in the range [1,5], and

6. $C5$ – two-point crossover of two vectors of network pipe diameters.

Although each WDN problem has the same objective function $f$ to be minimised, the range of $f$ varies considerably between problem instances. Without a priori knowledge of the objective function ranges, the objective function values from different problems cannot be compared directly. Instead, following the methodology introduced in Yates and Keedwell (2019), normalised subsequences of objective function values are compared. The definitions of the functions $α¯$ and $α¯f$ employed in this study are reproduced here.

The functions $α¯$ and $α¯f$ are the means of log values. The anti-log of the mean of the logs is equivalent to the geometric mean. The geometric mean is used so that no range dominates the average, and for this reason, in this study, $α¯$ and $α¯f$ are used in preference to the arithmetic mean of the decimal returns when comparing results across multiple problems.

Section 3.1 introduces the DBGen hyper-heuristic that is used to generate the offline learning database, while Section 3.2 contains the definition of the $γ$-ratio which is used to select training subsequences from the database. Section 3.3 details the training and testing methodology used to demonstrate generalisation across the WDN problems. Lastly, in Section 3.4, the statistical test used to validate improvements in the optimisation results is described.

All experiments were conducted on a Mac Pro computer with a 3.5-GHz, 6-core, Intel Xeon E5 processor, and 16 GB of 1866 MHz memory.

The unbiased, random, single selection hyper-heuristic DBGen used to generate the database of heuristic selections and objective function values is shown in listing 1.

The function ranInt() (lines 12 and 19) returns a uniformly distributed pseudorandom number in the set ${1,2,3,4,5}$⁠, while the function ranFloat() (line 15) returns a uniformly distributed pseudorandom number in the interval (0,1). The function selectHeuristic() (line 11) selects a single heuristic class at random from the set ${M,S,R,C}$⁠. The function apply() (line 13) takes the heuristic class and chooses, again at random, a low-level heuristic and its parameters, from the available heuristics of that class. The low-level heuristic is then applied to the current solution, and if the class is $C$⁠, to a crossover solution which is randomly selected from the crossover pool (lines 5–6 and lines 12–13). An objective function evaluation (line 14) and an acceptance check (line 15) are then performed. If a new solution's objective value is less than the current solution's objective value or $ranFloat()<0.5$ then it is accepted, and also stored, at random, in the crossover pool (lines 19–21). Otherwise the new solution is rejected. The random term allows new solutions to be accepted regardless of their objective function for $50%$ of the cases. Accepting states that may lead to a large increase in objective function value forces the DBGen hyper-heuristic to explore the space of low-level heuristic selections instead of optimising the problem efficiently.

When crossover heuristics are available, the choice of crossover mechanism also affects hyper-heuristic performance (see Drake et al., 2015). The DBGen crossover mechanism (and the number of crossover solutions) is taken from the crossover management scheme employed by the AdapHH hyper-heuristic (see Drake et al., 2015). This crossover mechanism is also used by SSHH.

The DBGen hyper-heuristic is executed 40 times on each of the 12 WDN problem instances for $10,000$⁠, $20,000$⁠, $50,000$⁠, and $100,000$ iterations for the small, medium, intermediate, and large problems, respectively. The number of iterations (denoted by MAX_ITER in listing 1) for each size were chosen for computational feasibility. The number of 40 runs was chosen so as to ensure that robust statistics could be calculated for each problem instance.

For each problem, each DBGen run is seeded by a unique number which is used to initialise the pseudorandom number generator used by DBGen, and to randomly initialise a WDN problem instance.

Although the computational cost of constructing such a database is significant, the objective of this study is not to compare optimisation performance using equivalent computational resources, but rather (as mentioned in Section 2.2) it is to investigate what can be learned from a preexisting offline learning database that has been constructed using a method that is independent of the optimisation algorithm under consideration.

The function used to select effective subsequences from the offline learning database, which is introduced in Yates and Keedwell (2019), is defined here.

The Baum--Welch learning algorithm is used to estimate the parameters of a HMM in order to maximise the probability of generating a given sequence of training observations. In this study, the parameter to be estimated is the state transition probability matrix of the SSHH hyper-heuristic, and the training observations are subsequences of low-level heuristics that are selected from the offline learning database using the $γ$-ratio. Specifically, following the method presented in Yates and Keedwell (2019), the 10 subsequences of lengths two and three with the largest $γ$-ratio are chosen. Subsequences of lengths two and three are used as they occur more frequently in the offline learning database than longer subsequences, and as a result, the statistics calculated for these subsequences are more reliable. The number of 10 subsequences was chosen to ensure that the subsequence set contained sufficient heuristic “diversity,” that is, the subsequences consist of more that just one or two low-level heuristics.

The results of executing the offline trained SSHH hyper-heuristic on the WDN problems are compared with those of an untrained SSHH hyper-heuristic using a leave-one-out cross-validation methodology (see Bishop, 2006). Recall that there are 12 problem instances in WDN domain. For each target problem, the training set consists of the 10 subsequences with the largest $γ$-ratios, chosen from the subsequences of the remaining 11 problems. The subsequences are used to train the HMM of the SSHH hyper-heuristic which is then evaluated on the target problem. This ensures that SSHH is always evaluated on a problem that is “unseen.” The objective is to demonstrate empirically that an offline trained SSHH hyper-heuristic is able to learn and generalise from training subsequences selected across a number of problems. In this context, generalisation means that the trained SSHH hyper-heuristic is able to significantly outperform the untrained SSHH hyper-heuristic when evaluated on unseen problem instances.

After Baum--Welch training, the probability matrices are edited to ensure that every state transition and heuristic emission has a non-zero probability of at least 0.0001. The trained SSHH hyper-heuristic, denoted T-SSHH, is initialised with these edited matrices.

In the evolutionary computation literature statistical tests are widely used to compare and rank the performance of algorithms that have been evaluated on a number of benchmark problems (see, for example, Dietterich, 1998; Demšar, 2006; Derrac et al., 2011; and Veek et al., 2017).

In this article, following Demšar (2006), the non-parametric, one-tailed Wilcoxon signed-rank test is used to establish a stochastic ordering on two hyper-heuristics A and B. The null hypotheses of the Wilcoxon test is that the median difference between pairs of observations is zero. The null hypothesis is tested at a significance level of 0.01 on sample sizes of 480, and is rejected if the $p$-value is less than 0.01. In this case, the alternative hypothesis that the median difference between pairs of observations is less than zero is accepted with $99%$ confidence. This implies that the random variable $α¯f(A)$ is “smaller” than the random variable $α¯f(B)$⁠, and thus hyper-heuristic A is more effective than hyper-heuristic B. The $αf(A)$ and $αf(B)$ values can be paired because the initial seed and therefore the initial solution for each problem instance $p=1,…,12$⁠, and each run $r=1,…,40$ is the same for both hyper-heuristics.

A more detailed discussion of the Wilcoxon test and its appropriateness for this work can be found in the Appendix.

In Section 4.1, the SSHH hyper-heuristic is used to optimise the 12 WDN problems. In Section 4.2, in order to improve optimisation performance the SSHH hyper-heuristic's HMM is trained offline with the Baum--Welch learning algorithm. However, in this case, offline learning fails to improve optimisation performance. With this in mind, Section 4.3 presents an analysis of the results of offline learning, which leads, in Section 4.4, to an effective offline learning methodology. Finally, in Section 4.5, the potential for scalable learning is explored.

In this section, the SSHH hyper-heuristic is used to optimise the 12 WDN problems. This experiment extends the work presented in Kheiri et al. (2015) by evaluating the SSHH hyper-heuristic, with an additional crossover operator, on multiple WDN problems. The objective is to compare the performance of the SSHH algorithm with that of the evolutionary algorithms, and to determine the utility of the extra crossover heuristic.

The SSHH hyper-heuristic is executed 40 times on each of the 12 problems in the WDN domain. For each problem, each run is seeded by a number that is distinct to the seeds used to generate the offline learning database. The number of iterations used by SSHH varies with the problem size. Specifically, SSHH is executed for $10,000$⁠, $20,000$⁠, $50,000$⁠, and $100,000$ iterations for the small, medium, intermediate, and large problems, respectively. The number of iterations for each size were chosen for computational feasibility. The number of 40 runs was chosen so as to ensure that robust statistics could be calculated for each problem instance. The SSHH hyper-heuristic is compared to the MOEAs on a single point on the published Pareto fronts; the solutions with the cheapest monetary cost. The cheapest solutions found by each algorithm for each problem are shown in Table 3. As the SSHH only evaluates the objective function after an acceptance check, rather than every iteration, the number of objective function evaluations (Evals.) for SSHH is always less than the number of iterations.

The cheapest solutions found by SSHH and the MOEAs for the problems BAK and NYT are identical. For the small problem TRN and TLN, the medium-sized problems BLA, HAN, GOY, and FOS no solution dominates. For the intermediate problem PES, and the large problems MOD and BIN, the MOEA solutions (shown in boldface) dominate those found by SSHH.

The solution produced by SSHH for EXN has a head deficit of $H=2.6940$⁠. The head penalties are calculated by summing the pressure violations over the whole network. When the violations are small, and spread evenly across a network, the solution can be viewed as semi-viable, or approaching viability. For the EXN network, the pressure violation occurs at a single pipe, and so the SSHH solution is deemed nonviable. However, by choosing the alternative constants $a=2$⁠, $b=5000$⁠, and $c=0.1$ for the objective function, SSHH can find a viable solution where $C=17.0886$ and $In=0.3373$⁠, using 49552 objective function evaluations. This result is denoted $EXN*$⁠.

Table 4 shows the cheapest cost solutions found by executing SSHH on PES, MOD, BIN, and EXN for additional iterations. Specifically, SSHH is run 40 times for $1,000,000$ iterations on PES and $2,000,000$ iterations on MOD, BIN and EXN. As extending the number of iterations for EXN using the original objective function constants also yields a nonviable solution, the result shown in Table 4, denoted $EXN*$⁠, uses the alternative constants defined above. While the MOD and BIN solutions remain dominant after the additional iterations, the solutions for PES and $EXN*$ are now non-dominated. For PES, SSHH actually finds a lower cost solution than any of the MOEAs.

As the constants used to define the objective function are chosen to ensure that the function is positive for all the problems in the WDN domain, one would expect a multiobjective algorithm to outperform a single objective function optimiser. However, the differences in cost and network resilience are modest, while SSHH employs significantly less objective function evaluations than the MOEAs. This is important because, for many large problems (such as EXN), evaluating the objective function is computationally expensive.

These results demonstrate that the SSHH hyper-heuristic is a computationally efficient alternative to a multiobjective evolutionary algorithm for the optimisation of WDN problems. Furthermore the SSHH algorithm can be extended to optimise multiple objectives (see Walker and Keedwell, 2016).

In order to improve optimisation performance the SSHH hyper-heuristic's HMM is trained offline with the Baum--Welch learning algorithm. The training observations consist of effective subsequences of low-level heuristics selected from the offline database using the $γ$-ratio.

The SSHH hyper-heuristic is trained offline with the Baum--Welch learning algorithm on the 10 subsequences of heuristics of lengths two and three with the largest $γ$-ratios in the offline learning database. The training sets are constructed from these effective subsequences using a leave-one-out cross-validation methodology. Specifically, for each WDN problem, the training set consists of the 10 subsequences with the largest $γ$-ratios, chosen from the subsequences of the remaining problems. These subsequences are used to offline train the HMM of the SSHH hyper-heuristic which is then used to optimise the “unseen” target problem. This methodology gives rise to 12 training sets, one for each of the WDN problems.

The offline trained hyper-heuristic T-SSHH is executed 40 times on each of the 12 problems in the WDN domain. The mean final objective function values for each problem are shown in Table 5.

The problem-by-problem results demonstrate that the SSHH hyper-heuristic outperforms the offline trained hyper-heuristic T-SSHH on 10 of the 12 WDN problems. The overall results are shown in Table 6.

Although the mean final log return $α¯f$ is is slightly better for T-SSHH than SSHH, the mean percentage change in optimisation value is worse. The differences in final log returns are tested for statistical significance as explained in Section 3.4 and the results are shown in Table 7.

The results of the one-tailed Wilcoxon test together with the results contained in Tables 5 and 6 indicate that T-SSHH is an inferior optimiser when compared with SSHH, and that offline learning has failed to improve optimisation performance.

In Yates and Keedwell (2019) the $γ$-ratio is used to select effective subsequences of heuristics from a number of distinct problem domains. The $γ$-ratio can also be used, in certain circumstances, to select effective subsequences across a number of problem domains. For example, consider two comparable domains, and define the $α¯$-order of a domain to be the order of its low-level heuristics when ranked by their mean log return $α¯$⁠. If the difference between the $α¯$-orderings of the two domains is small, then a subsequence that is effective in one domain is likely to be effective in the other. In this section, the failure of offline training to improve optimisation performance in the previous section is investigated by considering the $α¯$-order of the low-level heuristics of the WDN domain during the course of the optimisation process.

Two sets of heuristic instances are selected from the offline learning database. These sets contain the heuristic instances that occur at the “beginning” of the optimisation process, when objective function values are relatively high, and at the “end” of the optimisation process, when objective function values are relatively low.

Calculating the percentile values over all 480 sequences in the offline learning database can lead to some problem instances dominating the sets; those that produce very high or very low objective function values. To mitigate this, the percentiles $Pi$ are calculated locally over the objective function values of each sequence $i=1,…,480$⁠. This ensures that heuristic instances from all the problems in the domain are included in the LOW and HIGH sets.

The mean log return $α¯$ of the low-level heuristics is calculated from the LOW and HIGH sets. The heuristics are then ranked by their $α¯$⁠. The resulting LOW and HIGH orderings for the smallest problem in each size category, EXN, and overall, are shown in Table 8.

This large change in heuristic effectiveness is particularly relevant to SSHH as the efficient optimisation of such problems require online learning strategies, as any offline learned heuristic subsequences can only be effective at particular points during optimisation. The result of this analysis is also important for other optimisation techniques. For example, a vanilla genetic algorithm will typically execute a mutation operation and crossover operation at a given fixed rate for every iteration of the algorithm. For problems where the effectiveness of the crossover and mutate operation varies significantly during optimisation, optimisation performance can be improved by varying this rate accordingly.

In order to test the hypothesis that it is the large variance in heuristic performance during optimisation that prevents the Baum--Welch algorithm from finding a suitable set of HMM parameters, the experiment in Section 4.2 is repeated using subsequences that are effective when the objective function value is relatively low. Specifically, the 10 subsequences of lengths two and three with the largest $γ$-ratios are selected from the LOW set defined in Section 4.3. The effective LOW subsequences are then used by the Baum--Welch algorithm to train a HMM using a leave-one-out cross-validation methodology.

The SSHH hyper-heuristic is initialised with an identity heuristic emission matrix and equiprobable transition, parameter and acceptance matrices. The optimisation process with online learning is started, and when half of the specified iterations have been performed, the current HMM is switched for the offline trained HMM. The optimisation process, again with online learning, is then resumed. Although the method of switching to the LOW trained HMM is simple, it is trivial to implement, and requires no information regarding the objective function. The results for the SSHH hyper-heuristic when it is trained offline using effective subsequences chosen from the LOW set are denoted T-SSHH-L. The results for the offline trained hyper-heuristic with no switching mechanism, where the offline trained HMM is used from the outset of the optimisation process, are included for comparison purposes, and are denoted T-SSHH-L-NS.

The offline trained hyper-heuristic T-SSHH-L and T-SSHH-L-NS are each executed 40 times on each of the 12 problems in the WDN domain. Table 9 contains the mean final objective function value for each problem in the WDN domain. The T-SSHH-L hyper-heuristic outperforms SSHH on all 12 problems.

Overall, the results in Table 10 show that T-SSHH-L outperforms SSHH. The T-SSHH-L hyper-heuristic also outperforms T-SSHH-L-NS which demonstrates the importance of applying the effective subsequences at an appropriate point in the optimisation process.

The overall differences in the final log returns are tested for statistical significance as before, and the results are shown in Table 11. They demonstrate that the differences, while small, are statistically significant, and so the offline trained hyper-heuristic T-SSHH-L outperforms the SSHH hyper-heuristic with $99%$ confidence.

These results show that, despite large variations in heuristic performance over the optimisation process, it is possible to significantly improve the optimisation performance of the SSHH hyper-heuristic on unseen WDN problems with offline learning. The result for EXN is particularly encouraging, as it shows that a training set constructed from a number of smaller problems can lead to improved optimisation on a larger, more complex, problem.

Table 12 contains a comparison of the cheapest monetary cost solutions found by SSHH, T-SSHH-L, and the MOEAs for each problem. Dominant solutions are shown in boldface. The performance of T-SSHH-L is markedly inferior to the MOEAs on the large problems MOD, BIN and EXN. This could be due to the relatively low number of iterations employed by SSHH on these problems. Another cause could be that the 100000 iterations employed by DBGen is insufficient to discover the subsequences of heuristics necessary to construct effective training sets for the large problems.

It should be noted that although the values of $C$⁠, $H$ and $In$ vary considerably across the WDN problems, the objective function constants $a$⁠, $b$⁠, and $c$ are identical for each task. The $EXN*$ result (see Table 4) demonstrates that optimisation performance can be improved by “tuning” the constants for a particular problem. However, preliminary experiments suggest that using the same objective function for each problem facilitates learning and generalisation.

In this section, the potential for scalable learning is explored. Scalable learning is where a model developed for a small, computationally tractable task, is reused as the starting point of a model for a larger, second task. The concept of scalable learning is appropriated from Burke et al. (2007) where the authors use evolutionary algorithms to generate novel heuristics for small problems which are then shown to perform well when tested on larger problems.

The idea is to generate training subsequences from a small problem which are then used, offline, to improve optimisation performance on a larger, more computationally expensive problem. With this in mind, a training set is constructed from the LOW selections and objective function values of the TRN problem. Specifically, the 10 subsequences of lengths two and three, with the largest $γ$-ratios are selected from the LOW set for TRN, and this training set is used to offline train SSHH. The offline trained hyper-heuristic, denoted T-SSHH-L-TRN, uses the same switching mechanism as T-SSHH-L described in the previous section. As all the training subsequences are drawn from one problem and evaluated on the remaining 11 problems, the leave-one-out cross validation methodology is not required. The results for the TRN problem are included for completeness.

The offline trained hyper-heuristic T-SSHH-L-TRN is executed 40 times on each of the 12 problems in the WDN domain. Table 13 contains the mean final objective function values for each problem in the WDN domain. The results demonstrate that T-SSHH-L-TRN outperforms SSHH on every problem (see Table 9), and outperforms T-SSHH-L on five out of 11 problems.

The overall optimisation results shown in Table 14 indicate that T-SSHH-L-TRN outperforms SSHH, and is slightly less effective than T-SSHH-L.

The overall differences in final log returns are tested for statistical significance. The results, shown in Table 15, indicate that the offline trained hyper-heuristic T-SSHH-L-TRN outperforms the SSHH hyper-heuristic on 11 of the WDN problems with $99%$ confidence.

The result that subsequences drawn from TRN, which is the smallest problem in the WDN domain, can be used to the improve the optimisation of EXN which is the largest, is notable. However, the improvement in performance is perhaps less surprising when one notes that the heuristic orderings in the LOW sets are very similar for TRN and EXN (see Table 8).

These experimental results demonstrate that it is possible to learn offline from a small, computationally inexpensive problem, and then use this knowledge to improve optimisation performance on larger, more computationally expensive WDN problems with the same objective function.

The sequence-based selection hyper-heuristic SSHH is able to produce viable solutions for the 12 WDN problems that are comparable in monetary cost and resilience to the cheapest solutions presented in Wang et al. (2015) but using significantly less computational resources. In addition, the two-point crossover heuristic is shown to perform well when compared with the five low-level heuristics employed in Kheiri et al. (2015).

The selection hyper-heuristic DBGen is used to generate an offline learning database of low-level heuristic selections and their objective function values across the 12 problems in the WDN domain. By employing the framework presented in Yates and Keedwell (2019), low-level heuristics and subsequences of heuristics can be identified in the database as being either effective or disruptive. Effective subsequences tend to decrease the objective function value, while disruptive subsequences tend to increase the objective function value. The most effective heuristic subsequences are used to offline train the HMM of the SSHH hyper-heuristic using the Baum--Welch learning algorithm. However, the Baum--Welch algorithm is unable to learn an effective optimisation strategy because the performance of the effective subsequences varies considerably during the optimisation process, and this variation can be quantified by the Spearman's footrule metric. In order to test this hypothesis, subsequences that are effective when the objective function value is relatively low are selected from the database. These subsequences are used to train another HMM which is employed by the SSHH hyper-heuristic after the midpoint of the optimisation process. Although the method of switching between optimisation strategies is simple, it produces a small, but notable improvement in performance. The final experiment demonstrates that it is possible to learn offline from a small WDN problem which is computationally inexpensive, and transfer this learning to larger, more computationally expensive problems. Furthermore, this improvement in optimisation performance is statistically significant, with $99%$ confidence.

Although the offline learning gains are modest, it should be possible to improve on these results. For example, one obvious enhancement would be to improve the mechanism employed to switch between the optimisation strategies encoded in the initial and offline trained HMMs.

Another limitation of this study is that SSHH and the MOEAs are compared at only a single point; the cheapest viable solution. This weakness could be addressed by employing the multiobjective version of SSHH described in Walker and Keedwell (2016). This would allow SSHH to generate a Pareto front of solutions during optimisation, enabling a direct comparison with the Pareto fronts presented in Wang et al. (2015), and facilitating an analysis of the trade-offs between the conflicting design objectives of cost and resilience.

It is clear from the $EXN*$ result in Section 4.1 that optimisation can also be improved by “tuning” the objective function constants $a$⁠, $b$⁠, and $c$ for each problem. However preliminary experiments suggest that using the same objective function facilitates the generalisation exhibited by T-SSHH-L and the transfer learning exhibited by T-SSHH-L-TRN. A fuller investigation into learning when these constants vary for each problem is left to future research.