Reproducible and Efficient Benchmarks for Hyperparameter Optimization of Neural Machine Translation Systems

Abstract Hyperparameter selection is a crucial part of building neural machine translation (NMT) systems across both academia and industry. Fine-grained adjustments to a model’s architecture or training recipe can mean the difference between a positive and negative research result or between a state-of-the-art and underperforming system. While recent literature has proposed methods for automatic hyperparameter optimization (HPO), there has been limited work on applying these methods to neural machine translation (NMT), due in part to the high costs associated with experiments that train large numbers of model variants. To facilitate research in this space, we introduce a lookup-based approach that uses a library of pre-trained models for fast, low cost HPO experimentation. Our contributions include (1) the release of a large collection of trained NMT models covering a wide range of hyperparameters, (2) the proposal of targeted metrics for evaluating HPO methods on NMT, and (3) a reproducible benchmark of several HPO methods against our model library, including novel graph-based and multiobjective methods.


Introduction
Choosing effective hyperparameters is crucial for building strong neural machine translation (NMT) systems. Although some choices present obvious trade-offs (e.g., more and larger layers tend to increase quality at the cost of speed), others are more subtle (e.g., effects of batch size, learning rate, and normalization techniques on different layer types). Optimal versus suboptimal hyperparameters can lead to dramatic swings in system performance; consider the wide range of BLEU scores for variants of the same base system in Figure 1 (left). In practice, these hyperparame-ters are often tuned manually based on intuition and heuristics, a tedious and error-prone process that can lead to unreliable experimental results and underperforming shared task or production systems. The difficulty is compounded when system builders must jointly optimize multiple objectives, such as translation accuracy (BLEU) and decoding speed, which are largely uncorrelated as shown in Figure 1 (right).
In the past decade, various hyperparameter optimization (HPO) methods have emerged in the machine learning literature under the labels of ''AutoML'' (Bergstra et al., 2011;Hutter et al., 2011;Bardenet et al., 2013;Snoek et al., 2015) and ''neural architecture search'' (Zoph and Le, 2016;Liu et al., 2018a,b;Cai et al., 2018;. However, it is unclear how they perform on NMT; we are not aware of any prior work with comprehensive evaluation. One challenge is that the state-of-the-art NMT models (Sutskever et al., 2014;Bahdanau et al., 2015;Gehring et al., 2017;Vaswani et al., 2017) require significant computational resources for training. Secondly, they usually have large hyperparameter search spaces. Thus, it is prohibitively expensive in practice to compare HPO methods on NMT tasks.
In order to enable reproducible HPO research on NMT tasks, we adopt a benchmark procedure based on ''table-lookup''. This approach was recently introduced to neural architecture search by Ying et al. (2019), and to hyperparameter optimization by . First, we train an extremely large number of NMT models with diverse hyperparameter settings and record their performance metrics (e.g., BLEU, decoding time) in a table. Then, we constrain our HPO methods to sample from this finite set of models. This allows us to simply ''look-up'' their precomputed performance metrics, and amortizes the burden of computation: As long as we ensure that Figure 1: Left: Histogram of BLEU scores that show wide variance in performance for a base NMT system (transformer) with different hyperparameters (e.g., BPE operations, # of layers, initial learning rate). Right: Scatterplot of BLEU and decoding time with different hyperparameters. Gold stars represent the Pareto-optimal systems.
we have trained and pre-computed a large number of representative NMT models beforehand, HPO algorithm developers no longer need to deal with the cost of training NMT. Importantly, this kind of benchmark significantly speeds up the HPO experiment turnover time, enabling fast repeated trials for rigorous tests and facilitates detailed error analysis.
The main contributions of this work are: 1. Dataset: We release a benchmark dataset 1 for comparing HPO methods on NMT models. This ''table-lookup'' HPO dataset supports both single-objective and multiobjective optimization of translation accuracy and decoding time (Section 3). Specifically, we trained a total of 2,245 Transformers (Vaswani et al., 2017) on six different corpora (with a cost of approximately 1,547 GPU days), and collected all pairs of hyperparameter settings and corresponding performance metrics.
2. Evaluation protocols: We provide three kinds of metrics for evaluating HPO methods, based on different computational budgets (Section 4). We also demonstrate error analysis techniques that are enabled by this ''table-lookup'' framework, which provide insights into algorithm behavior (Section 7).
3. HPO method benchmarks: We benchmark the performance of several HPO methods on our dataset (Section 6). These include Bayesian optimization as well as a novel graph-based method that exploits the structure of the hyperparameter space (Section 5). We also extend these methods to handle the 1 https://github.com/Este1le/hpo_nmt. multiobjective optimization of both BLEU and decoding time. These experiments illustrate how to utilize the dataset to rigorously evaluate HPO for NMT.

HPO Problem Definition
Given a machine learning algorithm with H hyperparameters, we denote the domain of the h-th hyperparameter by Λ h and the overall hyperparameter configuration space as Λ = Λ 1 × Λ 2 × . . . Λ H . When trained with a hyperparameter setting λ ∈ Λ on data D train , the algorithm's performance metric on some validation data D valid is f (λ) := V (λ, D train , D valid ). In the context of NMT, f (·) or V(·) could be the perplexity, translation accuracy (e.g., BLEU score), or decoding time on D valid . In general, f (·) is computationally expensive to obtain; it requires training a model to completion, then evaluating some performance metric on a validation set. For purposes of exposition, we assume that lower f (·) is better, so we might define f (·) as 1 − BLEU. The goal of hyperparameter optimization is then to find a λ ⋆ = arg min λ∈Λ f (λ), with as few evaluations of f (·) as possible. An HPO problem can be challenging for three reasons: (a) Λ may be a combinatorially large space, prohibiting grid search over hyperparameters. (b) f (·) may be expensive to compute, so there is a tight budget on how many evaluations of f (·) are allowed. (c) f is not a continuous function and no gradient information can be exploited, forcing us to view the arg min as a blackbox discrete search problem. NMT HPO search exhibits all these conditions.
One class of algorithms that tackles the HPO problem is sequential model-based optimization (SMBO), illustrated in Figure 2. SMBO approximates f with a cheap-to-evaluate surrogate modelf (Feurer and Hutter, 2019;Luo, 2016;. SMBO starts by querying f with initial hyperparameters {λ init } and recording the resulting (λ init , f (λ init )) pairs. Then, it iteratively (1) fits the surrogatef on pairs observed so far; (2) gets the predictionsf (λ i ) for unlabeled/unobserved hyperparameters; and (3) selects a promising λ p to query next based on these predictions and an acquisition function, whose role is to trade off exploration in Λ with high model uncertainty and exploitation in Λ with lowf (·). Figure 2: SMBO framework. The part shaded in light blue contains two ingredients required for implementing a SMBO method: the surrogate model and the acquisition function, for which we will present two choices in Section 5.
Evolutionary algorithms (Eberhart and Shi, 1998;Simon, 2013) are also used to solve HPO problems. Unlike SMBO, they do not approximate f with a surrogatef ; rather, they directly sample hyperparameters with high f (·) from a population and recombine them to form the next query. 2 3 Table-Lookup HPO Datasets

Table-Lookup Framework
To evaluate a newly devised HPO algorithm, one needs to run each component of the loop in Figure 2. However, the ''query'' step is computationally expensive: We need to train a new NMT system each time we sample a new hyperparameter.
The idea of table lookup is to simply pre-train a large set of I NMT systems and record the pairs {λ i , f (λ i )} i=1,...,I in a table. Thus, when running the loop in Figure 2, the HPO algorithm developer can look up f (λ i ) whenever necessary, without having to train a NMT model from scratch. This significantly speeds up the experimental process. The advantages are: 1. One can perform multiple random trials of the same algorithm, to test robustness.
2. One can perform comparisons with more baseline algorithms, to make stronger claims.
3. One can perform the same experiment under different budget constraints, to simulate different real-world use cases. To be effective, table lookup depends on two important assumptions: First, the table has to be sufficiently large to cover the space of hyperparameters Λ. Second, the HPO algorithm needs to be modified to sample from the finite set of hyperparameters in the table; this is usually easy to implement but the assumption is that finite-sample results will generalize.

HPO Algorithm Selection/Development
There exist many choices of HPO algorithm, which can be evaluated or further developed on our lookup tables. Figure 3 illustrates this process. The performance of HPO algorithm candidates on various MT datasets serves as the basis for HPO selection. The selected HPO algorithm can then be applied on new MT datasets.
There are two kinds of generalization effects at play: (1) generalization of an HPO algorithm across MT datasets, and (2) generalization of MT models and their associated hyperparameters across MT datasets. We mainly care about (1) in the algorithm development process, which is why we opt to provide six distinct datasets described in Section 3.3 (as opposed to, e.g., 1 dataset trained on large MT data). If an HPO algorithm performs efficiently in finding good hyperparameter configurations on many MT datasets, then we can more reasonably believe that it will run quickly on a new dataset, regardless of the underlying MT data characteristics. Even if the best configuration on one MT dataset does not transfer to another, a robust HPO algorithm should still be capable of finding good hyperparameters because the algorithm learns from scratch on each dataset independently.

MT Data and Setup
To create a robust HPO benchmark, we trained NMT models on six different parallel corpora, which exhibit a variety of characteristics: TED Talks: We trained Chinese-English (zh-en) and Russian-English (ru-en) models on the datasplit of Duh (2018). This is a mid-resource setup, where D train consists of 170k lines for zh-en and 180k lines for ru-zh. D valid has 1,958 sentences and is multiway parallel for both language-pairs. WMT2019 Robustness task : We trained models on Japanese-English data, in both directions (ja-en, en-ja). D train has 4 M lines from a mix of domains. D valid is a concatenation of 4k mixed-domain sentences and 1k Reddit sentences, for a total of 5,405 lines. The goal of the Robustness task is to test how NMT systems perform on non-standard and noisy text (e.g., Reddit). Low Resource tasks: We trained models using the IARPA MATERIAL datasets for Swahili-English (sw-en) and Somali-English (so-en). D train consists of only 24k lines for both language pairs (BUILD set), and D valid consists of 2675 lines (ANALYSIS2 set).
Although there are many potential MT datasets we could choose from, we believe these six datasets form a good representative set. It ranges from high-to-low resource; it contains both noisy and clean settings. These datasets also have different levels of similarity-for example, zh-en and ru-en TED talks use the same multiway parallel D valid , so one could ask whether the optimal hyperparameters transfer.
The text is tokenized by Jieba for Chinese, by Kytea for Japanese, and by the Moses tokenizer for the rest. Byte pair encoding (BPE) segmentation (Sennrich et al., 2016) is learned and applied separately for each side of bitext. We train Transformer NMT models with Sockeye 3 (Hieber et al., 2017), focusing on these hyperparameters: 3 https://github.com/awslabs/sockeye.
• preprocessing configurations: number of BPE symbols 4 (bpe) • training settings: initial learning rate (init lr) for the Adam optimizer • architecture designs: 5 number of layers (#layers), embedding size (#embed), number of hidden units in each layer (#hidden), number of heads in self-attention (#att heads).
These hyperparameters are chosen because they significantly affect both accuracy and speed of the resulting NMT. Other hyperparameters are kept at their Sockeye defaults. 6 Table 1 shows our overall hyperparameter space Λ; in total among all six datasets, we have 1,983 models; Table 2 shows the exact number of models per dataset, along with the best models and their hyperparameter settings. 7 Rationale for Hyperparameter Values: There are various design trade-offs in deciding the range and granularity of hyperparameter values. First, we might expand on a wider range of values (e.g., change #hidden = {1024, 2048} to {512, 1024, 2048, 4096}). The effect of this is that we test the HPO algorithm on a wider range of inputs, with potentially more variability in metrics like BLEU and inference speed. Second, we might expand on a more finegrained range of values (e.g., change #hidden = {1024, 2048} to {1024, 1536, 2048}). This might result in smoother metrics, making it easier for HPO algorithms to learn. Although wider range and finer granularity are desirable properties for a HPO dataset, each additional value causes an exponential increase in the number of models because of the cross-product of all values. In general, we think Table 1 Table 2: For each language pair, we report the number of NMT systems trained on it, the oracle best BLEU we obtained, and its corresponding hyperparameter configuration.

Objectives: Accuracy and Cost
We train all models on D train until they converge in terms of perplexity on D valid . We then record various performance measurements: • Translation accuracy: BLEU (Papineni et al., 2002) and perplexity on D valid .
• Computational cost: GPU wall clock time for decoding D valid , number of updates for the model to converge, GPU memory used for training, total number of model parameters.
In this paper, we use BLEU on D valid for single-objective experiments; we use BLEU and decoding time for multiobjective experiments.

Hyperparameter Importance/Correlation
We might be interested in seeing whether good configurations are always good across datasets. This can be done by ranking configurations by BLEU for each dataset, then measuring correlation between rankings. We show the Spearman's correlation coefficient in Figure 4. NMT systems with same language pairs (ja-en vs. en-ja) are highly correlated. On the contrary, other pairs show low correlation (0.084 for ja-en vs. so-en), implying the need to run HPO on new datasets separately.
The table-lookup approach also enables indepth analyses of how hyperparameters generally affect system performance. Following , we assess the importance of  Questions may arise over whether the results on en-ja can be taken as general conclusions. We find that it is dataset-dependent-hyperparameter importance ranking differs across language pairs, and is dependent on the range and granularity of hyperparameters considered. As shown in the right column of Figure 5, bpe is the most important hyperparameter for sw-en, instead of #embed. This shows the diversity of our selected MT datasets and the hyperparameter importance analysis is a good tool for probing the search space characteristics of these datasets.

Reproducible and Efficient Benchmarks
Our table-lookup dataset enables reproducible and efficient benchmarks for HPO of NMT systems. Li and Talwalkar (2019) introduce two notions of reproducibility: exact reproducibility (the reproducibility of reported experimental results); and broad reproducibility (the generalization of the experimental results). 9 Our benchmarks are exact reproducible in the sense that we provide the tables that record all model results (Section 3.3) and the code to run and evaluate our HPO algorithms (Section 6). However, they are not guaranteed to be broad reproducible, because the generalizability of the results might be restricted due to fixed collections of hyperparameter configurations, the variance associated with multiple runs, and the unknown best representative set of MT data. As a result, in this work, we should be careful to not make general conclusions from the observations, but to show how the dataset can be potentially used in facilitating HPO research. 9 They comment: ''Of the 12 papers published since 2018 at NeurIPS, ICML, and ICLR that introduce novel Neural Architecture Search methods, none are exactly reproducible.''

Evaluation Protocols
To assess HPO method performance, we measure the runtime to reach a quality indicator (e.g., BLEU) target value. The runtime is defined as the number of NMT models trained, or equivalently the number of function evaluations f (λ) in Figure 2. We consider two ways to measure the HPO performance: fixed-target and fixedbudget.

Single-Objective Evaluation Metrics
For single-objective optimization, we have: • fixed-target best (ftb): We fix the quality indicator value to the best value in the dataset and measure runtime to reach this target.
• fixed-target close (ftc): We measure the runtime to reach a target that is slightly less than the oracle best. This is useful when one can tolerate some performance loss.
• fixed-budget (fb): We fix the budget of function evaluations and measure the difference between the oracle best quality indicator value (e.g., oracle best BLEU) in the dataset vs. the best value achieved by systems queried by the HPO method.
The fixed-budget metric asks what is the best possible system assuming a hard constraint on training resources. The fixed-target metrics ask how much training information is needed to find the best (or approximate best) system in the dataset.

Multiobjective Evaluation Metrics
In practice, one might desire to optimize multiple objectives, such as translation accuracy and speed. Suppose we have J objectives, and they can be jointly represented as F (λ) = [f 1 (λ), f 2 (λ), · · · , f J (λ)]. As it is unlikely that any one λ will optimize all objectives simultaneously, we adopt the concept of Pareto optimality (Godfrey et al., 2007). In the context of minimization, for at least one j. If nothing dominates λ, we call it the Pareto optimal solution. The set of all Pareto solutions is referred to as the Pareto front, that is, {λ | ∄λ ′ ∈ Λ : λ ′ ≺ λ}. Intuitively, these are solutions satisfying all possible trade-offs in the multiobjective space. Figure 1 shows an example of Pareto solutions that maximize BLEU and minimize speed.
For multiobjective optimization, the quality indicator becomes the Pareto front, thus we have: • fixed-target all (fta): We measure the runtime to find all points on the Pareto front.
• fixed-target one (fto): We measure the runtime to get at least one Pareto point.
• fixed-budget (fbp): We fix the budget of function evaluations and measure the number of Pareto-optimal points obtained.
In the literature, a common way to compare HPO methods is to plot quality indicator value as a function of runtime on a graph (e.g., see Figure 6). The proposed metrics can be viewed as summary statistics drawn as line thresholds on such graphs (Hansen et al., 2016), where the budget/target is set to a value appropriate for the use case.

Repeated Trials
Some HPO methods may be sensitive to randomness in initial seeds {λ init } (Feurer et al., 2015). We suggest that repeated randomized trials are important for a rigorous evaluation, and this is only feasible with a table-lookup dataset. In our experiments, we average results of HPO runs across 100 trials, where each trial is seeded with a different set of 3 random initial hyperparameter settings.

Methods
We now describe two HPO/SMBO methods used in our experiments: Bayesian optimization 10 is a popular method. Graph-based SMBO is a novel method that adapts ideas in graph-based semi-supervised learning to the HPO problem.

Bayesian Optimization (BO)
Given a target function f : Λ → R, Bayesian optimization (Brochu et al., 2010;Shahriari et al., 2015;Frazier, 2018) aims to find an input λ ⋆ ∈ arg min λ∈Λ f (λ). It models f with a posterior probability distribution p(f | L), where L is a set of observed points. This posterior distribution is updated each time we observe f at a new point λ p . The utility of each candidate 10 There are works adopting Bayesian optimization for HPO of statistical machine translation systems (Miao et al., 2014;Beck et al., 2016). point is quantified by an acquisition function a : Λ → R, and λ p ∈ arg max λ∈Λ a(λ). In practice, a prominent choice for p(f | L) is Gaussian process regression, and a common acquisition function is Expected Improvement (EI).
In the case of HPO, the kernel k() measures the similarity between hyperparameter configurations and µ() is a prediction of the f () values of not-evaluated hyperparameters.

Expected Improvement (EI)
The EI score  is defined as: where f min is the best observed value thus far, andf (λ) = µ(λ). When the predictionf (λ) follows a normal distribution as in the GP, EI can be computed in a closed form. Our acquisition function computes EI for each point in the grid of hyperparameters, and queries the one with largest value.

Graph-Based SMBO (GB)
Semi-supervised learning addresses the question how to utilize a handful of labeled data and a large amount of unlabeled data to improve prediction accuracy. Graph-based semi-supervised learning (GBSSL, Zhu et al., 2003;Zhu, 2005) describes the structure of data with a graph, where each vertex is a data point and each weighted edge reflects the similarity between vertices. It makes a smoothness assumption that neighbors connected 11 For simplicity, we assume a mean of 0 for the prior. by edges tend to have similar labels, and labels can propagate throughout the graph. In SMBO surrogate modeling, we hope to make predictions for the unlabeled or not-evaluated points in the hyperparameter space based on the information of labeled or evaluated points. If we pre-define the set of all potential points, then this becomes highly related to semisupervised learning. From this point of view, we propose GBSSL equipped with suitable acquisition functions as a new SMBO method for searching over a grid of representative hyperparameter configurations.

Graph-Based Regression
Suppose we have a graph G = (V, E) with nodes V corresponding to n points, of which L denotes the set of labeled points {(λ 1 , f (1)), · · · , (λ l , f (l))}, where f (i) is short for f (λ i ), and U denotes the set of unlabeled points {λ l+1 , · · · , λ l+u }, where n = l + u. The edges E are represented by a n × n weight matrix W . For instance, W can be given as the radial basis function (RBF): Note that G is not necessarily fully connected; in practice, kNN graphs with a small k turn out to perform well, where nodes i, j are connected if i is in j's k-nearest-neighborhood or vice versa. 12 Because closer points are assumed to have similar labels, we define the energy function as: and we constrain f (i), i ∈ L or f L to be true labels and aim to find f (i), i ∈ U or f U that minimizes the energy. We define a diagonal matrix D, where D ii = j W ij and the combinatorial Laplacian ∆ = D − W , Equation (5) can then be rewritten to E(f ) = f T ∆f . If we partition the Laplacian matrix into blocks: 12 In experiments, based on initial tuning, we set kNN so that each point has on average n 7 neighbors.
we can predict the f () values for unlabeled points by:

Expected Influence (EIF)
We propose a novel acquisition function called expected influence that exploits the graph structure. The idea is to query the point such that, if its f () is observed, has the highest potential to change the f () of all other points as we re-run label propagation through the graph. We first scale the labels on the graph f (i) ∈ R to be between 0 or 1. The best labeled point is set to 1; for the other labeled points, we first compute the probability that a random walk starting at 1 reaches it, then set the label to be 1 if the probability is larger than 0.5 and 0 otherwise.
If we were to query an unlabeled point k, there are two scenarios: Its label is either 1 with probability f (k) or 0 with probability 1−f (k). For each scenario, we then consider including k as a newly added ''labeled'' point and re-running label propagation. f +(λ k ,1) (i) are the new predictions for points i in the scenario where k is added with label 1. If k is an influencer in the positive direction, this means that many points i will now have large f +(λ k ,1) (i); otherwise, f +(λ k ,1) (i) might be small on average in magnitude. On the other hand, suppose we add k with label 0 and run label propagation again to obtain new predictions f +(λ k ,0) (i). If k is an influencer in the negative direction, this means that f +(λ k ,0) (i) will be small (or conversely 1 − f +(λ k ,0) (i) will be large).
We can now define an influence score and have the acquisition function seeking point p that maximizes the following: Intuitively, we try adding each unlabeled point as either a desirable point (label 1) or undesirable point (0). We measure whether this addition changes the result of GB regression, and finally query the hyperparameter that is expected to cause the most significant change.

BO vs. GB
There is a connection between the BO and GB due to the link between GPs and graphs. The Algorithm 1: Multiobjective SMBO Input : Initial seeds {λ init }, Budget B Output: Pareto-front approximation P 1 L ← {· · · (λ init , F (λ init )) · · · } 2 while b <= B do 3 P ← Compute the Pareto front of L 4 Fit surrogate modelsf 1 , · · · ,f J on L 5 Select a new point λ p based on an infill criterion and surrogate model predictions GB method defines a Gaussian random field on the graph, which is a multivariate Gaussian distribution on the nodes. This is equivalent to ''finite set'' GPs. Zhu (2005) showed that the kernel matrix K of the finite set GP is equivalent to the inverse of a function of the graph Laplacian ∆, that is, K = (2β(∆ + I σ 2 )) −113 . The difference between the finite set GP and GP is that the kernel matrix of the former is defined on L ∪ U , while the latter is defined on Λ. As a semisupervised method, the label propagation rule of GB (Equation (7)) shows that all the nodes on the graph contribute to the prediction of a single unlabeled node, whereas for GP, the posterior predictive distribution of a new point does not depend on other unlabeled points as shown by Equation (1).
The main advantage of GB is that it offers flexibility to build graphs over the search space. For instance, one can build a graph with configurations from different model architectures, for example, RNN, CNN, and Transformers. Nodes of the same architecture might gather into a cluster, and clusters can be connected with each other. One can also manipulate the edge weights by manually defined heuristics. One example of such rules could be Euclidean distance scaled by hyperparameter importance. We leave this as future work.
The theoretical caveat of the GB method is that it is restricted to a discrete search space defined by a graph. If a dense grid is desired to mimic a continuous search space, increasing time and space complexity would make it a less efficient method.

Multiobjective Optimization
For multiobjective optimization, we can use the same surrogate models to estimate eachf j independently; but we need a new acquisition 13 β and σ are adjustable parameters. function that considers the Pareto front. Various methods have been proposed (Zitzler and Thiele, 1998;Ponweiser et al., 2008;Picheny, 2015;Shah and Ghahramani, 2016;Svenson and Santner, 2016). Here, we adopt the expected hypervolume improvement (EHVI) method (Emmerich et al., 2011), which is a generalization of EI. EHVI as an infill criterion and can be combined with different surrogate models. Algorithm 1 provides pseudo-code for the framework.

Experiments and Results
We evaluate HPO methods on six NMT tasks with the provided benchmark dataset and report their performance measured by three runtime-based assessment metrics mentioned in Section 4. The code base is provided to ensure reproducibility. 14

Single-Objective Optimization
For single-objective optimization, our goal is to find a hyperparameter configuration giving the highest BLEU score over a predefined grid.

Experimental Comparison
We run the comparison with two surrogate models, two kernels, 15 and two acquisition functions, leading to the following HPO systems, where all the GB systems are introduced by this work: • RS: random search (Bergstra and Bengio, 2012), which uniformly samples hyperparameter configurations at random over the grid.
• BO EI M: GP-based BO with Matérn52 covariance function and expected improvement as acquisition function.
• BO EI R: GP-based BO with RBF kernel and EI as acquisition function.
• GB EI M: GB with Matérn52 kernel and EI as acquisition function. 16 14 https://github.com/Este1le/gbopt. 15 We choose Matérn52 and RBF kernel because they exhibit different properties and are both frequently used in literature. As shown in Rasmussen (2003), a parameter ν of the Matérn class of covariance functions can affect the smoothness of the functions drawn from GP. For ν = 1 2 , the process becomes very rough, and for ν → ∞, the covariance function converges to RBF kernel. 16 We can make an equivalence between the covariance matrix in multivariate Gaussian distribution and the inverse of a function of the graph Laplacian ∆ (see Section 5.3 for details), so EI can also be applied to GB models. Table 3: Evaluation on NMT models trained with different language pairs for single-objective (BLEU) optimization. Results are averaged over 100 trials and standard deviations are also reported. Fixed-target best (ftb) and fixed-target close (ftc) are measured by number of model evaluations, and fixed-budget (fb) is measured by BLEU difference. For ftc, the tolerance of performance degradation is set to 0.5 BLEU. 17 For fb, the runtime budget is set to 20. 18

zh-en
• GB EI R: GB with RBF kernel and EI.
• GB EIF M: GB with Matérn52 kernel and expected influence as acquisition function.
• GB EIF R: GB with RBF and EIF.
We use the George library (Ambikasaran et al., 2014) for GP implementation. For all the methods, configurations are sampled without replacement.

Results
Results for single-objective optimization are summarized in Table 3: • RS always needs to explore roughly half of all the NMT models to get the best one (ftb).
• The effectiveness of BO is confirmed: On swen, BO EI M only takes 10% of the runtime used by RS to achieve the optima.
• For ftb, the best GB outperforms the best BO on four of the six datasets: on en-ja, GB EI M reduces the ftb runtime of BO EI M by 38. GB EIF often works better than GB EI.
• Matérn kernel and RBF kernel are almost equally good for both BO and GB. 17 Except for en-ja, where tolerance is set to 1 BLEU, because BLEU difference between top two models is > 0.5. 18 Including three initial evaluations.
• Adjusting initialization can result in a noticeable variance on performance. We suggest that researchers experiment with enough random trials when evaluating HPO systems.

Multiobjective Optimization
We now show benchmarks for multiobjective optimization. Our goal is to search for configurations achieving higher BLEU and less decoding time.

Experimental Comparison
We run the comparison on the following systems, where GB systems are introduced by this work: • RS: random search, uniformly samples the configurations at random.
• BO M: GP-based BO equipped with Matérn kernel and EHVI as the infill criterion.
• BO R: GP-based BO with RBF kernel and EHVI.
• GB M: GB equipped with Matérn kernel and EHVI as the infill criterion.
• GB R: GB with RBF kernel and EHVI.

Results
The multiobjective optimization evaluation results are summarized in Table 4: • RS is a bad choice for multiobjective optimization, if one aims to quickly collect as many Pareto-optimal configurations as possible: To get all the true optima, RS usually needs to go through the whole search space (fta), and with fixed budget it obtains much fewer Pareto points than other methods (fbp).
• BO is generally superior across datasets. On sw-en, it only spends less than half of the time that RS takes to get the Pareto set (344 vs. 719), and can find 8.6 more Pareto points than RS with 200 NMT models evaluated.
• GB provides comparable performance as BO on four datasets, whereas on sw-en and so-en, BO noticeably outperforms GB, which might not be a perfect solution for a multiobjective task. For single-objective optimization, we compare the best BLEU and mean squared error (MSE), which is the averaged squared difference between ground-truth BLEU and predictions, achieved by different HPO methods across time. We can see from Figure 6 (left) that BO and GB converge much faster than RS, and GB is superior over time. This could be partly explained by Figure 6 (right), GB can already fit the data well in the beginning, while BO starts from a much larger MSE and decreases gradually.
For multiobjective optimization, we show the evolution of Pareto-optimal fronts in Figure 7. There is a trend that Pareto fronts are moving towards the lower right corner at each iteration, verifying the effectiveness of our HPO methods.

Effect of Random Initialization
NMT training might not be deterministic due to the random initialization of model parameters. All the experimental results so far are obtained by a single run using one random seed. In order to explore the variance of the model performance induced by initialization effects, we fix the hyperparameter configurations and train models initialized with various random seeds. Specifically, we select five hyperparameter configurations, 20 and retrained them for additional five times each with different random initializations. We did this for two datasets: the low-resource sw-en task and the larger WMT2019 ja-en task.  Step'' is the number of evaluated MT models. Gray circles form the Pareto set of initial seeds. In this example, all three initial seeds happen to be Pareto points. Gold stars are the Pareto solutions of the dataset. Lower-right corner is better.
The results on ja-en and sw-en are shown in Figure 8. The variance of performance is kept in a small range in most cases and the ranking of configurations remains about the same when different random seeds are applied. Based on this observation, we think that it is a reasonable strategy to use a single run to build table-lookup datasets; but at the same time it should be understood that the BLEU scores in the lookup table are only approximations. We note that there can be a few cases where variance is large, and this might be best addressed by inventing HPO methods that explicitly accounts for such uncertainty.

Related Work
To alleviate the computational burden for benchmarking HPO methods and to improve research reproducibility, several studies have explored the table-lookup framework.  published a mix of datasets focusing on feed forward neural networks. Ying et al. (2019) released a dataset of convolutional architectures for image classification problems. To the best of our knowledge, this work is the first that focuses on NMT and transformer models.
One challenge with table-lookup is that sufficient coverage of the hyperparameter grid is assumed. Eggensperger et al. (2015) and  propose using a predictive metamodel trained on a table-lookup benchmark to approximate hyperparameters that are not in the table. This is an interesting avenue for future work.
Studies on HPO for NMT are scarce. Qin et al. (2017) propose an evolution strategy-based HPO method for NMT. So et al. (2019) apply NAS to Transformer on NMT tasks. There is also work on empirically exploring hyperparameters and architectures of NMT systems (Bahar et al., 2017;Britz et al., 2017;Lim et al., 2018), though the focus is on finding general best-practice configurations. This differs from the goal of HPO, which aims to find the best configuration specific to a given dataset.

Conclusions
In this paper, we presented a benchmark dataset for hyperparameter optimization of neural machine translation systems. We provided multiple evaluation protocols and analysis approaches for comparing HPO methods. We benchmarked Bayesian optimization and a novel graph-based semi-supervised learning method on the dataset for both single-objective and multiobjective optimization. Our hope is that this kind of dataset will facilitate reproducible research and rigorous evaluation of HPO for complex and expensive models.