Pushdown Automata in Statistical Machine Translation

Synchronous context-free grammars (SCFGs) are now widely used in statistical machine translation, with Hiero as the preeminent example (Chiang 2007). Given an SCFG and an n-gram language model, the challenge is to decode with them, that is, to apply them to source text to generate a target translation.

Decoding is complex in practice, but it can be described simply and exactly in terms of the formal languages and relations involved. We will use this description to introduce and analyze pushdown automata (PDAs) for machine translation. This formal description will allow close comparison of PDAs to existing decoders which are based on other forms of automata. Decoding can be described in terms of the following steps:

• 1. 

  Translation: The first step is to compose the finite language {s}, which represents the source sentence to be translated, with the algebraic relation for the translation grammar G. The result of this composition projected on the output side is , a weighted context-free grammar that contains all possible translations of s under G. Following the usual definition of Hiero grammars,we assume that G does not allow unbounded insertions so that is a regular language.

• 2. 

  Language Model Application: The next step is to compose the result of Step 1 with the weighted regular grammar defined by the n-gram language model, M. The result of this composition is , whose paths are weighted by the combined language model and translation scores.

• 3. 

  Search: The final step is to find the path through that has the best combinedtranslation and language model score.

We will show that PDAs provide a general framework to describe key aspects of several existing and novel translation algorithms. We note that PDAs have long been used to describe parsing algorithms (Aho and Ullman 1972; Lang 1974), and it is well known that pushdown transducers, the extended version of PDA with input and output labels in each transition, do not have the expressive power needed to generate synchronous context-free languages. For this reason, we do not use PDAs to implement Step 1 in decoding: throughout this article a CYK-like parsing algorithm is always used for Step 1. However, we do use PDAs to represent the regular languages produced in Step 1 and in the intersection and shortest distance operations needed for Steps 2 and 3.

We introduce HiPDT, a hierarchical phrase-based decoder that uses a PDA representation for the target language. The architecture of the system is shown in Figure 2, where we contrast it with HiFST (de Gispert et al. 2010). Both decoders parse the sentence with a grammar G using a modified version of the CYK algorithm to generate the translation search space as an RTN. Each decoder then follows a different path: HiFST expands the RTN into an FSA, intersects it with the language model, and then prunes the result; HiPDT performs the following steps:

• 1. 

  Convert the RTN into PDA using the replacement algorithm. The PDA representation for the example grammar in Section 1 is shown in Figure 1.The algorithm will be described in Section 3.2.

• 2. 

  Apply the language model scores to the PDA by composition. This operation is described in Section 3.3.

• 3. 

  Perform either one of the following operations:

  • (a) 

    Shortest path through the PDA to get the exact best translation under the model. Shortest distance/path algorithm is described in Section 3.4.

  • (b) 

    Pruned expansion to an FSA. This expansion uses admissible pruning and outputs a lattice. We do this for posterior rescoring steps. The algorithm will be presented in detail in Sections 3.5 and 3.5.2.

The principal difference between the two decoders is the point at which finite-state expansion is performed. In HiFST, the RTN representation is immediately expanded to an FSA. In HiPDT, the PDA pruned expansion or shortest path computation is done after the language model is applied, so that all computation is done with respect to both the translation and language model scores.

The use of RTNs as an initial translation representation is somewhat influenced by the development history of our FST and SMT systems. RTN algorithms were available in OpenFST at the time HiFST was developed. HiPDT was developed as an extension to HiFST using PDA algorithms, and these have subsequently been included in OpenFST. A possible alternative approach could be to produce a PDA directly by traversing the CYK grid. WFSAs could then be generated by PDA expansion, with a computational complexity in speed and memory usage similar to the RTN-based approach. We present RTNs as the initial translation representation because the generation of RTNs during parsing is straightforward and has been previously presented (de Gispert et al. 2010). We note, however, that RTN composition is algorithmically more complex than PDA (and FSA) composition, so that RTNs themselves are not ideal representations of if a language model is to be applied. Composition of PDAs with FSAs will be discussed in Section 3.3.

Figure 3 continues the simple translation example from earlier, showing how HiPDT and HiFST both benefit from the compactness offered by WFSA epsilon removal, determinization, and minimization operations. When applied to PDAs, these operations treat parentheses as regular symbols. Compact representations of RTNs are shared by both approaches. Figure 4 illustrates the PDA representation of the translation space under a slightly more complex grammar that includes rules with alternative orderings of nonterminals. The rule S→〈X₁s₂X₂, t₁X₁X₂〉 produces the sequence ‘t₁t₃t₄t₅t₆’, and S→〈X₁s₂X₂, t₂X₂X₁〉 produces ‘t₂t₅t₆t₃t₄’. The PDA efficiently represents the alternative orderings of the phrases ‘t₃t₄’ and ‘t₅t₆’ allowed under this grammar.

In addition to translation, this architecture can also be used directly to carry out source-to-target alignment, or synchronous parsing, under the SCFG in a two-step composition rather than one synchronous parsing stage. For example, by using M as the automata that accepts ‘t₁t₂t₃t₆t₇’, Step 2 will yield all derivations that yield this string as a translation of the source string. This is the approach taken in Iglesias et al. (2009a) and de Gispert et al. (2010) for the RTN/FSA and in Dyer (2010b) for hypergraphs. In Section 4 we analyze how PDAs can be used for alignment.

We summarize here the aims of this article.

• We will show how PDAs can be used as compact representations of the spaceof candidate translations generated by a hierarchical phrase-based SCFG when applied to an input sentencesand intersected with a language model M.

  We have described the architecture of HiPDT, a hierarchical phrase-based decoder based on PDAs, and have identified the general-purpose algorithms needed to perform translation and alignment; in doing so we have highlighted the similarities and differences relative to translation with FSAs (Section 1.1). We will provide a formal description of PDAs (Section 2) and present in detail the associated PDA algorithms required to carry out Steps 2 and 3, including RTN replacement, composition, shortest path, expansion, and pruned expansion (Section 3).

• We will show both theoretically and experimentally that the PDA representation is well suited for exact decoding under a large SCFG and a small language model.

  An analysis of decoder complexity in terms of the automata used in the representation is presented (Section 3). One important aspect of the translation task is whether the search for the best translation is admissible (or exact) under the translation and language models. Stated differently, we wish to know whether a decoder produces the actual shortest path found or whether some form of pruning might have introduced search errors. In our formulation, we can exclude inadmissible pruning from the shortest-path algorithms, and doing so makes it straightforward to compare the computational complexity of a full translation pipeline using different representations of (Section 4). We empirically demonstrate that a PDA representation is superior to an FSA representation in the ability to perform exact decoding both in an inversion transduction grammar–style word alignment task and in a translation task with a small language model (Section 4). In these experiments we take HiFST as a contrastive system for HiPDT, but we do not present experimental results with hypergraph representations. Hypergraphs are widely used by the SMT community, and discussions and contrastive experiments between HiFST and cube pruning decoders are available in the literature (Iglesias et al. 2009a; de Gispert et al. 2010).

• We will propose a two-pass translation decoding strategy for HiPDT based on entropy-pruned first-pass language models.

  Our complexity analysis prompts us to investigate decoding strategies based on large translation grammars and small language models. We describe, implement, and evaluate a two-pass decoding strategy for a large-scale translation task using HiPDT (Section 5). We show that entropy-pruned language models can be used in first-pass translation, followed by admissible beam pruning of the output lattice and subsequent rescoring with a full language model. We analyze the search errors that might be introduced by a two-pass translation approach and show that these can be negligible if pruning thresholds are set appropriately (Section 5.2). Finally, we detail the experimental conditions and speed/performance tradeoffs that allow HiPDT to achieve state-of-the-art performance for large-scale SMT under a large grammar (Section 5.3), including lattice rescoring steps under a vast 5-gram language model and lattice minimum Bayes risk decoding (Section 5.4).

  With this translation strategy HiPDT can yield very good translation performance. For comparison, the performance of this Chinese-to-English SMT described in Section 5.4 is equivalent to that of the University of Cambridge submission to the NIST OpenMT 2012 Evaluation.¹

Informally, pushdown transducers are finite-state transducers that have been augmented with a stack. Typically this is done by adding a stack alphabet and labeling each transition with a stack operation (a stack symbol to be pushed onto, popped, or read from the stack) in addition to the usual input and output labels (Aho and Ullman 1972; Berstel 1979) and weight (Kuich and Salomaa 1986; Petre and Salomaa 2009). Our equivalent representation allows a transition to be labeled by a stack operation or a regular input/output symbol, but not both. Stack operations are represented by pairs of open and close parentheses (pushing a symbol on and popping it from the stack). The advantage of this representation is that it is identical to the finite automaton representation except that certain symbols (the parentheses) have special semantics. As such, several finite-state algorithms either immediately generalize to this PDA representation or do so with minimal changes. In this section we formally define pushdown automata and transducers.

A (restricted) Dyck language consist of “well-formed” or “balanced” strings over a finite number of pairs of parentheses. Thus the string ( [ ( ) ( ) ] { } [ ] ) ( ) is in the Dyck language over three pairs of parentheses (see Berstel 1979 for a more detailed presentation).

More formally, let A and be two finite alphabets such that there exists a bijection f from A to . Intuitively, f maps an open parenthesis to its corresponding close parenthesis. Let denote f(a) if a ∈ A and f⁻¹(a) if . The Dyck languageD_A over the alphabet is then the language defined by the following context-free grammar: S → ε, S → SS and for all a ∈ A. We define the mapping as follows. c_A(x) is the string obtained by iteratively deleting from x all factors of the form with a ∈ A. Observe that . Finally, for a subset B ⊆ A, we define the mapping r_B : A* → B* by r_B(x₁ … x_n) = y₁ … y_n with y_i = x_i if x_i ∈ B and y_i = ε otherwise.

A semiring is a ring that may lack negation. It is specified by a set of values , two binary operations ⊕ and ⊗, and two designated values and . The operation ⊕ is associative, commutative, and has as identity. The operation ⊗ is associative, has identity , distributes with respect to ⊕, and has as annihilator: for all . If ⊗ is also commutative, we say that the semiring is commutative.

The probability semiring (ℝ₊, +, ×, 0, 1) is used when the weights represent probabilities. The log semiring (ℝ ∪ {∞}, ⊕_log, +, ∞, 0), isomorphic to the probability semiring via the negative-log mapping, is often used in practice for numerical stability. The tropical semiring (ℝ ∪ {∞}, min, +, ∞, 0) is derived from the log semiring using the Viterbi approximation. These three semirings are commutative.

A weighted pushdown automaton (PDA) T over a semiring is an 8-tuple where Σ is the finite input alphabet, Π and are the finite open and close parenthesis alphabets, Q is a finite set of states, I ∈ Q the initial state, F ⊆ Q the set of final states, a finite set of transitions, and the final weight function. Let e = (p[e], i[e], w[e], n[e]) denote a transition in E; for simplicity, (p[e], i[e], n[e]) denotes an unweighted transition (i.e., a transition with weight ).

A path π is a sequence of transitions π = e₁ …e_n such that n[e_i] = p[e_{i + 1}] for 1 ≤ i < n. We then define p[π] = p[e₁], n[π] = n[e_n], i[π] = i[e₁] ⋯ i[e_n], and w[π] = w[e₁] ⊗ … ⊗ w[e_n]. A path π is accepting if p[π] = I and n[π] ∈ F. A path π is balanced if . A balanced path π accepts the string x ∈ Σ* if it is a balanced accepting path such that r_Σ(i[π]) = x.

Finally, a weighted pushdown transducer (PDT) T over a semiring is a 9-tuple where Σ is the finite input alphabet, Δ is the finite output alphabet, Π and are the finite open and close parenthesis alphabets, Q is a finite set of states, I ∈ Q the initial state, F ⊆ Q the set of final states, E ⊆ Q × (Σ ∪ ∪ {ε}) × × Q a finite set of transitions, and the final weight function. Let e = (p[e], i[e], o[e], w[e], n[e]) denote a transition in E. Note that a PDA can be seen as a particular case of a PDT where i[e] = o[e] for all its transitions. For simplicity, our following presentation focuses on acceptors, rather than the more general case of transducers. This is adequate for the translation applications we describe, with the exception of the treatment of alignment in Section 4.3, for which the intersection algorithm for PDTs and FSTs is given in Appendix A.

In this section we describe in detail the following PDA algorithms: Replacement, Composition, Shortest Path, and (Pruned) Expansion. Although these are needed to implement HiPDT, these are general purpose algorithms, and suitable for many other applications outside the focus of this article. The algorithms described in this section have been implemented in the PDT extension (Allauzen and Riley 2011) of the OpenFst library (Allauzen et al. 2007). In this section, in order to simplify the presentation we will only consider machines over the tropical semiring (ℝ₊ ∪ {∞}, min, +, ∞, 0). However, for each operation, we will specify in which semirings it can be applied.

We briefly give formal definitions for RTNs that will be needed to present the RTN expansion operation. Examples are shown earlier in Figures 1(b) and 3(a). Informally, an RTN is an automaton where some labels, nonterminals, are recursively replaced by other automata. We give the formal definition for acceptors; the extension to RTN transducers is straightforward.

An RTN R over the tropical semiring (ℝ₊ ∪ {∞}, min, +, ∞, 0) is a 4-tuple (N, Σ, (T_ν)_{ν ∈ N}, S) where N is the alphabet of nonterminals, Σ is the input alphabet, (T_ν)_{ν ∈ N} is a family of FSTs with input alphabet Σ ∪ N, and S ∈ N is the root nonterminal.

A sequence x ∈ Σ* is accepted by (R,ν) if there exists an accepting path π in T_ν such that π = π₁e₁ …π_ne_nπ_{n + 1} with i[π_k] ∈ Σ*, i[e_k] ∈ N and such that there exists sequences x_k such that x_k is accepted by (R,i[e_k]) and x = i[π₁] x₁ …i[π_n]x_ni[π_{n + 1}]. We say that x is accepted by R when it is accepted by (R,S). The weight associated by (R, ν) (and by R) to x can be defined in the same recursive manner.

As an example of testing whether an RTN accepts a sequence, consider the RTN R of Figure 6 and the sequence x = aab. The path in the automata T_S can be written as π = π₁e₁π₂, with i[π₁] = a, i[e₁] = X₁, and i[π₂] = b. In addition, the machine (R, i[e₁]) accepts x₁ = a. Because x = i[π₁] x₁i[π₂] , it follows that x is accepted by (R, S).

This algorithm converts an RTN into a PDA. As explained in Section 1.1, this PDT operation is applied by the HiPDT decoder in Step 1, and examples are given in earlier sections (e.g., in figures 1 and 3).

Replacement acts on every transition of the RTN that is associated with a nonterminal. The source and destination states of these transitions are used to define the matched opening and closing parentheses, respectively, in the new PDA. Each RTN nonterminal transition is deleted and replaced by two new transitions that lead to and from the automaton indicated by the nonterminal. These new transitions have matched parentheses, taken from the source and destination states of the RTN transition they replace. Figure 6 gives a simple example.

The complexity of the construction is in O(|T|). If |F_ν| = 1 for all ν ∈ N, then |T| = O( ∑ _{ν ∈ N} |T_ν|) = O(|R|). Creating a superfinal state for each T_ν would lead to a T whose size is always linear in the size of R. In this article, we assume this optimization is always performed. We note here that RTNs can be defined and the replacement operation can be applied in any semiring.

Once we have created the PDA with translation scores, Step 2 in Section 1.1 applies the language model scores to the translation space. This is done by composition with an FSA containing the relevant language model weights.

The class of weighted pushdown transducers is closed under composition with weighted finite-state transducers (Bar-Hillel, Perles, and Shamir 1964; Nederhof and Satta 2003). OpenFST supports composition between automata T₁ and T₂, where T₁ is a weighted pushdown transducer and T₂ is a weighted finite-state transducer. If both T₁ and T₂ are acceptors, rather than transducers, the composition of a PDA and an FSA produces a PDA containing their intersection, and so no separate intersection algorithm is required for these automata. Given this, we describe only the simpler, special case of intersection between a PDA and an FSA, as this is sufficient for most of the translation applications described in this article. The alignment experiments of Section 4.3 do require composition of transducers; the algorithm for composition of transducers is given in Appendix A.

An example of composition by intersection is given in Figure 7. The states of T are created as the product of all the states in T₁ and T₂. Transitions are added as illustrated in Figure 8. These correspond to all paths through T₁ and T₂ that can be taken by a synchronized reading of strings from {a,b}*. The algorithm is very similar to the composition algorithm for finite-state transducers, the difference being the handling of the parentheses. The parenthesis-labeled transitions are treated similarly to epsilon transitions, but the parenthesis labels are preserved in the result. This adds many unbalanced paths to T. In this example, T has five paths but only one balanced path, so that T accepts the string aabb.

Formally, given a PDA and an FSA T₂ = (Σ, Q₂, E₂, I₂, F₂, ρ₂), intersection constructs a new PDA , where T = T₁ ∩ T₂ as follows:

• 1. 

  The new state space is in the product of the input state spaces: Q ⊂ Q₁×Q₂.

• 2. 

  The new initial and final states are I = (I₁, I₂), and F = {(q₁, q₂) : q₁ ∈ F₁, q₂ ∈ F₂}.

• 3. 

  Weights are assigned to final states (q₁, q₂) ∈ Q as ρ(q₁,q₂) = ρ(q₁) + ρ(q₂).

• 4. 

  For pairs of transitions (q₁, a₁, w₁, q₁′) ∈ E₁ and (q₂, a₂, w₂, q₂′) ∈ E₂, a transitionis added between states (q₁,q₂) and (q′₁,q′₂) as specified in Figure 8.

The intersection algorithm given here assumes that T₂ has no input-ε transitions. When T₂ has input-ε transitions, an epsilon filter (Mohri 2009; Allauzen, Riley, and Schalkwyk 2011) generalized to handle parentheses can be used. Note that Steps 1 and 2 do not require the construction of all possible pairs of states; only those states reachable from the initial state and needed in Step 4 are actually generated. The complexity of the algorithm is in O(|T₁| |T₂|) in the worst case, as will be discussed in Section 4. Composition requires the semiring to be commutative.

With a PDA including both translation and language model weights, HiPDT can extract the best translation (Step 3a in Section 1.1). To this end, a general PDA shortest distance/path algorithm is needed.

A shortest path in a PDA T is a balanced accepting path with minimal weight and the shortest distance in T is the weight of such a path. We show that when T has a bounded stack, shortest distance and shortest path can be computed in O(|T|³ log|T|) time (assuming T has no negative weights) and O(|T|²) space. Figure 9 gives a pseudo-code description of the shortest-distance algorithm, which we now discuss.

The core of the shortest distance algorithm is the procedure GetDistance(T, s) which calculates the distances d[s,q] for all states q that can be reached from s. For an FSA, this procedure is called once, as GetDistance(T, I), to calculate d[I,q] ∀ q.

Figure 10 gives an example. When transition (2,(₁,0,5) is processed, GetDistance(T, 5) is called. The distance d[5,7] is computed, and following transitions are logged: B[5,(₁] ←{(7,)₁,0,8)} and B[5,(₂] ←{(7,)₂,0,9)}. Later, when the transition (4,(₂,0,5) is processed, its matching transition (7,)₂,0,9) is extracted from B[5,(₂]. The distance d[4,9] is then found by Equation (5) as d[5,7]. This avoids redundant re-calculation of distances along the shortest balanced path from state 4 to state 9.

We now briefly discuss the shortest distance pseudo-code given in Figure 9. The description may be easier to follow after reading the worked example in Figure 10. Note that the sets C_s are not computed explicitly by the algorithm.

The shortest distance calculation proceeds as follows. Self-distances, that is, d[q,q], are set initially to ∞; when GetDistance(T, q) is called it sets d[q,q] = 0 to note that q has been visited. GetDistance(T, s) starts a new instance of the shortest-distance algorithm from s using the queue , initially containing s. While the queue is not empty, a state is dequeued and its outgoing transitions examined (lines 7–11). Transitions labeled by non-parenthesis are treated as in Mohri 2009 (lines 7–8). When a transition e is labeled by a close parenthesis, e is added to to indicate that this transition balances all incoming open parentheses into s labeled by (lines 9–10). Finally, if e has an open parenthesis, and if its destination has not already been visited, a new instance of GetDistance is started from n[e] (lines 12–13). The destination states of all transitions balancing e are then relaxed (lines 14–16).

The space complexity of the algorithm is quadratic for two reasons. First, the number of non-infinity d[q,s] is |Q|². Second, the space required for storing B is at most in O(|E|²) because for each open parenthesis transition e, the size of |B[n[e], i[e]]| is O(|E|) in the worst case. This last observation also implies that the accumulated number of transitions examined at line 16 is in O(Z |Q| |E|²) in the worst case, where Z denotes the maximal number of times a state is inserted in the queue for a given call of GetDistance. Assuming the cost of a queue operation is Γ(n) for a queue containing n elements, the worst-case time complexity of the algorithm can then be expressed as O(Z |T|³Γ(|T|)). When T contains no negative weights, using a shortest-first queue discipline leads to a time complexity in O(|T|³ log|T|). When all the C_s's are acyclic, using a topological order queue discipline leads to a O(|T|³) time complexity.

As was shown in Section 3.2, when T has been obtained by converting an RTN or a hypergraph into a PDA, the polynomial dependency in |T| becomes a linear dependency both for the time and space complexities. Indeed, for each q in T, there exists a unique s such that d[s,q] is non-infinity. Moreover, for each open parenthesis transition e, there exists a unique close parenthesis transition e′ such that e′ ∈ B[n[e], i[e]]. When each component of the RTN is acyclic, the complexity of the algorithm is O(|T|) in time and space.

The algorithm can be modified (without changing the complexity) to compute the shortest path by keeping track of parent pointers. The notion of shortest path requires the semiring to have the path property: for all a, b in , a ⊕ b ∈ {a, b}. The shortest-distance operation as presented here and the shortest-path operation can be applied in any semiring having the path property by using the natural order defined by ⊕: a ≤ b iff a ⊕ b = a. However, the shortest distance algorithm given in Figure 9 can be extended to work for k-closed semirings using the same techniques that were used by Mohri 2002.

The shortest distance in the intersection of a string s and a PDA T determines if T recognizes s. PDA recognition is closely related to CFG parsing; a CFG can be represented as a PDT whose input recognizes the CFG and whose output identifies the parse (Aho and Ullman 1972). Lang 1974 showed that the cubic tabular method of Earley can be naturally applied to PDAs; others give the weighted generalizations (Stolcke 1995; Nederhof and Satta 2006). Earley's algorithm has its analogs in the algorithm in Figure 9: the scan step corresponds to taking a non-parenthesis transition at line 10, the predict step to taking an open parenthesis at lines 14–15, and the complete step to taking the closed parentheses at lines 16–18.

Specialization to Translation. Following the formalism of Section 1, we are interested in applying shortest distance and shortest path algorithms to automata created as L = T_p ∩ M, where T_p, the translation representation, is a PDA derived from an RTN (via replacement) and M, the language model, is a finite automaton.

For this particular case, the time complexity is and the space complexity is . The dependence on |T_p| is linear, rather than cubic or quadratic. The reasoning is as follows. Given a state q in T_p, there exists a unique s_q such that q belongs to . Given a state (q₁,q₂) in T_p ∩ M, only if , and hence (q₁,q₂) belongs to at most |M| components.

As explained in Section 1.1, HiPDT can apply Step 3b to generate translation lattices. This step is typically required for any posterior lattice rescoring strategies. We first describe the unpruned expansion. However, in practice a pruning strategy of some sort is required to avoid state explosion. Therefore, we also describe an implementation of the PDA expansion that includes admissible pruning under a likelihood beam, thus controlling on-the-fly the size of the output lattice.

Given a bounded-stack PDA T, the expansion of T is the FSA T′ equivalent to T. A simple example is given in Figure 11.

Expansion starts from the PDA initial state. States and transitions are added to the FSA as the expansion proceeds along paths through the PDA. In the new FSA, parentheses are replaced by epsilons, and as open parentheses are encountered on PDA transitions, they are “pushed” into the FSA state labels; in this way the stack depth is maintained along different paths through the PDA. Conversely, when a closing parenthesis is encountered on a PDA path, a corresponding opening parenthesis is “popped” from the FSA state label; if this is not possible, for example, as in state (5,ε) in Figure 11, expansion along that path halts.

The resulting automata accept the same language. The FSA topology changes, typically with more states and transitions than the original PDA, and the number of added states is controlled only by the maximum stack depth of the PDA.

The full expansion operation can be applied to PDA over any semiring. The complexity of the algorithm is linear in the size of T′. However, the size of T′ can be exponential in the size of T, which motivates the development of pruned expansion, as discussed next.

Given a bounded-stack PDA T, the pruned expansion of T with threshold β is an FST T′_β obtained by deleting from T′ all states and transitions that do not belong to any accepting path π in T′ such that w[π] ⊗ ρ[π] ≤ d + β, where d is the shortest distance in T.

A naive implementation consisting of fully expanding T and then applying the FST pruning algorithm would lead to a complexity in O(|T′| log|T′|) = O(e^|T| |T|). Assuming that the reverse T^R of T is also bounded-stack, an algorithm whose complexity is in can be obtained by first applying the shortest distance algorithm from the previous section to T^R and then using this to prune the expansion as it is generated. To simplify the presentation, we assume that F = {f} and ρ(f) = 0.

The motivation for using reversed automaton in pruning is easily seen by looking at FSAs. For an FSA, the cost of the shortest path through a transition (q,x,w,q′) can be stated as d[I,q] + w + d[q′,f]. Distances d[I,q] (i.e., distances from the start state) are computed by the shortest distance algorithm, as discussed in Section 3.4. However, distances of the form d[q′,f] are not readily available. To compute these, a shortest distance algorithm is run over the reversed automaton. Reversal preserves states and transitions, but swaps the source and destination state (see Figure 13 for a PDA example). The start state in the reversed machine is f, so that distances are computed from f; these are denoted d^R[f,q] and correspond to d[q,f] in the original FSA. The cost of the shortest path through an FSA transition (q,x,w,q′) can then be computed as d[I,q] + w + d^R[f,q′].

Calculation for PDAs is more complex. Transitions with parentheses must be handled such that distances through them are calculated over balanced paths. For example, if T in Figure 13 was an FSA, the shortest cost of any path through the transition e = (4,(₂,0,5) could be calculated as d[0,4] + 0 + d[5,10]. However, this is not correct, because d[5,10], the shortest distance from 5 to 10, is found via a path through the transition (7,)₁,0,8).

Pruned expansion is therefore able to avoid expanding transitions that would not contribute to any path that would survive pruning. Prior to expansion of a PDA T to an FSA T′, the shortest distance d in T is calculated. Transitions e = (q, a, w, q′), a ∈ Π, are expanded as transitions e = ((q, z), q, w, (q′,za)) in T′ only if c(e) ≤ d + β, as calculated by Equation (7).

The pruned expansion algorithm implemented in OpenFST is necessarily more complicated than the simple description given here. Pseudo-code describing the OpenFST implementation is given in Appendix B.

The pruned expansion operation can be applied in any semiring having the path property.

We now address the following questions:

• What are the differences between the FSA and PDA representations as observed in a translation/alignment task?

• How do their respective decoding algorithms perform in relation to the complexity analysis described here?

• How many times is exact decoding achievable in each case?

We will discuss the complexity of both HiPDT and HiFST decoders as well as the hypergraph representation, with an emphasis on Hiero-style SCFGs. We assess our analysis for FSA and PDA representations by contrasting HiFST and HiPDT with large grammars for translation and alignment. For convenience, we refer to the hypergraph representation as T_h, and to the FSA and PDA representations as T_f and T_p.

We first analyze the complexity of each MT step described in the introduction:

• 1. 

  SCFG Translation: Assuming that the parsing of the input is performed by a CYK parse, then the CFG, hypergraph, RTN, and PDA representations can be generated in O(|s|³ |G|) time and space (Aho and Ullman 1972). The FSA representation can require an additional time and space because the RTN expansion to FSA can be exponential.

• 2. 

  Intersection: The intersection of a CFG T_h with a finite automaton M can be performed by the classical Bar-Hillel algorithm (Bar-Hillel, Perles, and Shamir 1964) with time and space complexity , where l is the maximum number of symbols on the right-hand side of a grammar rule inT_h. Dyer 2010a presents a more practical intersection algorithm that avoids creating rules that are inaccessible from the start symbol. With deterministicM, the intersection complexity becomes , where l_N is the rank of the SCFG (i.e., l_N is the maximum number of nonterminals on the right-hand side of a grammar rule). With Hiero-styles rules, l_N = 2 so the complexity is in that case.⁵ The PDA intersection algorithm from Section 3.3 has time and space complexity O(|T_p| |M|). Finally, the FSA intersection algorithm has time and space complexity O(|T_f| |M|) (Mohri 2009).

• 3. 

  Shortest Path: The shortest path algorithm on the hypergraph, RTN, and FSA representations requires linear time and space (given the underlying acyclicity) (Huang 2008; Mohri 2009). As presented in Section 3.4, the PDA representation can require time cubic and space quadratic in |M|.

Table 1 summarizes the complexity results for SCFGs of rank 2. The PDA representation is equivalent in time and superior in space complexity to the CFG/hypergraph representation, in general, and it can be superior in both space and time to the FSA representation depending on the relative SCFG and language model (LM) sizes. The FSA representation favors smaller target translation grammars and larger language models.

In practice, the PDA and FSA representations benefit greatly from the optimizations mentioned previously (Figure 3 and accompanying discussion). For the FSA representation, these operations can offset the exponential dependencies in the worst-case complexity analysis. For example, in a translation of a 15-word sentence taken at random from the development sets described later, expansion of an RTN yields a WFSA with 174 × 10⁶ states. By contrast, if the RTN is determinized and minimized prior to expansion, the resulting WFSA has only 34 × 10³ states. Size reductions of this magnitude are typical. In general, the original RTN, hypergraph, or CFG representation can be exponentially larger than the RTN/PDT optimized as described.

Although our interest is primarily in Hiero-style translation grammars, which have rank 2 and a relatively small number of nonterminals, this complexity analysis can be extended to other grammars. For SCFGs of arbitrary rank l_N, translation complexity in time for hypergraphs becomes ; with FSAs the time complexity becomes ; and with PDAs the time complexity becomes . For more complex SCFGs with rules of rank greater than 2, such as SAMT (Zollmann and Venugopal 2006) or GHKM (Galley et al. 2004), this suggests that PDA representations may offer computational advantages in the worst case relative to hypergraph representations, although this must be balanced against other available strategies such as binarization (Zhang et al. 2006; Xiao et al. 2009) or scope pruning (Hopkins and Langmead 2010). Of course, practical translation systems introduce various pruning procedures to achieve much better decoding efficiency than the worst cases given here.

We will next describe the translation grammar and language model for our experiments, which will be used throughout the remainder of this article (except when stated otherwise). In the following sections we assess the complexity discussion with a contrast between HiFST (FSA representation) and HiPDT (PDA representation) under large grammars.

Translation grammars are extracted from a subset of the GALE 2008 evaluation parallel text;⁶ this is 2.1M sentences and approximately 45M words per language. We report translation results on a development set tune-nw (1,755 sentences) and a test set test-nw (1,671 sentences). These contain translations produced by the GALE program and portions of the newswire sections of the NIST evaluation sets MT02 through MT06.⁷ In tuning the systems, MERT (Och 2003) iterative parameter estimation under IBM BLEU⁸ is performed on the development set.

The parallel corpus is aligned using MTTK (Deng and Byrne 2008) in both source-to-target and target-to-source directions. We then follow published procedures (Chiang 2007; Iglesias et al. 2009b) to extract hierarchical phrases from the union of the directional word alignments. We call a translation grammar (G) the set of rules extracted from this process. For reference, the number of rules in G that can apply to the tune-nw is 1.1M, of which 593K are standard non-hierarchical phrases and 511K are strictly hierarchical rules.

We will use two English language models in these translation experiments. The first language model, denoted M₁, is a 4-gram estimated over 1.3B words taken from the target side of the parallel text and the AFP and Xinhua portions of the English Gigaword Fourth Edition (LDC2009T13). We use both Kneser-Ney (Kneser and Ney 1995) and Katz (Katz 1987) smoothing in estimating M₁. Where language model reduction is required, we apply Stolcke entropy pruning (Stolcke 1998) to M₁ under the relative perplexity threshold θ. The resulting language model is labeled as .

The reduction in size in terms of component n-grams is summarized in Table 2. For aggressive enough pruning, the original 4-gram model can be effectively reduced to a trigram, bigram, or unigram model. For both the Katz and the Kneser-Ney 4-gram language models: at θ = 7.5E − 05 the number of 4-grams in the LM is effectively reduced to zero; at θ = 7.5E − 4 the number of 3-grams is effectively 0; and at θ = 7.5E − 3, only unigrams remain. Development set perplexities increase as entropy pruning becomes more aggressive, with the Katz smoothed model performing better under pruning (Chelba et al. 2010; Roark, Allauzen, and Riley 2013).

We will also use a larger language model, denoted M₂, obtained by interpolating M₁ with a zero-cutoff stupid-backoff 5-gram model (Brants et al. 2007) estimated over 6.6B words of English newswire text; M₂ is estimated as needed for the n-grams required for the test sets.

We now compare HiFST and HiPDT in translation with our large grammar G. In this case we know that exact search is often not feasible for HiFST.

We run both decoders over tune-nw with a restriction on memory use of 10 GB. If this limit is reached in decoding, the process is killed.⁹Table 3 shows the number of times each decoder succeeds in finding a hypothesis under the memory limit when decoding with various entropy-pruned LMs . With θ = 7.5 × 10⁻⁹ (row 2), HiFST can only decode 218 sentences, and HiPDT succeeds in 703 cases. The difference in success rates between the decoders is more pronounced as the language model is more aggressively pruned: for θ = 7.5 × 10⁻⁷ HiPDT succeeds for all but three sentences.

As Table 3 shows, HiFST fails most frequently in its initial expansion from RTN to FSA; this operation depends only on the translation grammar and does not benefit from any reduction in the language model size. Subsequent intersection of the FSA with the language model can still pose a challenge, although as the language model is reduced, this intersection fails less often. By contrast, HiPDT intersects the translation grammar with the language model prior to expansion and this operation nearly always finishes successfully. The subsequent shortest path (or pruned expansion) operation is prone to failure, but the risk of this can be greatly reduced by using smaller language models.

In the next section we contrast both HiPDT and HiFST for alignment.

We continue to explore applications characterized by large translation grammars G and small language models M. As an extreme instance of a problem involving a large translation grammar and a simple target language model, we consider parallel text alignment under an Inversion Transduction Grammar (ITG) (Wu 1997). This task, or something like it, is often done in translation grammar induction. The process should yield the set of derivations, with scores, that generate the target sentence as a translation of the source sentence. In alignment the target language model is extremely simple: It is simply an acceptor for the target language sentence so that |M| is linear in the length of the target sentence. In contrast, the search space needs now to be represented with pushdown transducers (instead of pushdown automata) keeping track of both translations and derivations, that is, indices of the rules in the grammar (Iglesias et al. 2009a; de Gispert et al. 2010; Dyer 2010b).

We define a word-based translation grammar G_ITG for the alignment problem as follows. First, we obtain word-to-word translation rules of the form X → 〈s, t〉 based on probabilities from IBM Model 1 translation tables estimated over the parallel text, where s and t are one source and one target word, respectively (∼16M rules). Then, we allow monotonic and inversion transduction of two adjacent nonterminals in the usual ITG style (i.e., add X → 〈X₁X₂, X₁X₂〉 and X → 〈X₁X₂, X₂X₁〉). Additionally, we allow unrestricted source word deletions (X → 〈s, ε〉), and restricted target word insertions (X → 〈X₁X₂, X₁tX₂〉). This restriction, which is solely motivated by efficiency reasons, disallows the insertion of two consecutive target words. We make no claims about the suitability or appropriateness of this specific grammar for either alignment or translation; we introduce this grammar only to define a challenging alignment task.

A set of 2,500 sentence pairs of up to 50 source and 75 target words was chosen for alignment. These sentences come from the same Chinese-to-English parallel data described in Section 4.1. Hard limits on memory usage (10GB) and processing time (10 minutes) were imposed for processing each sentence pair. If HiPDT or HiFST exceeded either limit in aligning any sentence pair, alignment was stopped and a “memory/time failure” was noted. Even if the resource limits are not exceeded, alignment may fail due to limitations in the grammar. This happens when either a particular word pair rule that is not in our Model 1 table, or more than one consecutive target insertions are needed to reach alignment. In such cases, we record a “grammar failure,” as opposed to a “memory/time failure.”

Results are reported in Table 4. Of the 2,500 sentence pairs, HiFST successfully aligns only 41% of the sentence pairs under these time and memory constraints. The reason for this low success rate is that HiFST must generate and expand all possible derivations under the ITG for a given sentence pair. Even if it is strictly enforced that the FSA in every CYK cell contains only partial derivations which produce substrings of the target sentence, expansion often exceeds the memory/time constraints. In contrast, HiPDT succeeds in aligning all sentence pairs that can be aligned under the grammar (89%), because it never fails due to memory or time constraints. In this experiment, if alignment is at all possible, HiPDT will find the best derivation. Alignment success rate (or coverage) could trivially be improved by modifying the ITG to allow more consecutive target insertions, or by increasing the number of word-to-word rules, but that would not change the conclusion in the contrast between HiFST and HiPDT.

The computational analysis from the beginning of this section applies to alignment. The language model M is replaced by an acceptor for the target sentence, and if we assume that the target sentence length is proportional to the source sentence length, it follows that |M| ∝ |s| and the worst-case complexity for HiPDT in alignment mode is O(|s|⁶|G|). This is comparable to ITG alignment (Wu 1997) and the intersection algorithm of Dyer 2010b.

Our experimental results support the complexity analysis summarized in Table 1. HiPDT is more efficient in ITG alignment and this is consistent with its linear dependence on the grammar size, whereas HiFST suffers from its exponential dependence. This use of PDAs in alignment does not rely on properties specific either to Hiero or to ITGs. We expect that the approach should be applicable with other types of SCFGs, although we note that alignment under SCFGs with an arbitrary number of nonterminals can be NP-hard (Satta and Peserico 2005).

The previous complexity analysis suggests that PDAs should excel when used with large translation grammars and relatively small n-gram language models. In hierarchical phrase-based translation, this is a somewhat unusual scenario: It is far more typical that translation tasks requiring a large translation grammar also require large language models. To accommodate these requirements we have developed a two-pass decoding strategy in which a weak version of a large language model is applied prior to the expansion of the PDA, after which the full language model is applied to the resulting WFSA in a rescoring pass. An effective way of generating weak language models is by means of entropy pruning under a threshold θ; these are the language models of Section 4.1. Such a two-pass strategy is widely used in automatic speech recognition (Ljolje, Pereira, and Riley 1999). The steps in two-pass translation using entropy-pruned language models are given here, and depicted in Figure 14.

• Step 1. We translate with and G using the same parameters obtained by MERT for the baseline system, with the exception that the word penalty parameter is adjusted to produce hypotheses of roughly the correct length. This produces translation lattices that contain hypotheses with exact scores under G and : .

• Step 2. These translation lattices are pruned at beamwidth β: .

• Step 3. We remove the scores from the pruned translation lattices, reapply the full language model M₁, and restore the word penalty parameter to the baseline value obtained by MERT. This gives an approximation to : scores are correctly assigned under G and M₁, but only hypotheses that survived pruning at Step 2 are included.

We can rescore the lattices produced by the baseline system or by the two-pass system with the larger language model M₂. If β = ∞ or if θ = 0, the translation lattices obtained in Step 3 should be identical to lattices produced by the baseline system (i.e., the rescoring step is no longer needed). The aim is to increase θ to shrink the language model used at Step 1, but β will then have to increase accordingly to avoid pruning away desirable hypotheses in Step 2.

The two-pass translation procedure requires removal of the weak language model scores used in the initial expansion of the translation search space; this is done so that only the translation scores under G remain after pruning. In the tropical semiring, the weak LM scores can be “subtracted” at the path level from the lattice, but this involves a determinization of an unweighted translation lattice, which can be very inefficient.

We wish to analyze the degree to which the two-pass decoding strategy introduces “modeling errors” into translation. A modeling error occurs in two-pass decoding whenever the decoder produces a translation whose score is less than the best attainable under the grammar and language model (i.e., whenever the best possible translation is discarded by pruning at Step 2). We refer to these as modeling errors, rather than search errors, because they are due to differences in scores assigned by the models M₁ and .

Ideally, we would compare the two-pass translation system against a baseline system that performs exact translation, without pruning in search, under the grammar G and language model M₁. This would allow us to address the following questions:

• Is a two-pass decoding procedure that uses entropy-pruned language models adequate for translation? How many modeling errors are introduced? Does two-pass decoding impact on translation quality?

• Which smoothing/discounting technique is best suited for the first-pass language model in two-pass translation, and which smoothing/discounting technique is best at avoiding modeling errors?

Our grammar G is not suitable for these experiments, as we do not have a system capable of exact decoding under both G and M₁. To create a suitable baseline we therefore reduce G by excluding rules that have a forward translation probability p < 0.01, and refer to this reduced grammar as G_small. This process reduces the number of strictly hierarchical rules that apply to our tune-nw set from 511K to 189K, while the number of standard phrases is unchanged.

Under G_small, both HiFST and HiPDT are able to exactly compose the entire space of possible candidate hypotheses with the language model and to extract the shortest path hypothesis. Because an exact decoding baseline is thus available, we can empirically evaluate the proposed two-pass strategy. Any degradation in translation quality can only be due to the modeling errors introduced by pruning under β with respect to the entropy-pruned .

Figure 15 shows translation performance under grammar G_small for different values of entropy pruning threshold θ. Performance is reported after first-pass decoding with (Step 1, Section 5), and after rescoring with M₁ (Step 3, Section 5) the first-pass lattices pruned at alternative β beams. The first column reports the baseline for either Kneser-Ney and Katz language models, which are found by translation without entropy pruning, that is, performed with M₁. Both yield 34.5 on test-nw.

The first and main conclusion from this figure is that the two-pass strategy is adequate because we are always able to recover the baseline performance. As expected, the harsher the entropy-pruning of M₁ (as we lower θ) the greater β must be to recover from the significant degradation in first-pass decoding. But even at a harsh θ = 7.5 ×10⁻⁷, when first-pass performance drops over 7 BLEU points, a relatively-low value of β = 15 can recover the baseline performance.

Although this is true independently of the LM smoothing approach, a second conclusion from the figure is that the choice of LM smoothing does impact first-pass translation performance. For entropy pruning at θ = 7.5 × 10⁻⁷, the Katz LMs perform better for smaller beamwidths β. These results are consistent with the test set perplexities of the entropy pruned LMs (Table 2), and are also in line with other studies of Kneser-Ney smoothing and entropy pruning (Chelba et al. 2010; Roark, Allauzen, and Riley 2013).

Modeling errors are reported in Table 5 at the entropy pruning threshold θ = 7.5 × 10⁻⁷. As expected, modeling errors decrease as the beamwidth β increases, although we find that the language model with Katz smoothing has fewer modeling errors. However, modeling errors do not necessarily impact corpus level BLEU scores. For wide beamwidths (e.g., β = 15 here), there are still some modeling errors, but these are either few enough or subtle enough that two-pass decoding under either smoothing method yields the same corpus level BLEU score as the exact decoding baseline.

• What are the speed and quality tradeoffs for HiPDT as a function of first-pass LM size and translation grammar complexity?

• How do these compare against the predicted computational complexity?

In this section we turn back to the original large grammar, for which HiFST cannot perform exact decoding (see Table 3). In contrast, HiPDT is able to do exact decoding so we study tradeoffs in speed and translation performance. The speed of two-pass decoding can be increased by decreasing β and/or increasing θ, but at the risk of degradation in translation performance. For grammar G and language model M₁ we plot in Figure 16 the BLEU score against speed as a function of β for a selection of θ values. BLEU score is measured over the entire test set test-nw but speed is calculated only on sentences of length up to 20 words (∼500 sentences). In computing speed we measure not only the PDA operations, but the entire HiPDT decoding process described in Figure 14, including CYK parsing and the application of M₁. We note in passing that these unusually slow decoding speeds are a consequence of the large grammars, language models, and broad pruning thresholds chosen for these experiments; in practice, translation with either HiPDT or HiFST is much faster.

In these experiments we find that the language model entropy pruning threshold θ and the likelihood beamwidth β work together to balance speed against translation quality. For every entropy pruning threshold θ value considered, there is a value of β for which there is no degradation in translation quality. For example, suppose we want to attain a translation quality of 34.5 BLEU: then β should be set to 12 or greater. If the goal is to find the fastest system at this level, then we choose θ = 7.5 × 10⁻⁵.

The interaction between pruning in expansion and pruning of the language model is explained by Figure 17, where decoding and rescoring times are shown for various values of θ and β that achieve at least the translation quality target of 34.5. As θ increases, decoding time decreases because a smaller language model is easier to apply; however, rescoring times increase, because the larger values of β lead to larger WFSAs after expansion, and these are costly to rescore. The balance occurs at θ = 7.5 × 10⁻⁵ and a translation rate of 3.0 words/sec. In this case, entropy pruning yields a severely shrunken bigram language model, but this may vary depending on the translation grammar and the original, unpruned LM.

• Does the HiPDT two-pass decoding generate lattices that can be useful in rescoring?

We now report on rescoring experiments using WFSAs produced by the two-pass HiPDT translation system under the large translation grammar G. We demonstrate that HiPDT can be used to generate large, compact representations of the translation space that are suitable for rescoring with large language models or by alternative decoding procedures. We investigate translation performance by applying versions of the language model M₂ estimated with stupid backoff. We also investigate minimum Bayes risk (MBR) decoding (Kumar and Byrne 2004) as an alternative search strategy. We are particularly interested in lattice MBR (LMBR) (Tromble et al. 2008), which is well suited for the large WFSAs that the system can generate; we use the implementation described by Blackwood, de Gispert, & Byrne 2010. There are two parameters to be tuned: a scaling parameter to normalize the evidence scores and a word penalty applied to the hypotheses space; these are tuned jointly on the tune-nw set. Results are reported in Figure 18.

We note first that rescoring with the large language model M₂, which is effectively interpolated with M₁, gives consistent gains over initial results obtained with M₁ alone. After 5-gram rescoring there is already +0.5 BLEU improvement compared with G_small. With a richer translation grammar we have generated a richer lattice that allows gains to be gotten by our lattice rescoring techniques.

We also find that BLEU scores degrade smoothly as β decreases and the expansion pruning beamwidth narrows, and at all values of β LMBR gives improvement over the MAP hypotheses. Because LMBR relies on posterior distributions over n-grams, we conclude that HiPDT is able to generate compact representations of large search spaces with posteriors that are robust to pruning conditions.

Finally, we find that increasing θ degrades performance quite smoothly for β ≥ 9. Again, with appropriate choices of θ and β we can easily reach a compromise between decoding speed and final performance of our HiPDT system. For instance, with θ = 7.5 × 10⁻⁷ and β = 12, for which we decode at a rate of 3 words/sec as seen in Figure 16, we are losing only 0.5 BLEU after LMBR compared to θ = 7.5 × 10⁻⁷ and β = 15.

There is extensive prior work on computational efficiency and algorithmic complexity in hierarchical phrase-based translation. The challenge is to find algorithms that can be made to work with large translation grammars and large language models.

Following the original algorithms and analysis of Chiang 2007, Huang and Chiang 2007 developed the cube-growing algorithm, and more recently Huang and Mi 2010 developed an incremental decoding approach that exploits the left-to-right nature of n-gram language models.

Search errors in hierarchical translation, and in translation more generally, have not been as extensively studied; this is undoubtedly due to the difficulties inherent in finding exact translations for use in comparison. Using a relatively simple phrase-based translation grammar, Iglesias et al. 2009b compared search via cube-pruning to an exact FST implementation (Kumar, Deng, and Byrne 2006) and found that cube-pruning suffered significant search errors. For Hiero translation, an extensive comparison of search errors between the cube pruning and FSA implementation was presented by Iglesias et al. 2009a and de Gispert et al. 2010. The effect of search errors has also been studied in phrase-based translation by Zens and Ney 2008. Relaxation techniques have also recently been shown to find exact solutions in parsing (Koo et al. 2010), phrase-based SMT (Chang and Collins 2011), and in tree-to-string translation under trigram language models (Rush and Collins 2011); this prior work involved much smaller grammars and languages models than have been considered here.

Efficiency in synchronous parsing with Hiero grammars and hypergraphs has been studied previously by Dyer 2010b, who showed that a single synchronous parsing algorithm (Wu 1997) can be significantly improved upon in practice through hypergraph compositions. We developed similar procedures for our HiFST decoder (Iglesias et al. 2009a; de Gispert et al. 2010) via a different route, after noting that with the space of translations represented as WFSAs, alignment can be performed using operations over WFSTs (Kumar and Byrne 2005).

Although entropy-pruned language models have been used to produce real-time translation systems (Prasad et al. 2007), we believe our use of entropy-pruned language models in two-pass translation to be novel. This is an approach that is widely used in automatic speech recognition (Ljolje, Pereira, and Riley 1999) and we note that it relies on efficient representation of very large search spaces for subsequent rescoring, as is possible with FSAs and PDAs.

In this article, we have described a novel approach to hierarchical machine translation using pushdown automata. We have presented fundamental PDA algorithms including composition, shortest-path, (pruned) expansion, and replacement and have shown how these can be used in PDA-based machine translation decoding and how this relates to and compares with hypergraph and FSA-based decoding.

On the basis of the experimental results presented in the previous sections, we can now address the questions laid out in Sections 4 and 5:

• A two-pass translation decoding procedure in which translation is first performed with a weak entropy-pruned language model and followed by admissible likelihood-based pruning and rescoring with a full language model can yield good quality translations. Translation performance does not degrade significantly unless the first-pass language model is very heavily pruned.

• As predicted by the analysis of algorithmic complexity, intersection and expansion algorithms based on the PDA representation are able to perform exact decoding with large translation and weak language models. By contrast, RTN to FSA expansion fails with large translation grammars, regardless of the size of the language model. With large translation grammars, language model composition prior to expansion may be more attractive than expansion prior to language model composition.

• Our experimental results suggest that for a translation grammar and a language model of a particular size, and given a value of language model entropy pruning threshold θ, there is a value of the pruned expansion parameter β for which there is no degradation in translation quality with HiPDT. This makes exact decoding under large translation grammars possible. The values of θ and β will be grammar- and task-dependent.

• Although there is some interaction between parameter tuning, pruning thresholds, and language modeling strategies, the variation is not significant enough to indicate that a particular language model or smoothing technique is best. This is particularly true if minimum Bayes risk decoding is applied to the output translation lattices.

Several questions naturally arise about the decoding strategies presented here. One is whether inadmissible pruning methods can be applied to the PDA-based systems that are analogous to those used in current hypergraph-based systems such as cube-pruning (Chiang 2007). Another is whether a hybrid PDA–FSA system, where some parts of the PDA are pre-expanded and some not, could provide benefits over full pre-expansion (FSA) or none (PDA). We leave these questions for future work.

A state in T is a pair (q₁, q₂) where q₁ is a state of T₁ and q₂ a state of T₂. Given a transition e₁ = (q₁, a, b, w₁, q′₁) in T₁, transitions out of (q₁, q₂) in T are obtained using the following rules. If b ∈ Δ, then e₁ can be matched with a transition (q₂, b, c, w₂, q₂′) in T₂ resulting in a transition ((q₁, q₂), a, c, w₁ + w₂, (q₁′, q₂′)) in T. If b = ε, then e₁ is matched with staying in q₂ resulting in a transition ((q₁, q₂), a, ε, w₁, (q₁′, q₂)). Finally, if , e₁ is also matched with staying in q₂, resulting in a transition ((q₁, q₂), a, a, w₁, (q₁′, q₂)) in T. The initial state is (I₁,I₂) and a state (q₁, q₂) in T is final when both q₁ and q₂ are both final. Weight values are assigned as ρ((q₁, q₂)) = ρ₁(q₁) + ρ₂(q₂).

Let d^R and B^R be the data structures computed by the shortest-distance algorithm applied to T^R. For a state q in T′ (or equivalently T′_β), let d[q] denote the shortest distance from the initial state to q, denote the shortest distance from q to the final state, and s[q] denote the destination state of the last unbalanced open-parenthesis transition on a shortest path from the initial state to q.

The pseudo code of the algorithm is given in Figure 19. First, the shortest distance algorithm is applied to T^R and the absolute pruning threshold is computed accordingly (lines 1–2). The resulting balanced data information is then reversed (line 3). The initial and final states are created (lines 4–5) and the d, , and D data structures are initialized accordingly (lines 6–8). The default value in these data structures is assumed to be ∞. The queue is initialized containing the initial state (line 9).

The state (q, z) at the head of the queue is dequeued (lines 10–12). If (q, z) admits an incoming open-parenthesis transition, B contains the balance information for that state and D can be updated accordingly (lines 13–18).

If e is a regular transition, the resulting transition ((q, z), i[e], o[e], w[e], (n[e],z)) in T′ can be pruned using the criterion derived from Equation (B.11). If it is retained, the transition is created as well as its destination state (n[e],z) if needed (lines 20–22).

If e is an open-parenthesis transition, each balanced path starting by the resulting transition in T′ and ending by a close-parenthesis transition is treated as a meta-transition and pruned using the same criterion as regular transitions (lines 23–29). If any of these meta-transitions is retained, the transition ((q, z), ε, ε, w[e], (n[e], zi[e])) resulting from e is created as well as its destination state (n[e], zi[e]) if needed (lines 30–35).

If e is a closed-parenthesis transition, it is created if it belongs to a balanced path below the threshold (lines 36–39).

Finally, the resulting transducer T′_β is returned (line 40).

The research leading to these results has received funding from the European Union Seventh Framework Programme (FP7-ICT-2009-4) under grant agreement number 247762, and was supported in part by the GALE program of the Defense Advanced Research Projects Agency, contract no. HR0011-06-C-0022, and a May 2010 Google Faculty Research Award.