Formal Language Recognition by Hard Attention Transformers: Perspectives from Circuit Complexity

The Transformer architecture for neural networks (Vaswani et al., 2017) has yielded remarkable advances in performance on a variety of benchmark tasks in natural language processing. These advances have spurred considerable interest in understanding the capabilities and limitations of the Transformer architecture. While Transformer networks are extremely complex when deployed at scale, theoretical studies such as those of Pérez et al. (2019), Yun et al. (2020), Hahn (2020), and Merrill et al. (2022) have uncovered meaningful insights about the expressive power of Transformers by formulating abstract models of the self-attention mechanism and analyzing their computational power.

In this work, we analyze three restricted models of self-attention based on their ability to recognize formal languages. All three models use hard attention—meaning that each attention head attends only to the position or positions with the highest attention score, with no attention paid to any of the other positions—but differ in how they behave in the case of ties in the maximum attention value. In the first two models we study, the attention mechanism returns the value at exactly one position (for example, the leftmost) in case several positions tie for the maximum attention value. The first such model, which we call generalized unique hard attention Transformers (GUHAT) and was defined by Hahn (2020), imposes no restrictions on the nature of activation values or the functions the network uses to compute them. The second model, unique hard attention Transformers (UHAT), was defined and studied by Yao et al. (2021) and is a more concrete version of GUHAT that incorporates restrictions on the nature of activation values and computations. In the third model, which we call averaging hard attention Transformers (AHAT), the attention mechanism returns the uniform average of the values at positions with the maximum attention value. This is the definition of hard attention used by Pérez et al. (2019), Yun et al. (2020), and Merrill et al. (2022).¹

Let Σ be a fixed finite alphabet of symbols, and let $ be a distinct end-of-sequence symbol not in Σ. The set of strings of symbols over Σ of length n is denoted by Σⁿ, and the set of all finite strings over Σ is denoted Σ^*. A (formal) language over Σ is any subset of Σ^*.

The set of integers between i and j inclusive is denoted [i..j]. Define the function ℓ : ℕ →ℕ as $ℓ(n)=⌈log2(n+1)⌉$ for all n, and define bin(i,n) to be the binary representation of i as a string of length ℓ(n), for every i ∈ [1..n]. For example, bin(6,30) = 00110. If P is a logical predicate, then {P} denotes the truth value of P; for example, {(x ≥ y) ∨ (z > 3)} = 1 when x ≥ y or z > 3, and is 0 otherwise.

Our analysis of UHAT and GUHAT is carried out within the framework of circuit complexity, in which the complexity of a computational system is measured by the size, depth, and types of gates of a Boolean circuit implementing that system. In this section we review the basic concepts, definitions, and results of circuit complexity used by our analysis. A detailed overview is provided in Chapter 6 of Arora and Barak (2009).

Boolean circuits are a formal model of computational systems based on logic gates. Roughly speaking, a Boolean circuit consists of binary-valued input and output layers, with feedforward connections² to one another via intermediate gates that implement logical operations. We use the following definition of Boolean circuits.

We refer to the edges of a Boolean circuit as wires. The size of a circuit is the number of wires it contains, and the depth of a circuit is the maximum length of a directed path of wires from an input vertex to an output. A Boolean circuit C computes a Boolean function from {0,1}ⁿ to {0,1}^m; we denote its output on input x = x₁x₂…x_n by C(x).

Observe that a Boolean circuit has a fixed number of input vertices, and therefore can only take as input bit strings of a fixed length. We would like to define circuit computation for a map defined on all of {0,1}^*. To that end, we allow different circuits for inputs of different lengths.

The class AC⁰ is defined by setting restrictions on the size and depth of circuits within a family of Boolean circuits.

We relate formal languages with families of circuits by identifying languages L ⊆ Σ^* with Boolean functions that classify strings as belonging to L or not. Formally speaking, let Σ be a finite alphabet. A binary symbol encoding of Σ is an injective map h from Σ to {0,1}^s, where s = ℓ(|Σ|). Thus h maps each symbol to a distinct binary string of length s. We extend h to a homomorphism on strings from Σ^*. We say that the circuit family {C_n} recognizes the language L over Σ if there is a binary symbol encoding h of Σ such that for every n and every x ∈ Σⁿ, C_sn(h(x)) = 1 if and only if x ∈ L. With this definition of language recognition via Boolean circuits, we say that a language is in AC⁰ if and only if it is recognized by a family of Boolean circuits in AC⁰.

Having defined the class AC⁰, we present some examples of languages not belonging to this class. First, the following three languages were shown by Furst et al. (1984) to fall outside AC⁰.

Additionally, we show later in this paper (Corollary 3) that DYCK-1 also falls outside AC⁰.

Finally, we define PALINDROMES, a language shown in Section 5 to be in GUHAT.

We now define the three kinds of hard attention Transformers studied in this paper: GUHAT, UHAT, and AHAT. These formalisms are models of computation inspired by the encoder portion of the Transformer architecture. They conceptualize Transformers as cascading layers of feature extractors that convert a sequence of embeddings into increasingly higher-level representations.

We begin by presenting a general framework that subsumes the three hard attention Transformer models. Formally, a generalized Transformer is a device that maps a string x ∈ Σ^*$ to 1 or 0, signifying that x is accepted or rejected, respectively. Each such device is parameterized by a collection of functions described as follows.

The formalism we have presented above is fully generalized in the sense that we have placed no restrictions on the activation values $A$ or the functions f, $fk,hatt$⁠, f^pool, $fkact$⁠, or g, other than to specify their domains and co-domains. The three hard attention Transformer models are derived by placing restrictions upon these elements.

The first restriction we consider is on the form of the pooling function. We consider two types of pooling functions: the unique hard attention function, used in GUHAT and UHAT, and the averaging hard attention function, used in AHAT.

In unique hard attention, the pooling function simply selects the activation value from the previous layer corresponding to the argmax of the row of attention scores. In case of a tie, the leftmost activation value is selected.

Averaging hard attention is similar to unique hard attention, except that in the case of a tie, the selected activation values are averaged.

The GUHAT model is defined as the class of generalized Transformers that use unique hard attention.

The GUHAT model mostly follows the definitions of Hahn (2020). It is slightly generalized in allowing the input function f to depend on the input length n, and in allowing the activation function $fkact$ to depend on the layer k, but these generalizations are immaterial. In particular, it is not necessary to assume that the input length n is provided to the input function: if the input function were f(σ,i) = (σ,i), the subsequent layer could direct attention at every position to position n (because it uniquely contains the end-of-sequence symbol $), at which point the value of n is available at every position.

GUHAT allows the activation values $A$ and the functions f, $fk,hatt$⁠, $fkact$⁠, and g to take on any arbitrary mathematical value. In practical applications of Transformer networks, however, these components are restricted in specific ways. Many variations of hard attention Transformers attempt to incorporate these restrictions into theoretical models, though they do not entirely agree on the details of these restrictions.

The UHAT and AHAT models adopt many of these restrictions, largely following the definitions of Yao et al. (2021). For the sake of computability, we require activation values to be vectors of rational numbers. Following Pérez et al. (2019), we restrict scalars to be rational as well. Next, we assume that the input function f is decomposed into a token embedding function and a position embedding function. Mirroring the more familiar description of attention functions in terms of query, key, and value matrices, we use a bilinear form for attention functions proposed in Luong et al. (2015). In addition to the unique and averaging hard attention mechanisms, we allow the pooling function to be future-masked (where for position i only those positions j with j ≤ i are considered in the attention computation) or past-masked (similarly for j ≥ i). Finally, we assume that activation functions and the model output function are computed by feedforward neural networks with ReLU activation. These restrictions are summarized below.

Because Σ is finite, we may assume that the token embedding function is given by a table lookup. Our formulation of position embedding is somewhat more general than the definition of Yao et al. (2021), who take the position embedding to be a scalar defined as p(i,n) = i/n that occupies one position of the initial activation vector.

The UHAT and AHAT models are defined to be restricted Transformers that satisfy the above conditions and use unique and averaging hard attention, respectively.

UHAT is clearly a subclass of GUHAT because the former imposes restrictions on the form of the input, attention, activation, and output functions. We suspect, but do not prove, that this inclusion is proper. Moreover, we briefly argue below that UHAT is properly contained in AHAT.

Hahn (2020) shows that the languages 1^* and {aⁿbⁿ∣n ≥ 1} are in GUHAT, and the languages PARITY and DYCK-k for all k ≥ 1 are not in GUHAT. Pérez et al. (2019) show that even without positional information, the language MAJORITY is in AHAT. Bhattamishra et al. (2020) show that SHUFFLE-k is in AHAT, which implies that DYCK-1 is in AHAT. Yao et al. (2021) show that the language DYCK-(k,D) is in UHAT. The latter two results use positional masking, but no other positional information.

Let us now illustrate how a GUHAT computes by way of example. In this section, we describe a GUHAT with 2 layers and 1 head that recognizes the language PALINDROMES over the alphabet Σ = {a,b,c}. Broadly speaking, this Transformer works as follows. The first layer is responsible for comparing each symbol of the input string with the corresponding symbol on the opposite side of the string, and marking whether the two symbols match. The second layer reads these markings, searching for a mismatch identified by the first layer. If one is found, the model output function returns 0; otherwise, it returns 1. For intuition, we simultaneously illustrate the Transformer’s computation on the input abcca$, which should be rejected.

The input function is defined as f(σ,i,n) = (σ,i,n) for each σ ∈ Σ and i ∈ [1..n]. For our example input, the initial (layer 0) activation values are shown in the first row of Figure 1. These activation values are not rational-valued vectors, of course, but the GUHAT model imposes no restriction on the form these values can take.

We define the attention function for layer 1, $f1,1att((σ,i,n),(σ′,j,n))$⁠, to be {(j = n − i) ∨ (i = j = n)}. For each position i < n, this selects the activation at the correct corresponding position, n − i. For position n, it selects the activation at position n.

The layer 1 activation values for our example input are shown in the second row of Figure 1. This indicates that positions 2 and 4 found mismatched symbols, and positions 1, 3, and 5 did not.

Layer 2 gathers the relevant information from layer 1 into the last position. The layer 2 attention function is defined by $f2,1att((r,i),(s,j))=s$⁠. This directs the attention at every position to the leftmost activation value (s,j) from layer 1 such that s = 1. In our example, the leftmost such position is 2, with its activation of (1,2). If the input sequence had instead been a valid palindrome, none of the positions i ∈ [1..n − 1] would have been marked with (1,i) by layer 1. In this case, the leftmost position with s = 1 would have been the final position n, which has the activation value of (1,n).

We define the layer 2 activation function as $f2act((r,i),(s,j))=(i,j)$⁠. For our example input, the activation values for layer 2 are shown in the third row of Figure 1. The activation value at position n will be (n,n) if and only if no earlier position found a symbol mismatch, so the model output function is simply g((i,j)) = {i = j}. With the input sequence abcca$, the activation for position 6 at layer 2 is (6,2) and the output value is 0. For a valid palindrome such as abcba$, the activation for position 6 at layer 2 is (6,6) and the output value is 1. Generalizing this construction to an arbitrary alphabet Σ, we have the following.

Despite the abstractness and generality of the GUHAT model, we can define a normal form representation and show that every Transformer T in GUHAT is equivalent to a Transformer in GUHAT in this normal form with the same number of layers and heads. The key idea is to preserve in the activation values all the information from previous layers that has been used to compute them, by requiring that the input and activation functions just return the tuple of their arguments. We also require that attention values be integers in the smallest relevant range.

To illustrate the construction of $T^$ in the proof of Lemma 1, we briefly show how an informative normal form version of the Transformer for PALINDROMES from Section 5 would process the input x = abcca$. Because the attention functions in that example return 0 or 1, their translation is simplified.

The layer 2 attention function applied to a pair of layer 1 activation values ((x_i,i,n),(x_j,j,n)) and ((x_k,k,n),(x_ℓ,ℓ,n)) first applies the translation function t₁ to these two activation values to recover the pairs (r,i) and (s,j), and then applies the example’s layer 2 attention function to these to yield the attention value s.

In this section we show that for every language L ∈GUHAT, we can construct a family of Boolean circuits of constant depth and polynomial size that also recognizes L. This will prove the following, which is our main result.

Let L be a language over Σ that is in GUHAT. By Lemma 1, we may assume that L is recognized by GUHAT Transformer T in informative normal form. Assume T has K layers and H heads.

What we describe below is a family of circuits to recognize the end-marked language L$, which can easily be converted to a family of circuits that recognizes L by hard-wiring the representation of the end-of-sequence symbol $ at the end of the input string using constant gates. Let s = ℓ(|Σ| + 1) and let h be any binary symbol encoding for Σ ∪{$}. We construct a family of Boolean circuits {C_n} of constant depth and polynomial size such that for all positive integers n and all x ∈ Σⁿ⁻¹, x ∈ L if and only if C_sn(h(x$)) = 1.

The key step of the proof is to bound the number of bits needed to represent attention and activation values for an input sequence of length n by $O(logn)$⁠, where the suppressed constants depend on K and H.

It is worth observing that the bounds provided by Lemma 2 do not hold in the case of AHAT. Attention scores may be the result of the average of an arbitrary subset of the possible inputs, which means that there are exponentially more possible activation values at each layer.

The following elementary facts about Boolean circuits will be useful.

With the $O(logn)$ bound on the number of bits to represent activation and attention values, Lemma 2 yields circuits of constant depth and size polynomial in n for the input, attention, activation, and output functions. Additional circuitry is necessary to implement the comparison of attention scores and selection of the activation value to attend to for each position, layer, and head.

We construct the overall circuit C_sn according to the layers of T, starting with the input function. Let the inputs to T be x_i for i ∈ [1..n]. The inputs to C_sn are x_i,j for i ∈ [1..n] and j ∈ [1..s], where x_i,j are the bits of h(x_i), representing the binary encoding of input symbol x_i. At layer 0 for position i, the value of $yi(0)=f(xi,i,n)=(xi,i,n)$ is achieved by having the input wires x_i,j for j ∈ [1..s] followed by a sequence of constants 0 or 1 representing bin(i,n) and bin(n,n) for a total of 2ℓ(n) + s wires representing the value (x_i,i,n).

Inducting on layers, we assume that for some k ∈ [1..K] the circuit C_sn has been constructed to contain the wires representing all the activation values $yi(k−1)$ for i ∈ [1..n] at layer k − 1. The portion of the circuit computing the representations of activation values at layer k is described as follows. Fix a position i ∈ [1..n] and a head h ∈ [1..H]. For each j ∈ [1..n], there is a circuit A_i,j,k,h that has as input the wires for the activation values $yi(k−1)$ and $yj(k−1)$ and as output, wires representing the nonnegative integer attention score a_i,j,k,h in binary. Each of these circuits A_i,j,k,h has 2(H +1)^k−1(2ℓ(n) + s) inputs and outputs by Lemma 2, and therefore can be computed using depth 3 and size polynomial in n, by Corollary 1. All Hn² such circuits for layer k operate in parallel, for overall depth 3 and size polynomial in n.

We next describe the circuit that implements the pooling function f^UHA. For each pair j,j′ ∈ [1..n], there is a circuit D_i,j,j′,k,h whose inputs are the outputs of A_i,j,k,h and A_i,j′,k,h and whose output is a single wire g_i,j,j′,k,h with a value of 1 if a_i,j,k,h ≥ a_i,j′,k,h and 0 otherwise. Because of the bounds on the number of inputs and outputs, each of these circuits can have depth 3 and size polynomial in n by Corollary 1. These n² circuits all compute in parallel.³ Then for each position j, whether j maximizes a_i,j,k,h can be computed by an AND gate whose inputs are g_i,j,j′,k,h for all j′ ∈ [1..n]. Let the output of this AND gate be denoted m_i,j,k,h. Then m_i,j,k,h = 1 if and only if the position j maximizes a_i,j,k,h. This increases the depth by 1.

For each j, an indicator z_i,j,k,h is computed by an AND gate whose inputs are m_i,j,k,h and NOT(m_i,j′,k,h) for all j′ < j. Thus, z_i,j,k,h = 1 if and only if j is the leftmost position that maximizes a_i,j,k,h. This increases the depth by 2.

Finally, these indicator values are used to combine the layer k − 1 activation values in a selection circuit, yielding the representation of the activation value $bi,k,h=yj(k−1)$ such that z_i,j,k,h = 1. In general, such a selection circuit takes as input t selector bits z₁,…,z_t, where exactly one z_j = 1, and t input values w₁,…,w_t, where each w_r consists of S bits. It outputs S bits representing the selected w_j (for which z_j = 1). Letting w_r,s denote bit s of w_r, the computation can be described as v_r,s = w_r,s ∧ z_r for r ∈ [1..t] and s ∈ [1..S], which can be computed by one layer of tS AND gates in parallel. Then the bits of the output are $us=∨r=1Svr,s$ for s ∈ [1..S], which can be computed by one layer of S OR gates in parallel. Thus, the selection circuit adds 2 to the depth, and a polynomial in n to the size.

Because each activation function for a GUHAT in informative normal form simply returns its arguments, no further computation is needed for the activation values. The representation of the activation value $yi(k)$ is just the sequence of wires representing $yi(k−1)$ followed by those representing b_i,k,1, through b_i,k,H.

To produce the output of the circuit, we note that the representation of $yn(L)$ has $O(logn)$ bits and the output of g is a single bit, so g can be implemented by a Boolean circuit of constant depth and size polynomial in n, by Corollary 1. This concludes the proof of Theorem 1.

Furst et al. (1984) prove that the PARITY, EQUALITY, and MAJORITY languages are not in AC⁰, which immediately implies the following.

To see that the DYCK-k languages are also not in AC⁰, we reduce from the EQUALITY language.

We have defined formal language recognition by the encoder portion of a Transformer network using generalized unique hard attention (GUHAT), unique hard attention (UHAT), and averaging hard attention (AHAT), and shown that languages in UHAT and GUHAT are recognizable by constant depth, polynomial size families of circuits, that is, families of circuits in the complexity class AC⁰. This strengthens the negative result of Hahn (2020) that the languages PARITY and DYCK-k are not in GUHAT, and provides a simpler and more general proof. Combined with prior results of Pérez et al. (2019) showing that the language MAJORITY is in AHAT, or Bhattamishra et al. (2020) showing that the language DYCK-1 is in AHAT, this shows that AHAT contains languages that are not in GUHAT or UHAT.

Many intriguing open questions remain. What classical closure properties hold for the classes of languages GUHAT, UHAT, and AHAT? Closure under complement just requires complementing the output function g, and closure under pairwise union and intersection should be straightforward using a parallel approach; but what about closure under homomorphism, inverse homomorphism, concatenation, or Kleene star? We briefly observe that GUHAT and UHAT cannot be closed under both Kleene star and concatentation lest they contain all regular languages, including PARITY.

Existing formal models and indeed practical implementations of Transformers vary in their representation of position information, whether as an absolute representation of position, a ratio (e.g., position i in a sequence of length n as i/n), through angle information (e.g., position i by the pair $(cosθi,sinθi)$ where θ_i = πi/2n), or as an arbitrary learned embedding. In the UHAT and AHAT models, the choice of positional encoding can facilitate positional comparison (e.g., an angle-based encoding allows for equality testing via dot products) or make it uncomputable (e.g., if positional encodings enumerate Turing machines that halt on their own encodings). It remains to be understood what effect such differences in position representation have on the expressive power of a model.

More generally, is it possible to prove that soft attention, which we have not addressed here, is strictly more powerful than even averaging hard attention? Yao et al. (2021, Theorem B.3) present a construction for a soft attention Transformer that recognizes DYCK-k. This construction crucially uses specialized encodings of position and layer normalization, whose formal power remains to be understood.

Finally, given the success that Transformers have had as models of natural language, it is perhaps surprising that these models’ expressive power seems to be best characterized (or at least bounded) in terms of circuit complexity. Mathematical explorations of natural language have most commonly used the approach to language complexity afforded by the Chomsky hierarchy and its refinements, which is based on automata and formal grammars. The apparent incomparability of these approaches suggests that the exploration of different types of Transformer models might offer a new approach to the study of the formal properties of natural language.

We thank the reviewers and the action editor for their work in reviewing this paper.