Amnesic Probing: Behavioral Explanation with Amnesic Counterfactuals

What drives a model to perform a specific prediction? What information is being used for prediction, and what would have happen if that information went missing? Because neural representation is opaque and hard to interpret, answering these questions is challenging.

The recent advancements in Language Models (LMs) and their success in transfer learning of many NLP tasks (e.g., Peters et al., 2018; Devlinet al., 2019; Liu et al., 2019b) spiked interest in understanding how these models work and what is being encoded in them. One prominent methodology that attempts to shed light on those questions is probing (Conneau et al., 2018) (also known as auxilliary prediction [Adi et al., 2016] and diagnostic classification [Hupkes et al., 2018]). Under this methodology, one trains a simple model —a probe—to predict some desired information from the latent representations of the pre-trained model. High prediction performance is interpreted as evidence for the information being encoded in the representation. A key drawback of such an approach is that while it may indicate that the information can be extracted from the representation, it provides no evidence for or against the actual use of this information by the model. Indeed, Hewitt and Liang (2019) have shown that under certain conditions, above-random probing accuracy can be achieved even when the information that one probes for is linguistically meaningless noise, which is unlikely to have any use by the actual model. More recently, Ravichanderet al. (2020) showed that models encode linguistic properties, even when not required at all for solving the task, questioning the usefulness and common interpretation of probing. These results call for higher scrutiny of causal claims based on probing results.

In this paper, we propose a counterfactual approach that serves as a step towards causal attribution: Amnesic Probing (see Figure 1 for a schematic view). We build on the intuition that if a property Z (e.g., part-of-speech) is being used for a task T (e.g., language modeling), then the removal of Z should negatively influence the ability of the model to solve the task. Conversely, when the removal of Z has little or no influence on the ability to solve T, one can argue that knowing Z is not a significant contributing factor in the strategy the model employs in solving T.

As opposed to previous work that focused on intervention in the input space (Goyal et al., 2019; Kaushik et al., 2020; Vig et al., 2020) or in specific neurons (Vig et al., 2020), our intervention is done on the representation layers. This makes it easier than changing the input (which is non-trivial) and more efficient than querying hundred of neurons (which become combinatorial when considering the effect of multiple neurons simultaneously).

We demonstrate that amnesic probing can function as a debugging and analysis tool for neural models. Specifically, by using amnesic probing we show how to deduce whether a property is used by a given model in prediction.

In order to build the counterfactual representations, we need a function that operates on a pre-trained representation and returns a counterfactual version which no longer encodes the property we focus on. We use the recently proposed algorithm for neutralizing linear information: Iterative Null-space Projection (INLP) (Ravfogel et al., 2020). This approach allows us to ask the counterfactual question: “How will the prediction of a task differ without access to some property?” (Pearl andMackenzie, 2018). This approach relies on the assumption that the usefulness of some information can be measured by neutralizing it from the representation, and witnessing the resulting behavioral change. It echoes the basic idea of ablation tests where one removes some component and measures the influence of that intervention.

We study several linguistic properties such as part-of-speech (POS) and dependency labels. Overall, we find that as opposed to the common belief, high probing performance does not mean that the probed information is used for predicting the main task (§4). This is consistent with the recent findings of Ravichander et al. (2020). Our analysis also reveals that the properties we examine are often being used differently in the masked setting (which is mostly used in LM training) and in the non-masked setting (which is commonly used for probing or fine-tuning) (§5). We then dive deeper into a more fine-grained analysis, and show that not all of the linguistic property labels equally influence prediction (§6). Finally, we re-evaluate previous claims about the way that BERT process the traditional NLP pipeline (Tenney et al., 2019a) with amnesic probing and provide a novel interpretation on the utility of different layers (§7).

Given a set of labeled data of data points X =x₁, …, x_n² and task labels Y = y₁,…,y_n we analyze a model f that predicts the labels Y from X: $yi^=f(xi)$⁠. We assume that this model is composed of two parts: an encoder h that transforms input x_i into a representation vector $hxi$ and a classifier c that is used for predicting $yi^$ based on $hxi$⁠: $yi^=c(h(xi))$⁠. We refer by model to the component that follows the encoding function h and is used for the classification of the task of interest y. Each data point x_i is also associated with a property of interest z_i which represents additional information, which may or may not affect the decision of the classifier c.

In this work, we are interested in the change in prediction of the classifier c on the prediction $yi^$ which is caused due to the removal of the property Z from the representation h(x_i), that is h(x_i)^¬Z.

Under the counterfactual approach, we aim to evaluate the behavioral influence of a specific type of information Z (e.g., POS) on some task(e.g., language modeling). To do so, we selectively remove this information from the representation and observe the change in the behavior of the model on the main task.

One commonly used method for information removal relies on adversarial training through the gradient reversal layer technique (Ganin et al., 2016). However, this techniques requires changing the original encoding by retraining the model, which is not desired in our case as we wish to study the original model’s behavior. Additionally, Elazar and Goldberg (2018) found that this technique does not completely remove all the information from the learned representation.

Instead, we make use of a recently proposed algorithm called Iterative Nullspace Projection (INLP) (Ravfogel et al., 2020). Given a labeled dataset of representations H, and a property to remove, Z, INLP neutralizes the ability to linearly predict Z from H. It does so by training a sequence of linear classifiers (probes) c₁,…,c_k that predict Z, interpreting each one as conveying information on a unique direction in the latent space that corresponds to Z, and iteratively removing each of these directions. Concretely, we assume that the ith probe c_i is parameterized by a matrix W_i. In the ith iteration, c_i is trained to predict Z from H³ , and the data is projected onto its nullspace using a projection matrix $PN(Wi)$⁠. This operation guarantees $WiPN(Wi)H=0$⁠, i.e., it neutralizes the features in the latent space which were found by W_i to be indicative to Z. By repeating this process until no classifier achieves above-majority accuracy, INLP removes all such features.⁴

Note that amnesic probing extends conventional probing, as it is only relevant in cases where the property of interest can be predicted from the representation. If a probe gets random accuracy, the information cannot be used by the model to begin with. As such, amnesic probing can be seen as a complementary method, which inspects probe accuracy as a first step, but then proceeds to derive behavioral outcomes from the directions associated with the probe, with respect to a specific task.

The usage of INLP in this setup involves some subtleties we aim to account for: (1) Any modification to the representation, regardless of whether it removes information necessary to the task, may cause a decrease in performance. Can the drop in performance be attributed solely to the modification of the representation? (2) The removal of any property using INLP may also cause removal of correlating properties. Does the removed information only pertain to the property in question?

In order to control for the information loss of the representations, we make use of a baseline that removes the same number of directions as INLP does, but randomly.

For every INLP iteration the data matrix’s rank decreases by the number of labels of the inspected property. This operation removes information from the representation which might be used for prediction. Using this control, Rand, instead of finding the directions using a classifier that learned some task, we generate random vectors from a uniform distribution, that accounts for random directions. Then, we construct the projection matrix as in INLP, by finding the intersection of nullspaces.

If the Rand impact on performance is lower than the impact of amnesic probing for some property, we conclude that we removed important directions for the main task. Otherwise, when the Rand control has a similar or higher impact, we conclude that there is no evidence for property usage for the main task.

The result of the amnesic probing is taken as an indication to whether or not the model we query makes use of the inspected property for prediction. However, the removed features might solely correlate with the property (e.g., word position in the sentence has a nonzero correlation to syntactic function). To what extent is the information removal process we employ selective to the property in focus?

We test that by explicitly providing the gold information that has been removed from the representation, and finetuning the subsequent layers (while the rest of the network is frozen). Restoring the original performance is taken as evidence that the property we aimed to remove is enough to account for the damage sustained by the amnesic intervention (it may still be the case that the intervention removes unrelated properties; but given the explicitly-provided property information, the model can make up for the damage). However, if the original performance is not restored, this indicates that the intervention removed more information than intended, and this cannot be accounted for by merely explicitly providing the value of the single property we focused on.

Concretely, we concatenate feature vectors of the studied property to the amnesic representations. Those vectors are 32-dimensional, and are initialized randomly, with a unique vector for each value of the property of interest. Those are fine-tuned until convergence. We note that as the new representation vectors are of a higher dimension than the original ones, we cannot use the original matrix. For an easier learning process, we use the original embedding matrix and concatenate it with a new embedding matrix, randomly initialized, and treat it as the new decision function.

We use our proposed method to investigate BERT (Devlin et al., 2019),⁶ a popular and competitive masked language model (MLM) that has recently been the subject of many analysis works (e.g., Hewitt and Manning, 2019; Liu et al., 2019a; Tenney et al., 2019a). While most probing works focus on the ability to decode a certain linguistic property of the input text from the representation, we aim to understand which information is being used by it when predicting words from context. For example, we seek to answer questions such as the following: “Is POS information used by the model in word prediction?” The following experiments focus on language modeling, as a basic and popular task, but our method is more widely applicable.

We focus on six tasks of sequence tagging: coarse and fine-grained part-of-speech tags (c-pos and f-pos, respectively); syntactic dependency labels (dep); named-entity labels (ner); and syntactic constituency boundaries⁷ that mark the beginning and the end of a phrase (phrase start, and phrase end, respectively).

We use the training and dev data of the following datasets for each task: English UD Treebank (McDonald et al., 2013) for c-pos, f-pos, and dep; and English OntoNotes (Weischedel et al., 2013) for ner, phrase start, and phrase end. For training, we use 100,000 random tokens from those datasets.

We report the following metrics:

LM accuracy: Word prediction accuracy.

Kullback-Leibler Divergence (D_KL): We calculate the D_KL between the distribution of the model over tokens, before and after the amnesic intervention. This measure focuses on the entire distribution, rather than the correct token only. Larger values implies a more significant change.

By using the probing technique, different linguistic phenomenon such as POS, dependency information, and NER (Tenney et al., 2019a; Liu et al., 2019a; Alt et al., 2020) have been found to be “easily extractable” (typically using linear probes). A naive interpretation of these results may conclude that because information can be easily extracted by the probing model, this information is being used for the predictions. We show that this is not the case. Some properties such as syntactic structure and POS are very informative and are being used in practice to predict words. However, we also find some properties, such as phrase markers, which the model does not make use of when predicting tokens, in contrast to what one can naively deduce from probing results. This finding is in line with a recent work that observed the same behavior (Ravichander et al., 2020).

For each linguistic property, we report the probing accuracy using a linear model, as well as the word prediction accuracy after removing information about that property. The results are summarized in Table 1.⁸ Probing achieves substantially higher performance over majority across all tasks. Moreover, after neutralizing the studied property from the representation, the performance on that task drops to majority (not presented in the table for brevity). Next, we compare the LM performance before and after the projection and observe a major drop for dep and f-pos information (decrease of 87.0 and 81.8 accuracy points, respectively), and a moderate drop for c-pos and ner information (decrease of 32.2 and 10.8 accuracy points, respectively). For these tasks, Rand performance on LM-Acc is lower than the original scores, but substantially higher than the Amnesic scores. Recall that the Rand experiment is done with respect to the amnesic probing, thus the number of removed dimension is the same, but each task may differ in the amount of dimensions removed. Furthermore, the D_KL metric shows the same trend (but in reverse, as a lower value indicates on a smaller change). We also report the selectivity results, where in most experiments the LM performance is restored, indicating amnesic probing works as expected. Note that the dep performance is not fully restored, thus some non-related features must have been coupled and removed with the dependency features. We believe that this happens in part due to the large number of removed directions.⁹ These results suggests that to a large degree, the damage to LM performance is to be attributed to the specific information we remove, and not to rank-reduction alone. We conclude that dependency information, POS and NER are important for word prediction.

Interestingly, for phrase start and phrase end we observe a small improvement in accuracy of 0.21 and 0.32 points, respectively. The performance for the control on these properties is lower, therefore not only are these properties not important for the LM prediction at this part of the model, they slightly harm it. The last observation is rather surprising as phrase boundaries are coupled to the structure of sentences, and the words that form them. A potential explanation for this phenomenon is that this information is simply not being used at this part of the model, and is rather being processed in an earlier stage. We further inspect this hypothesis in Section 7. Finally, the probe accuracy does not correlate with task importance as measured by our method (Spearman correlation of 8.5, with a p-value of 0.871).

These results strengthen recent works that question the usefulness of probing as an analysis tool (Hewitt and Liang, 2019; Ravichander et al., 2020), but measure it from the usefulness of properties on the main task. We conclude that high probing performance does not entail this information is being used at a later part of the network.

Probing studies tend to focus on representations that are used for an end-task (usually the last hidden layer before the classification layer). In the case of MLM models, the words are not masked when encoding them for downstream tasks.

However, these representations are different from those used during the pre-training LM phase (of interest to us), where the input words are masked. It is therefore unclear if the conclusions drawn from conventional probing also apply to the way that the pre-trained model operates.

From this section on, unless mentioned otherwise, we report our experiments on the masked words. That is, given a sequence of tokens x₁,…,x_i,…,x_n we encode the representation of each token x_i using its context, as follows: x₁, …, x_i−1, [MASK], x_i+1, …, x_n. The rest of the tokens remain intact. We feed these input tokens to BERT, and only use the masked representation of each word in its context h(x₁, …, x_i−1, [MASK], x_i+1, …, x_n)_i.

We repeat the experiments from Section 4 and report the results in Table 2. As expected, the LM accuracy drops significantly, as the model does not have access to the original word, and it has to infer it only based on context. Overall, the trends in the masked setting are similar to the non-masked setting. However, this is not always the case, as we show in Section 7. We also report the selectivity control. Notice that the performance for this experiment was improved across all tasks. In the case of dep and f-pos, where we had to neutralize most of the dimensions the performance does not fully recover. Note that the number of classes in those experiments might be a factor in the large performance gaps (expressed by the number of removed dimensions, N. dir, in the table). While not part of this study, it would be interesting to control for this factor in future work. However for the rest of the properties (c-pos, ner, and the phrase-markers) the performance is fully recovered, showing our methods’ selectivity.

To further study the effect of INLP and inspect how the different dimensions removal affect performance, we display in Figure 2 the LM performance after each iteration, both with the amnesic probing and the control, and observe a consistent gap between them. Moreover, we highlight the difference in the slope for our method and the random direction removal. The amnesic probing exemplifies a much steeper slope than the random direction, indicating that the studied properties are indeed correlated with words prediction. We also provide the main task performance after each iteration in Figure 5 in the Appendix, which steadily decreases with each iteration.

In the previous sections we observed the impact (or lack thereof) of different properties on word prediction. But when a property affects words prediction, are all words affected similarly? In this section, we inspect a more fine-grained version of the properties of interest, and study the impact of those on word predictions.

When we remove the POS information from the representation, are nouns affected to the same degree as conjunctions? We repeat the masked experimental setting from Section 5, but this time we inspect the word prediction performance for the different labels. We report the results for the c-pos tagging in Table 3. We observe large differences in the word prediction performance before and after the POS removal between the labels. Nouns, numbers, and verbs show a relatively small impact in performance (8.64, 6.91, and 11.73 respectively), while conjunctions, particles and determiners demonstrate large performance drops (73.73, 77.66, and 65.65, respectively). We see that the information about POS labels at the word-level prediction is much more important in closed-set vocabularies (such as conjunctions and determiners) than with open vocabularies (such as nouns and verbs).

A manual inspection of predicted words after removing the POS information reveals that many of the changes are due to the transformation of function words to content words. For example, the words ‘and’, ‘of’, and ‘a’ become ‘rotate’, ‘say’, and ‘final’, respectively, in the inspected sentences. For a more quantitative analysis, we use a POS tagger in order to measure the POS label confusion before and after the intervention. Out of the 12,700 determiners conjuncions and punctuations, 200 of the predicted words by BERT were tagged as nouns and verbs before the intervention, compared to 3,982 after.

Following the observation that classes are affected differently when predicting words, we further investigate the differences of specific label removal. To this end, we repeat the amnesic probing experiments, but instead of removing the fine-grained information of a linguistic property, we make a cruder removal: The distinction between a specific label and the rest. For example, with POS as the general property, we now investigate whether the information of noun vs. the rest is important for predicting a word. We perform this experiment for all of the pos-c labels, and report the results in Table 4.¹⁰

We observe big performance gaps when removing different labels. For example, removing the distinctions between nouns and the rest, or verbs and the rest has minimal impact on performance. On the other hand, determiners and punctuations are highly affected. This is consistent with the previous observation on removing specific information. These results call for more detailed observations and experiments when studying a phenomenon as the fine-grained property distinction does not behave the same across labels.¹¹

The results up to this section treat all of BERT’s ‘Transformer blocks’ (Vaswani et al., 2017) as the encoding function and the embedding matrixas the model. But what happens when we remove the information of some linguistic property from earlier layers?

By using INLP to remove a property from an intermediate layer, we prevent the subsequent layer from using linearly present information originally stored in that layer. Though this operation does not erase all the information correlative with the studied property (as INLP only removes linearly present information), it makes it harder for the model to use this information. Concretely, we begin by extracting the representation of some text from the first k layers of BERT and then run INLP on these representations to remove the property of interest. Given that we wish to study the effect of a property on layer i, we project the representation using the corresponding projection matrix P_i that was learned on those representation, and then continue the encoding of the following layers.¹²

Is the property we linearly remove from a given layer recoverable by subsequent layers? We remove the information about some linguistic property from layer i, and learn a probe classifier on all subsequent layers i + 1, …, n. This tests how much information about this property the following layers have recovered. We experiment with the properties that could be removed without reducing too many dimensions: pos-c, ner, phrase start, and phrase end. These results are summarized in Figure 3, both for the non-masked version (upper row) and the masked version (lower row).

Notably, for the pos-c, non-masked version, the information is highly recoverable in subsequent layers when applying the amnesic operation on the first seven layers: the performance drops from the regular probing of that layer between 5.72 and 12.69 accuracy points. However, in the second part of the network, the drop is substantially larger: between 16.57 and 46.39 accuracy points. For the masked version, we witness an opposite trend: The pos-c information is much less recoverable in the lower parts of the network than the upper parts. In particular, the removal of pos-c from the secondlayer appears to affect the rest of the layers, which do not manage to recover a high score on this task, ranging from 32.7 to 42.1 accuracy.

For all of the non-masked experiments the upper layers seem to make it harder for the subsequent layers to extract the property. In the masked version however, there is no consistent trend. It is harder to extract properties after the lower parts for pos-c and ner. For phrase start the upper part makes it harder for further extraction and for phrase end both the lower and upper parts make it harder, as opposed to the middle layers. Further research is needed in order to understand the significance of those findings, and whether or not they are related to information usage across layers.

This lead us to the final experiment where we test for the main task performance after an amnesic operation at the intermediate layers.

In the previous set of experiments, we measured how much of the signal removed in layer i is recovered in subsequent layers. We now study how the removal of information in layer i affects the word prediction accuracy at the final layer, in order to get a complementary measure for layer importance with respect to a property. The results for the different properties are presented in Figure 4, where we plot the difference in word prediction performance between the control and the amnesic probing when removing a linguistic property from a certain layer.

These results provide a clear interpretation on the internal function of BERT’s layers. For the masked version (Figure 4), we observe that the pos-c properties are mostly important in layer 3 and its surrounding layers, as well as layer 12. However, this information is accurately extractable only towards the last layers. For ner, we observe that the main performance loss occurs at layer 4. For phrase-markers the middle layers are important: layers 5 and 7 for phrase start (although the absolute performance loss is not big) and layer 6 for phrase end contributes the most for the word prediction performance.

The story with the non-masked version is quite different (Figure 4). First, notice that the amnesic operation improves the LM performance for all properties, in some layers.¹³ Second, the drop in performance peak across all properties is different than the masked version experiments. Particularly, it seems that for pos-c, when the words are non-masked in the input, the most important layer for pos-c is 11 (and not layer 3, as in the masked version), while this information is easily extractable (by standard probing) across all layers (above 80% accuracy).

Interestingly, the conclusions we draw on layer-importance from amnesic probing partly differ from the ones in the “Pipeline processing” hypothesis (Tenney et al., 2019a), which aims to localize and attribute information processing of linguistic properties to parts of BERT (for the non-masked version).¹⁴ On one hand the ner experiment trends are similar: the last layers are much more important than earlier ones (in particular, layer 11 is the most affected in our case, with a decrease of 31.09 accuracy points. On the other hand, in contrast to their hypotheses, we find that POS information, pos-c (which was considered to be more important in the earlier layers) affects the word prediction performance much more in the upper layers (40.99 accuracy loss in the 11th layer). Finally, we note that our approach performs an ablation of these properties in the representation space, which reveals which layers are actually responsible for processing properties, as opposed to Tenney et al. (2019a), who focused on where this information is easily extractable.

We note the big differences in behavior when analyzing the masked vs. the non-masked version of BERT, and call for future work to make aclearer distinctions between the two. Finally, we stress that the different experiments should not be compared between one setting to the other, and thus the different y-scales in the figures. This is due to confounding variables (e.g., the number of removed dimensions from the representations), which we do not control for in this work.

With the established impressive performance of large pre-trained language models (Devlin et al., 2019; Liu et al., 2019b), based on the Transformer architecture (Vaswani et al., 2017), a large body of work started studying and gaining insight into how these models work and what do they encode.¹⁵ For a thorough summary of these advancements we refer the reader to a recent primer on the subject (Rogers et al., 2020).

A particularly popular and easy-to-use interpretation method is probing (Conneau et al., 2018). Despite its popularity, recent works have questioned the use of probing as an interpretation tool. Hewitt and Liang (2019) have emphasized the need to distinguish between decoding and learning the probing tasks. They introduced control tasks, a consistent but linguistically meaningless attribution of labels to tokens, and have shown that probes trained on the control tasks often perform well, due to the strong lexical information held in the representations and learned by the probe. This leads them to propose a selectivity measure that aims to choose probes which achieve high accuracy only on linguistically-meaningful tasks. Tamkin et al. (2020) claim that probing cannot serve as an explanation of downstream task success. They observe that the probing scores do not correlate with the transfer scores achieved by fine-tuning.

Finally, Ravichander et al. (2020) show that probing can achieve non-trivial results for linguistic properties that were not needed for the task the model was trained on. In this work, we observe a similar phenomenon, but from a different angle. We actively remove some property of interest from the queried representation, and measure the impact of the amnesic representation of the property on the main task.

Two recent works study the probing paradigm from an information-theory perspective. Pimentelet al. (2020) emphasize that under a mutual-information maximization objective, “better” probes are increasingly more accurate, regardless of complexity. They use the data-processing inequality to question the rationale behind methods that focus on encoding, and propose ease of extractability as an alternative criterion. Voita andTitov (2020) follow this direction, using the concept of minimum description length (MDL, Rissanen, 1978) to quantify the total information needed to transmit both the probing model and the labels it predicts. Our discussion here is somewhat orthogonal to those on the meaning of encoding and probe complexity, as we focus on the information influence on the model’s behavior, rather than on the ability to extract it from the representation.

Finally and concurrent to this work, Feder et al. (2020) have studied a similar question of a causal attribution of concepts to representations, using adversarial training guided by causal graphs.

Intuitively, we would like to completely neutralize the abstract property we are interested in—e.g., POS information (completeness), as represented by the model—while keeping the rest of the representation intact (selectivity). This is a nontrivial goal, as it is not clear whether neural models actually have abstract and disentangled representations of properties such as POS, which are independent of other properties of the text. It may be the case that the representation of many properties is intertwined. Indeed, there is an ongoing debate on the assertion that certain information is “encoded” in the representation (Voita and Titov, 2020; Pimentel et al., 2020). However, even if a disentangled representation of the information we focus on exists, it is not clear how to detect it.

We implement the information removal operation with INLP, which gives a first order approximation using linear classifiers; we note, however, that one can in principle use other approaches to achieve the same goal. While we show that we do remove the linear ability to predict the properties and provide some evidence to the selectivity of this method (§2), one has to bear in mind that we remove only linearly-present information, and that the classifiers can rely on arbitrary features that happen to correlate with the gold label, be it a result of spurious correlations or inherent encoding of the direct property. Indeed, we observe this behavior in Section 7.1 (Figure 3), where we neutralize the information from certain layers, but occasionally observe higher probing accuracy in following layers. We thus stress that the information we remove in practice should be seen only as an approximation for the abstract information we are interested in, and that one has to be cautious of causal interpretations of the results. Although in this paper we use the INLP algorithm in order to remove linear information, amnesic probing is not restricted to removing linear information. When non-linear removal methods become available, they can be swapped instead of INLP. This stresses the importance of creating algorithms for non-linear information removal.

Another unanswered question is how to quantify the relative importance of different properties encoded in the representation for the word prediction task. The different erasure portion for different properties makes it hard to draw conclusions on which property is more important for the task of interest. Although we do not make claims such as “dependency information is more important than POS”, these are interesting questions that should be further discussed and researched.

In this work, we propose a new method, Amnesic Probing, which aims to quantify the influence of specific properties on a model that is trained on a task of interest. We demonstrate that conventional probing falls short in answering such behavioral questions, and perform a series of experiments on different linguistic phenomenon, quantifying their influence on the masked language modeling task. Furthermore, we inspect both unmasked and masked BERT’s representation and detail the differences between them, which we find to be substantial. We also highlight the different influence of specific fine-grained properties (e.g., nouns and determiners) on the final task. Finally, we use our proposed method on the different layers of BERT, and study which parts of the model make use of the different properties. Taken together, we argue that compared with probing, counterfactual intervention—such as the one we present here—can provide a richer and more refined view of the way symbolic linguistic information is encoded and used by neural models with distributed representations.¹⁶

We would like to thank Hila Gonen, Amit Moryossef, Divyansh Kaushik, Abhilasha Ravichander, Uri Shalit, Felix Kreuk, Jurica Ševa, and Yonatan Belinkov for their helpful comments and discussions. We also thank the anonymous reviewers and the action editor, Radu Florian, for their valuable suggestions.

This project has received funding from the Europoean Research Council (ERC) under the Europoean Union’s Horizon 2020 research and innovation programme, grant agreement no. 802774 (iEXTRACT). Yanai Elazar is grateful to be partially supported by the PBC fellowship for outstanding PhD candidates in Data Science.

We provide additional experiments that depict the performance of the main task (e.g., POS) performance during the INLP iterations in Figure 5.

Control Task (Hewitt and Liang, 2019) has been suggested as a way to attribute the performance of the probe to extraction of encoded information, as opposed to lexical memorization. Our goal in this work, however, is not to extract information from the representation (as is done in conventional probing) but to measure a behavioural outcome. Since the control task is solved by lexical memorization, applying INLP on control task’s classifiers erases lexical information (i.e., erases the ability to distinguish between arbitrary words), which is at the core of the LM objective and which is highly correlated with many of the other linguistic properties, such as POS. We argue that even if we do see a significant drop in performance with the control task, this says little on the validity of the results of removal of the linguistic property (e.g., POS). However, for completeness, we provide the results in Figure 6. As can be seen from this figure, this control’s slope is smaller than the one of the amnesic probing, suggesting that those directions have less behavioral influence. However, the slopes are steeper than the ‘Rand’ experiment. This is due to the identity removal of groups of words, due to the label shuffle, as suggested in their setup. This is the reason we believe this test is not adequate in our case, and why we provide other tests to control for our method: Rand and Selectivity (§2.3).