SpanBERT: Improving Pre-training by Representing and Predicting Spans

We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text. Our approach extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it. SpanBERT consistently outperforms BERT and our better-tuned baselines, with substantial gains on span selection tasks such as question answering and coreference resolution. In particular, with the same training data and model size as BERTlarge, our single model obtains 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 respectively. We also achieve a new state of the art on the OntoNotes coreference resolution task (79.6% F1), strong performance on the TACRED relation extraction benchmark, and even gains on GLUE.1


Introduction
Pre-training methods like BERT (Devlin et al., 2019) have shown strong performance gains using self-supervised training that masks individual words or subword units.However, many NLP tasks involve reasoning about relationships between two or more spans of text.For example, in extractive question answering (Rajpurkar et al., 2016), determining that the "Denver Broncos" is a type of "NFL team" is critical for answering the question "Which NFL team won Super Bowl 50?"Such spans provide a more challenging target for self supervision tasks, for example predicting "Denver Broncos" is much harder than predicting only "Denver" when you know the next word is "Broncos".In this paper, we introduce a span-level pretraining approach that consistently outperforms BERT, with the largest gains on span selection tasks such as question answering and coreference resolution.
We present SpanBERT, a pre-training method that is designed to better represent and predict spans of text.Our method differs from BERT in both the masking scheme and the training objectives.First, we mask random contiguous spans, rather than random individual tokens.Second, we introduce a novel span-boundary objective (SBO) so the model learns to predict the entire masked span from the observed tokens at its boundary.Span-based masking forces the model to predict entire spans solely using the context in which they appear.Furthermore, the span-boundary objective encourages the model to store this span-level information at the boundary tokens, which can be easily accessed during the fine-tuning stage.Figure 1 illustrates our approach.
To implement SpanBERT, we build on a welltuned replica of BERT, which itself substantially outperforms the original BERT.While building on our baseline, we find that pre-training on single segments, instead of two half-length segments with the next sentence prediction (NSP) objective, considerably improves performance on most downstream tasks.Therefore, we add our modifications on top of the tuned single-sequence BERT baseline.
Together, our pre-training process yields models that outperform all BERT baselines on a wide < l a t e x i t s h a 1 _ b a s e 6 4 = " u y 6 m g 7 G g W I 9 p g a W R d j o U T 0 / l m Q k = " > A A A C X X i c b V H P S 8 M w G E 2 r 0 2 1 O n X r w 4 C U 4 h I k w W h H 0 I o x 5 8 T B x o l N h L S P N 0 i 0 s b U r y V R y l / 6 Q 3 v f i v m P 0 4 6 N y D w O O 9 9 5 F 8 L 0 x j d T / + W N K c 1 l / A S T h P k R G c Y 8 5 J S A k f p V 8 C I C I 0 p E 1 s 7 r H r B 3 y E I p I S B C 5 K f 4 G v + y + 9 n c v 2 v f 5 S u y Z 6 u y j 6 3 7 F d l + t e Y 0 n B n w f + I u S A 0 t 0 O l X P 7 y B p G n E Y q C C a N 1 z n Q T 8 j C j g V L C 8 7 K W a J Y S O y Z D 1 D I 1 J x L S f z d r J 8 Y l R B j i U y p w Y 8 E z 9 P Z G R S O t J F J j k d A G 9 7 E 3 F V V 4 v h f D K z 3 i c p M B i O r 8 o T A U G i a d V 4 w F X j I K Y G E K o 4 u a t m I 6 I I h T M h 5 R N C e 7 y y v / J 8 3 n D d R r u w 0 W t 2 V r U U U R H 6 B j V k Y s u U R P d o g 7 q I o q + L G S V r L L 1 b R f s i r 0 z j 9 r W Y u Y A / Y F 9 + A M 8 t 7 d x < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " u y 6 m g 7 G g W I 9 p g a W R d j o U T 0 / l m Q k = " > A A A C X X i c b V H P S 8 M w G E 2 r 0 2 1 O n X r w 4 C U 4 h I k w W h H 0 I o x 5 8 T B x o l N h L S P N 0 i 0 s b U r y V R y l / 6 Q 3 v f i v m P 0 4 6 N y D w O O 9 9 5 F 8 L 0 x j d T / + W N K c 1 l / A S T h P k R G c Y 8 5 J S A k f p V 8 C I C I 0 p E 1 s 7 r H r B 3 y E I p I S B C 5 K f 4 G v + y + 9 n c v 2 v f 5 S u y Z 6 u y j 6 3 7 F d l + t e Y 0 n B n w f + I u S A 0 t 0 O l X P 7 y B p G n E Y q C C a N 1 z n Q T 8 j C j g V L C 8 7 K W a J Y S O y Z D 1 D I 1 J x L S f z d r J 8 Y l R B j i U y p w Y 8 E z 9 P Z G R S O t J F J j k d A G 9 7 E 3 F V V 4 v h f D K z 3 i c p M B i O r 8 o T A U G i a d V 4 w F X j I K Y G E K o 4 u a t m I 6 I I h T M h 5 R N C e 7 y y v / J 8 3 n D d R r u w 0 W t 2 V r U U U R H 6 B j V k Y s u U R P d o g 7 q I o q + L G S V r L L 1 b R f s i r 0 z j 9 r W Y u Y A / Y F 9 + A M 8 t 7 d x < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " u y 6 m g 7 G g W I 9 p g a W R d j o U T 0 / l m Q k = " > A A A C X X i c b V H P S 8 M w G E 2 r 0 2 1 O n X r w 4 C U 4 h I k w W h H 0 I o x 5 8 T B x o l N h L S P N 0 i 0 s b U r y V R y l / 6 Q 3 v f i v m P 0 4 6 N y D w O O 9 9 5 F 8 L 0 x j d T / + W N K c 1 l / A S T h P k R G c Y 8 5 J S A k f p V 8 C I C I 0 p E 1 s 7 r H r B 3 y E I p I S B C 5 K f 4 G v + y + 9 n c v 2 v f 5 S u y Z 6 u y j 6 3 7 F d l + t e Y 0 n B n w f + I u S A 0 t 0 O l X P 7 y B p G n E Y q C C a N 1 z n Q T 8 j C j g V L C 8 7 K W a J Y S O y Z D 1 D I 1 J x L S f z d r J 8 Y l R B j i U y p w Y 8 E z 9 P Z G R S O t J F J j k d A G 9 7 E 3 F V V 4 v h f D K z 3 i c p M B i O r 8 o T A U G i a d V 4 w F X j I K Y G E K o 4 u a t m I 6 I I h T M h 5 R N C e 7 y y v / J 8 3 n D d R r u w 0 W t 2 V r U U U R H 6 B j V k Y s u U R P d o g 7 q I o q + L G S V r L L 1 b R f s i r 0 z j 9 r W Y u Y A / Y F 9 + A M 8 t 7 d x < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " u y 6 m g 7 G g W I 9 p g a W R d j o U T 0 / l m Q k = " > A A A C X X i c b V H P S 8 M w G E 2 r 0 2 1 O n X r w 4 C U 4 h I k w W h H 0 I o x 5 8 T B x o l N h L S P N 0 i 0 s b U r y V R y l / 6 Q 3 v f i v m P 0 4 6 N y D w O O 9 9 5 F 8 L 0 x j d T / + W N K c 1 l / A S T h P k R G c Y 8 5 J S A k f p V 8 C I C I 0 p E 1 s 7 r H r B 3 y E I p I S B C 5 K f 4 G v + y + 9 n c v 2 v f 5 S u y Z 6 u y j 6 3 7 F d l + t e Y 0 n B n w f + I u S A 0 t 0 O l X P 7 y B p G n E Y q C C a N 1 z n Q T 8 j C j g V L C 8 7 K W a J Y S O y Z D 1 D I 1 J x L S f z d r J 8 Y l R B j i U y p w Y 8 E z 9 P Z G R S O t J F J j k d A G 9 7 E 3 F V V 4 v h f D K z 3 i c p M B i O r 8 o T A U G i a d V 4 w F X j I K Y G E K o 4 u a t m I 6 I I h T M h 5 R N C e 7 y y v / J 8 3 n D d R r u w 0 W t 2 V r U U U R H 6 B j V k Y s u U R P d o g 7 q I o q + L G S V r L L 1 b R f s i r 0 z j 9 r W Y u Y A / Y F 9 + A M 8 t 7 d x < / l a t e x i t > = log P (football | x 7 ) log P (football | x 4 , x 9 , p 3 ) x 1 < l a t e x i t s h a 1 _ b a s e 6 4 = " Z Y + R s J i l E 2 D s + 9 + V G 7 h S 6 g 3 T w D K / w 5 q T O i / P u f C x H N 5 x i 5 w z + w P n 8 A S e u k b M = < / l a t e x i t > x 3 < l a t e x i t s h a 1 _ b a s e 6 4 = " 0 M Z D 8 j u x 4 < l a t e x i t s h a 1 _ b a s e 6 4 = " l L b j s G 9 M A Z d q Q r 7 I v d q C I a i l b g 0 A n e 6 p f X S e e q 7 r l 1 7 7 5 R a z a K O s p w B u d w C R 5 c Q x P u o A V t Y J D A M 7 z C m 5 M 6 L 8 6 7 8 7 E c L T n F z i n 8 g f P 5 A y q 2 k b U = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " l L b j s G 9 M A Z d q Q r 7 I v d q C I a i l b g 0 A n e 6 p f X S e e q 7 r l 1 7 7 5 R a z a K O s p w B u d w C R 5 c Q x 6 < l a t e x i t s h a 1 _ b a s e 6 4 = " Z j P F 9 0 A n e 6 p f X S e e q 7 r l 1 7 / 6 6 1 r w u 6 A n e 6 p f X S e e q 7 r l 1 7 / 6 6 1 r w u 6 A n e 6 p f X S e e q 7 r l 1 7 / 6 6 1 r w u 6 A n e 6 p f X S e e q 7 r l 1 7 / 6 6 1 r w u 6 x 8 < l a t e x i t s h a 1 _ b a s e 6 4 = " y s K c S C + e 8 M b M h / J 6 6 6 O 9 c 8 k w o G x 9 < l a t e x i t s h a 1 _ b a s e 6 4 = " x 10 < l a t e x i t s h a 1 _ b a s e 6 4 = " 1 m b t u r c i G w x 6 x 11 < l a t e x i t s h a 1 _ b a s e 6 4 = " T Z z n U P D W L P 2 t j L y x Z T 9 a L 9 W 5 9 L E c r V r l z A n 9 g f f 4 A 3 v C T K g = = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " T Z z n U P D W L P 2 t j L y x 12 < l a t e x i t s h a 1 _ b a s e 6 4 = " B / n N x h / 4 9 4 variety of tasks, and reach substantially better performance on span selection tasks in particular.Specifically, our method reaches 94.6% and 88.7% F1 on SQuAD 1.1 and 2.0 (Rajpurkar et al., 2016(Rajpurkar et al., , 2018)), respectively -reducing error by as much as 27% compared to our tuned BERT replica.We also observe similar gains on five additional extractive question answering benchmarks (NewsQA, TriviaQA, SearchQA, HotpotQA, and Natural Questions). 2panBERT also arrives at a new state of the art on the challenging CoNLL-2012 ("OntoNotes") shared task for document-level coreference resolution, where we reach 79.6% F1, exceeding the previous top model by 6.6% absolute.Finally, we demonstrate that SpanBERT also helps on tasks that do not explicitly involve span selection, and show that our approach even improves performance on TACRED (Zhang et al., 2017) and GLUE (Wang et al., 2019).
While others show the benefits of adding more data (Yang et al., 2019) and increasing model size (Lample and Conneau, 2019), this work demonstrates the importance of designing good pretraining tasks and objectives, which can also have a remarkable impact.
2 Background: BERT BERT (Devlin et al., 2019) is a self-supervised approach for pre-training a deep transformer encoder (Vaswani et al., 2017), before fine-tuning it for a particular downstream task.BERT optimizes two training objectives -masked language model (MLM) and next sentence prediction (NSP) -which only require a large collection of unlabeled text.
Masked Language Model (MLM) Also known as a cloze test, MLM is the task of predicting missing tokens in a sequence from their placeholders.Specifically, a subset of tokens Y ⊆ X is sampled and substituted with a different set of tokens.In BERT's implementation, Y accounts for 15% of the tokens in X; of those, 80% are replaced with [MASK], 10% are replaced with a random token (according to the unigram distribution), and 10% are kept unchanged.The task is to predict the original tokens in Y from the modified input.
BERT selects each token in Y independently by randomly selecting a subset.In SpanBERT, we define Y by randomly selecting contiguous spans (Section 3.1).

Next Sentence Prediction (NSP)
The NSP task takes two sequences (X A , X B ) as input, and predicts whether X B is the direct continuation of X A .This is implemented in BERT by first reading X A from the corpus, and then (1) either reading X B from the point where X A ended, or (2) randomly sampling X B from a different point in the cor-pus.The two sequences are separated by a special [SEP]token.Additionally, a special [CLS]token is added to X A , X B to form the input, where the target of [CLS]is whether X B indeed follows X A in the corpus.
In summary, BERT optimizes the MLM and the NSP objectives by masking word pieces uniformly at random in data generated by the bisequence sampling procedure.In the next section, we will present our modifications to the data pipeline, masking, and pre-training objectives.

Model
We present SpanBERT, a self-supervised pretraining method designed to better represent and predict spans of text.Our approach is inspired by BERT (Devlin et al., 2019), but deviates from its bi-text classification framework in three ways.First, we use a different random process to mask spans of tokens, rather than individual ones.We also introduce a novel auxiliary objective -the span boundary objective (SBO) -which tries to predict the entire masked span using only the representations of the tokens at the span's boundary.Finally, SpanBERT samples a single contiguous segment of text for each training example (instead of two), and thus does not use BERT's next sentence prediction objective, which we omit.

Span Masking
Given a sequence of tokens X = (x 1 , x 2 , . . ., x n ), we select a subset of tokens Y ⊆ X by iteratively sampling spans of text until the masking budget (e.g.15% of X) has been spent.At each iteration, we first sample a span length (number of words) from a geometric distribution ∼ Geo(p), which is skewed towards shorter spans.We then randomly (uniformly) select the starting point for the span to be masked.We always sample a sequence of complete words (instead of subword tokens) and the starting point must be the beginning of one word.Following preliminary trials3 , we set p = 0.2, and also clip at max = 10.This yields a mean span length of mean( ) = 3.8.Figure 2 shows the distribution of span mask lengths.
As in BERT, we also mask 15% of the tokens in total: replacing 80% of the masked tokens with [MASK], 10% with random tokens and 10% with the original tokens.However, we perform this re- 1 2 3 4 5 6 7 8 9  placement at the span level and not for each token individually; i.e. all the tokens in a span are replaced with [MASK]or sampled tokens.

Span Boundary Objective (SBO)
Span selection models (Lee et al., 2016(Lee et al., , 2017;;He et al., 2018) typically create a fixed-length representation of a span using its boundary tokens (start and end).To support such models, we would ideally like the representations for the end of the span to summarize as much of the internal span content as possible.We do so by introducing a span boundary objective that involves predicting each token of a masked span using only the representations of the observed tokens at the boundaries (Figure 1).Formally, we denote the output of the transformer encoder for each token in the sequence by x 1 , . . ., x n .Given a masked span of tokens (x s , . . ., x e ) ∈ Y , where (s, e) indicates its start and end positions, we represent each token x i in the span using the output encodings of the external boundary tokens x s−1 and x e+1 , as well as the position embedding of the target token p i−s+1 : where position embeddings p 1 , p 2 , . . .mark relative positions of the masked tokens with respect to the left boundary token x s−1 .We implement the representation function f (•) as a 2layer feed-forward network with GeLU activations (Hendrycks and Gimpel, 2016) and layer normal-ization (Ba et al., 2016): We then use the vector representation y i to predict the token x i and compute the cross-entropy loss exactly like the MLM objective.
SpanBERT sums the loss from both the span boundary and the regular masked language model objectives for each token x i in the masked span (x s , ..., x e ), while reusing the input embedding (Press and Wolf, 2017) for the target tokens in both MLM and SBO:

Single-Sequence Training
As described in Section 2, BERT's examples contain two sequences of text (X A , X B ), and an objective that trains the model to predict whether they are connected (NSP).We find that this setting is almost always worse than simply using a single sequence without the NSP objective (see Section 5 for further details).We conjecture that single-sequence training is superior to bi-sequence training with NSP because (a) the model benefits from longer full-length contexts, or (b) conditioning on, often unrelated, context from another document adds noise to the masked language model.Therefore, in our approach, we remove both the NSP objective and the two-segment sampling procedure, and simply sample a single contiguous segment of up to n = 512 tokens, rather than two half-segments that sum up to n tokens together.
In summary, SpanBERT pre-trains span representations by: (1) masking spans of full words using a geometric distribution based masking scheme (Section 3.1), ( 2) optimizing an auxiliary span-boundary objective (Section 3.2) in addition to MLM using a single-sequence data pipeline (Section 3.3).A procedural description can be found in Appendix A.

Tasks
We evaluate on a comprehensive suite of tasks, including seven question answering tasks, corefer-ence resolution, nine tasks in the GLUE benchmark (Wang et al., 2019), and relation extraction.We expect that the span selection tasks, question answering and coreference resolution, will particularly benefit from our span-based pre-training.
Extractive Question Answering Given a short passage of text and a question as input, the task of extractive question answering is to select a contiguous span of text in the passage as the answer.
We first evaluate on SQuAD 1.1 and 2.0 (Rajpurkar et al., 2016(Rajpurkar et al., , 2018)), which have served as major question answering benchmarks, particularly for pre-trained models (Peters et al., 2018;Devlin et al., 2019;Yang et al., 2019).We also evaluate on five more datasets from the MRQA shared task (Fisch et al., 2019) 4 : NewsQA (Trischler et al., 2017), SearchQA (Dunn et al., 2017), TriviaQA (Joshi et al., 2017), Hot-potQA (Yang et al., 2018) and Natural Questions (Kwiatkowski et al., 2019).Because the MRQA shared task does not have a public test set, we split the development set in half to make new development and test sets.The datasets vary in both domain and collection methodology, making this collection a good testbed for evaluating whether our pre-trained models can generalize well across different data distributions.
Following BERT (Devlin et al., 2019), we use the same QA model architecture for all the datasets.
We first convert the passage P = (p 1 , p 2 , . . ., p l ) and question Q = (q 1 , q 2 , . . ., q l ) into a single sequence X = [CLS]p 1 p 2 . . .p l [SEP]q 1 q 2 . . .q l [SEP], pass it to the pre-trained transformer encoder, and train two linear classifiers independently on top of it for predicting the answer span boundary (start and end).For the unanswerable questions in SQuAD 2.0, we simply set the answer span to be the special token [CLS]for both training and testing.
Coreference Resolution Coreference resolution is the task of clustering mentions in text which refer to the same real-world entities.We evaluate on the CoNLL-2012 shared task (Pradhan et al., 2012) for document-level coreference resolution.We use the independent version of the Joshi et al. (2019b) implementation of the higher-order coref-erence model (Lee et al., 2018).The document is divided into non-overlapping segments of a predefined length. 5Each segment is encoded independently by the pre-trained transformer encoder, which replaces the original LSTM-based encoder.For each mention span x, the model learns a distribution P (•) over possible antecedent spans Y : ,y)   y ∈Y e s(x,y ) The span pair scoring function s(x, y) is a feedforward neural network over fixed-length span representations and hand-engineered features over x and y: Here g x and g y denote the span representations, which are a concatenation of the two transformer output states of the span endpoints and an attention vector computed over the output representations of the token in the span.FFNN m and FFNN c represent two feedforward neural networks with one hidden layer, and φ(x, y) represents the handengineered features (e.g.speaker and genre information).A more detailed description of the model can be found in Joshi et al. (2019b).
Relation Extraction TACRED (Zhang et al., 2017) is a challenging relation extraction dataset.Given one sentence and two spans within it -subject and object -the task is to predict the relation between the spans from 42 pre-defined relation types, including no_relation.We follow the entity masking schema from Zhang et al. (2017)  GLUE The General Language Understanding Evaluation (GLUE) benchmark (Wang et al., 2019) consists of 9 sentence-level classification tasks: • Two sentence-level classification tasks including CoLA (Warstadt et al., 2018) for evaluating linguistic acceptability and SST-2 (Socher et al., 2013) for sentiment classification.
• Three sentence-pair similarity tasks including MRPC (Dolan and Brockett, 2005), a binary paraphrasing task sentence pairs from news sources, STS-B (Cer et al., 2017), a graded similarity task for news headlines, and QQP6 , a binary paraphrasing tasking between Quora question pairs.
Unlike question answering, coreference resolution, and relation extraction, these sentence-level tasks do not require explicit modeling of spanlevel semantics.However, they might still benefit from implicit span-based reasoning (e.g., the Prime Minister is the head of the government).
Following previous work (Devlin et al., 2019;Radford et al., 2018) 7 , we exclude WNLI from the results to enable a fair comparison.While recent work Liu et al. (2019a) has applied several taskspecific strategies to increase performance on the individual GLUE tasks, we follow BERT's singletask setting and only add a linear classifier on top of the [CLS]token for these classification tasks.

Implementation
We reimplemented BERT's model and pretraining method in fairseq (Ott et al., 2019).We used the model configuration of BERT large as in Devlin et al. (2019) and also pre-trained all our models on the same corpus: BooksCorpus and English Wikipedia using cased Wordpiece tokens.Compared to the original BERT implementation, the main differences in our implementation include: (a) We use different masks at each epoch while BERT samples 10 different masks for each sequence during data processing.(b) We remove all the short-sequence strategies used before (they sampled shorter sequences with a small probability 0.1; they also first pre-trained with smaller se-quence length of 128 for 90% of the steps).Instead, we always take sequences of up to 512 tokens until it reaches a document boundary.We refer readers to Liu et al. (2019b) for further discussion on these modifications and their effects.
As in BERT, the learning rate is warmed up over the first 10,000 steps to a peak value of 1e-4, and then linearly decayed.We retain β hyperparameters (β 1 = 0.9, β 2 = 0.999) and a decoupled weight decay (Loshchilov and Hutter, 2019) of 0.1.We also keep a dropout of 0.1 on all layers and attention weights, and a GeLU activation function (Hendrycks and Gimpel, 2016).We deviate from the optimization by running for 2.4M steps and using an epsilon of 1e-8 for AdamW (Kingma and Ba, 2015), which converges to a better set of model parameters.Our implementation uses a batch size of 256 sequences with a maximum of 512 tokens. 8For the SBO, we use 200 dimension position embeddings p 1 , p 2 , . . . to mark positions relative to the left boundary token.The pre-training was done on 32 Volta V100 GPUs and took 15 days to complete.
Fine-tuning is implemented based on Hugging-Face's codebase (Wolf et al., 2019) and more details are given in Appendix B.

Baselines
We compare SpanBERT to three baselines: Google BERT The pre-trained models released by Devlin et al. (2019).9 Our BERT Our reimplementation of BERT with improved data preprocessing and optimization (Section 4.2).
Our BERT-1seq Our reimplementation of BERT trained on single full-length sequences without NSP (Section 3.3).

Results
We compare SpanBERT to the baselines per task, and draw conclusions based on the overall trends.

Per-Task Results
Extractive Question Answering Table 1 shows the performance on both SQuAD 1.1 and 2.0.SpanBERT exceeds our BERT baseline by 2.0% and 2.8% F1 respectively (3.3% and 5.4% over Google BERT).In SQuAD 1.1, this result accounts for over 27% error reduction, reaching 3.4% F1 above human performance.Table 2 demonstrates that this trend goes beyond SQuAD, and is consistent in every MRQA dataset.On average, we see a 2.9% F1 improvement from our reimplementation of BERT.Although some gains are coming from singlesequence training (+1.1%), most of the improvement stems from span masking and the span boundary objective (+1.8%), with particularly large gains on TriviaQA (+3.2%) and HotpotQA (+2.7%).
Coreference Resolution Table 3 shows the performance on the OntoNotes coreference resolution benchmark.Our BERT reimplementation improves the Google BERT model by 1.2% on the average F1 metric and single-sequence training brings another 0.5% gain.Finally, SpanBERT improves considerably on top of that, achieving a new state of the art of 79.6% F1 (previous best result is 73.0%).(but considerable) improvements on MRPC and MNLI.The main gains from SpanBERT are in the SQuAD-based QNLI dataset (+1.3%) and in RTE (+6.9%), the latter accounting for most of the rise in SpanBERT's GLUE average.

Overall Trends
We compared our approach to three BERT baselines on 17 benchmarks, and found that Span-BERT outperforms BERT on almost every task.In 14 tasks, SpanBERT performed better than all baselines.In 2 tasks (MRPC and QQP), it performed on-par in terms of accuracy with singlesequence trained BERT, but still outperformed the other baselines.In one task (SST-2), Google's BERT baseline performed better than SpanBERT by 0.4% accuracy.
When considering the magnitude of the gains, it appears that SpanBERT is especially better at extractive question answering.In SQuAD 1.1, for example, we observe a solid gain of 2.0% F1 even though the baseline is already well above human performance.On MRQA, SpanBERT improves between 2.0% (Natural Questions) and 4.6% (TriviaQA) F1 on top of our BERT baseline.
Finally, we observe that single-sequence training works considerably better than bi-sequence training with next sentence prediction (NSP) with BERT's choice of sequence lengths for a wide variety of tasks .This is surprising because BERT's ablations showed gains from the NSP objective (Devlin et al., 2019).However, the ablation studies still involved bi-sequence data processing, i.e. the pre-training stage only controlled for the NSP objective while still sampling two half-length sequences.We hypothesize that bisequence training, as it is implemented in BERT (see Section 2), impedes the model from learning longer-range features, and consequently hurts performance on many downstream tasks.

Ablation Studies
We compare our random span masking scheme with linguistically-informed masking schemes, and find that masking random spans is a competitive and often better approach.We then study the impact of the span boundary objective (SBO), and contrast it with BERT's next sentence prediction noun phrases) are often competitive with random spans, their performance is not consistent; for instance, masking noun phrases achieves parity with random spans on NewsQA, but underperforms on TriviaQA (-1.1% F1).
On coreference resolution, we see that masking random subword tokens is preferable to any form of span masking.Nevertheless, we shall see in the following experiment that combining random span masking with the span boundary objective can improve upon this result considerably.

Auxiliary Objectives
In Section 5, we saw that bi-sequence training with the next sentence prediction (NSP) objective can hurt performance on downstream tasks, when compared to single-sequence training.We test whether this holds true for models pre-trained with span masking, and also evaluate the effect of replacing the NSP objective with the span boundary objective (SBO).
Table 7 confirms that single-sequence training typically improves performance.Adding SBO further improves performance, with a substantial gain on coreference resolution (+2.7% F1) over span masking alone.Unlike the NSP objective, SBO does not appear to have any adverse effects.

Related Work
Pre-trained contextualized word representations that can be trained from unlabeled text (Dai and Le, 2015;Melamud et al., 2016;Peters et al., 2018) have had immense impact on NLP lately, particularly as methods for initializing a large model before fine-tuning it for a specific task (Howard and Ruder, 2018;Radford et al., 2018;Devlin et al., 2019).Beyond differences in model hyperparameters and corpora, these methods mainly differ in their pre-training tasks and loss functions, with a considerable amount of contemporary literature proposing augmentations of BERT's masked language modeling (MLM) objective.
While previous and concurrent work has looked at masking (Sun et al., 2019) or dropping (Song et al., 2019;Chan et al., 2019) multiple words from the input -particularly as pretraining for language generation tasks -SpanBERT pretrains span representations (Lee et al., 2016), which are widely used for question answering, coreference resolution and a variety of other tasks.ERNIE (Sun et al., 2019) shows improvements on Chinese NLP tasks using phrase and named entity masking.MASS (Song et al., 2019) focuses on language generation tasks, and adopts the encoderdecoder framework to reconstruct a sentence fragment given the remaining part of the sentence.We attempt to more explicitly model spans using the SBO objective, and show that (geometrically distributed) random span masking works as well, and sometimes better than, masking linguisticallycoherent spans.We evaluate on English benchmarks for question answering, relation extraction, and coreference resolution in addition to GLUE.
A different ERNIE (Zhang et al., 2019) focuses on integrating structured knowledge bases with contextualized representations with an eye on knowledge-driven tasks like entity typing and relation classification.UNILM (Dong et al., 2019) uses multiple language modeling objectives -unidirectional (both left-to-right and right-to-left), bidirectional, and sequence-to-sequence prediction -to aid generation tasks like summarization and question generation.XLM (Lample and Conneau, 2019) explores cross-lingual pre-training for multilingual tasks such as translation and crosslingual classification.Kermit (Chan et al., 2019), an insertion based approach, fills in missing tokens (instead of predicting masked ones) during pretraining; they show improvements on machine translation and zero-shot question answering.
Concurrent with our work, RoBERTa (Liu et al., 2019b) presents a replication study of BERT pre-training that measures the impact of many key hyperparameters and training data size.Also concurrent, XLNet (Yang et al., 2019) combines an autoregressive loss and the Transformer-XL (Dai et al., 2019) architecture with a more than an eight-fold increase in data to achieve current stateof-the-art results on multiple benchmarks.XL-Net also masks spans (of 1-5 tokens) during pretraining, but predicts them autoregressively.Our model focuses on incorporating span-based pretraining, and as a side effect, we present a stronger BERT baseline while controlling for the corpus, architecture, and the number of parameters.Related to our SBO objective, pair2vec (Joshi et al., 2019a) encodes word-pair relations using a negative sampling-based multivariate objective during pre-training.Later, the word-pair representations are injected into the attention-layer of downstream tasks, and thus encode limited down-stream context.Unlike pair2vec, our SBO objective yields "pair" (start and end tokens of spans) representations which more fully encode the context during both pre-training and finetuning, and are thus more appropriately viewed as span representations.Stern et al. (2018) focus on improving language generation speed using a block-wise parallel decoding scheme; they make predictions for multiple time steps in parallel and then back off to the longest prefix validated by a scoring model.Also related are sentence representation methods (Kiros et al., 2015;Logeswaran and Lee, 2018) which focus on predicting surrounding contexts from sentence embeddings.

Conclusion
We presented a new method for span-based pretraining which extends BERT by (1) masking contiguous random spans, rather than random tokens, and (2) training the span boundary representations to predict the entire content of the masked span, without relying on the individual token representations within it.Together, our pre-training process yields models that outperform BERT baselines on a variety of tasks, and reach substantially better performance on span selection tasks in particular.
) = L MLM (football) + L SBO (football) H N 5 x i 5 w z + w P n 8 A S e u k b M = < / l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " / A i Q R o 6 P e m d E R O W 9 8 6 e B W M M c x 9 k = " > A A A B 8 3 i c b V D L S s N A F L 3 x W e u r 6 t L N Y B F c l a Q U d F l w 4 7 K C f U A b y m R 6 0 w 6 d T M L M R C y h v + H G h S J u / R l 3 / o 2 T N g t t P T B w O O d e 7 p k T J I J r 4 7 r f z s b m 1 v b O b m l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y s K c S C + e 8 M b M h / J 6 6 6 O 9 c 8 k w o l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y s K c S C + e 8 M b M h / J 6 6 6 O 9 c 8 k w o l a t e x i t > < l a t e x i t s h a 1 _ b a s e 6 4 = " y s K c S C + e 8 M b M h / J 6 6 6 O 9 c 8 k w o

Figure 1 :
Figure 1: An illustration of SpanBERT training.The span an American football game is masked.The span boundary objective (SBO) uses the output representations of the boundary tokens, x 4 and x 9 (in blue), to predict each token in the masked span.The equation shows the MLM and SBO loss terms for predicting the token, football (in pink), which as marked by the position embedding p 3 , is the third token from x 4 .
and replace the subject and object entities by their NER tags such as "[CLS][SUBJ-PER] was born in [OBJ-LOC] , Michigan, . . .", and finally add a linear classifier on top of the [CLS]token to predict the relation type.

Table 1 :
Test results on SQuAD 1.1 and SQuAD 2.0.

Table 4
GLUE Table5shows the performance on GLUE.For most tasks, the different models appear to perform similarly.Moving to singlesequence training without the NSP objective substantially improves CoLA, and yields smaller NewsQA TriviaQA SearchQA HotpotQA Natural Questions Avg.

Table 3 :
Performance on the OntoNotes coreference resolution benchmark.The main evaluation is the average F1 of three metrics: MUC, B 3 , and CEAF φ4 on the test set.

Table 4 :
Test performance on the TACRED relation extraction benchmark.BERT EM and BERT EM +MTB from Soares et al. (2019) are the current state-of-theart.
* : BERT EM +MTB incorporated an intermediate "matching the blanks" pre-training on the entity-linked text based on English Wikipedia, which is not a direct comparison to ours trained only from raw text.