SECTOR: A Neural Model for Coherent Topic Segmentation and Classification

When searching for information, a human reader first glances over a document, spots relevant sections, and then focuses on a few sentences for resolving her intention. However, the high variance of document structure complicates the identification of the salient topic of a given section at a glance. To tackle this challenge, we present SECTOR, a model to support machine reading systems by segmenting documents into coherent sections and assigning topic labels to each section. Our deep neural network architecture learns a latent topic embedding over the course of a document. This can be leveraged to classify local topics from plain text and segment a document at topic shifts. In addition, we contribute WikiSection, a publicly available data set with 242k labeled sections in English and German from two distinct domains: diseases and cities. From our extensive evaluation of 20 architectures, we report a highest score of 71.6% F1 for the segmentation and classification of 30 topics from the English city domain, scored by our SECTOR long short-term memory model with Bloom filter embeddings and bidirectional segmentation. This is a significant improvement of 29.5 points F1 over state-of-the-art CNN classifiers with baseline segmentation.


Introduction
Today's systems for natural language understanding are comprised of building blocks that extract semantic information from the text, such as named entities, relations, topics or discourse structure.In traditional natural language processing (NLP), these extractors are typically applied to bags of words or full sentences (Hirschberg and Manning, 2015).Recent neural architectures build upon pretrained word or sentence embeddings (Mikolov et al., 2013;Le and Mikolov, 2014), which fo-cus on semantic relations that can be learned from large sets of paradigmatic examples, even from long ranges (Dieng et al., 2017).
From a human perspective, however, it is mostly the authors themselves who help best to understand a text.Especially in long documents, an author thoughtfully designs a readable structure and guides the reader through the text by arranging topics into coherent passages (Glavaš et al., 2016).In many cases, this structure is not formally expressed as section headings (e.g. in news articles, reviews, discussion forums) or it is structured according to domain-specific aspects (e.g.health reports, research papers, insurance documents).
Ideally, systems for text analytics, such as topic detection and tracking (TDT) (Allan, 2002), text summarization (Huang et al., 2003), information retrieval (IR) (Dias et al., 2007) or question answering (QA) (Cohen et al., 2018) could access a document representation that is aware of both topical (i.e.latent semantic content) and structural information (i.e.segmentation) in the text (MacAvaney et al., 2018).The challenge in building such a representation is to combine these two dimensions which are strongly interwoven in the author's mind.It is therefore important to understand topic segmentation and classification as a mutual task that requires to encode both topic information and document structure coherently.
In this article, we present SECTOR 1 , an endto-end model which learns an embedding of latent topics from potentially ambiguous headings and can be applied to entire documents to predict local topics on sentence level.Our model encodes topical information on a vertical dimension and structural information on a horizontal dimension.We show that the resulting embedding can be leveraged in a downstream pipeline to seg-ment a document into coherent sections and classify the sections into one of up to 30 topic categories reaching 71.6% F 1 -or alternatively attach up to 2.8k topic labels with 71.1% MAP.We further show that segmentation performance of our bidirectional LSTM architecture is comparable to specialized state-of-the-art segmentation methods on various real-world datasets.
To the best of our knowledge, the combined task of segmentation and classification has not been approached on full document level before.There exist a large number of datasets for text segmentation, but most of them do not reflect real-world topic drifts (Choi, 2000;Sehikh et al., 2017), do not include topic labels (Eisenstein and Barzilay, 2008;Jeong and Titov, 2010;Glavaš et al., 2016) or are heavily normalized and too small to be used for training neural networks (Chen et al., 2009).We can utilize a generic segmentation dataset derived from Wikipedia that includes headings (Koshorek et al., 2018), but there is also a need in IR and QA for supervised structural topic labels (Agarwal and Yu, 2009;MacAvaney et al., 2018), different languages and more specific domains, such as clinical or biomedical research (Tepper et al., 2012;Tsatsaronis et al., 2012) and news-based TDT (Kumaran and Allan, 2004;Leetaru and Schrodt, 2013).
Therefore we introduce WIKISECTION2 , a large novel dataset of 38k articles from the English and German Wikipedia labeled with 242k sections, original headings and normalized topic labels for up to 30 topics from two domains: diseases and cities.We chose these subsets to cover both clinical/biomedical aspects (e.g.symptoms, treatments, complications) and news-based topics (e.g.history, politics, economy, climate).Both article types are reasonably well-structured according to Wikipedia guidelines (Piccardi et al., 2018), but we show that they are also complementary: diseases is a typical scientific domain with low entropy, i.e. very narrow topics, precise language and low word ambiguity.In contrast, cities resembles a diversified domain, with high entropy, i.e. broader topics, common language and higher word ambiguity, and will be more applicable to e.g.news, risk reports or travel reviews.
We compare SECTOR to existing segmentation and classification methods based on latent dirich-let allocation (LDA), paragraph embeddings, convolutional neural networks (CNNs) and recurrent neural networks (RNNs).We show that SECTOR significantly improves these methods in a combined task by up to 29.5 points F 1 when applied to plain text with no given segmentation.
The rest of this article is structured as follows: We introduce related work in Section 2. Next, we describe the task and dataset creation process in Section 3. We formalize our model in Section 4. We report results and insights from the evaluation in Section 5. Finally, we conclude in Section 6.

Related Work
The analysis of emerging topics over the course of a document is related to a large number of research areas.In particular, topic modeling (Blei et al., 2003) and topic detection and tracking (TDT) (Jin et al., 1999) focus on representing and extracting the semantic topical content of text.Text segmentation (Beeferman et al., 1999) is used to split documents into smaller coherent chunks.Finally, text classification (Joachims, 1998) is often applied to detect topics on text chunks.Our method unifies those strongly interwoven tasks and is the first to evaluate the combined topic segmentation and classification task using a corresponding dataset with long structured documents.
Topic modeling is commonly applied to entire documents using probabilistic models, such as latent Dirichlet allocation (LDA) (Blei et al., 2003).AlSumait et al. (2008) introduced an online topic model that captures emerging topics when new documents appear.Gabrilovich and Markovitch (2007) proposed the Explicit Semantic Analysis method in which concepts from Wikipedia articles are indexed and assigned to documents.Later, and to overcome the vocabulary mismatch problem, Cimiano et al. (2009) introduced a method for assigning latent concepts to documents.More recently, Liu et al. (2016) represented documents with vectors of closely related domain keyphrases.Yeh et al. (2016) proposed a conceptual dynamic LDA model for tracking topics in conversations.Bhatia et al. (2016) utilized Wikipedia document titles to learn neural topic embeddings and assign document labels.Dieng et al. (2017) focused on the issue of long-range dependencies and proposed a latent topic model based on recurrent neural networks (RNNs).However, the authors did not apply the RNN to predict local topics.Text segmentation has been approached with a wide variety of methods.
Later, supervised methods included topic models (Riedl and Biemann, 2012) by calculating a coherence score using dense topic vectors obtained by LDA.Bayomi et al. (2015) exploited ontologies to measure semantic similarity between text blocks.Alemi and Ginsparg (2015) and Naili et al. (2017) studied how word embeddings can improve classical segmentation approaches.Glavaš et al. (2016) utilized semantic relatedness of word embeddings by identifying cliques in a graph.
More recently, Sehikh et al. (2017) utilized long short-term memory (LSTM) networks and showed that cohesion between bidirectional layers can be leveraged to predict topic changes.In contrast to our method, the authors focused on segmenting speech recognition transcripts on word level without explicit topic labels.The network was trained with supervised pairs of contrary examples and was mainly evaluated on artificiallysegmented documents.Our approach extends this idea so it can be applied to dense topic embeddings which are learned from raw section headings.Wang et al. (2017) tackled segmentation by training a CNN to learn coherence scores for text pairs.Similar to Sehikh et al. (2017), the network was trained with short contrary examples and no topic objective.The authors showed that their point-wise ranking model performs well on datasets by Jeong and Titov (2010).In contrast to our method, the ranking algorithm strictly requires a given ground truth number of segments for each document and no topic labels are predicted.Koshorek et al. (2018) presented a large new dataset for text segmentation based on Wikipedia that includes section headings.The authors introduced a neural architecture for segmentation which is based on sentence embeddings and four layers of bidirectional LSTM.Similar to Sehikh et al. (2017), the authors used a binary segmentation objective on sentence level, but trained on entire documents.Our work takes up this idea of end-to-end training and enriches the neural model with a layer of latent topic embeddings that can be utilized for topic classification.
Text classification is mostly applied at paragraph or sentence level using machine learning methods such as Support Vector Machines (Joachims, 1998) or, more recently, shallow and deep neural networks (Hoa T. Le et al., 2018;Conneau et al., 2017).Notably, Paragraph Vectors (Le and Mikolov, 2014) is an extension of word2vec for learning fixed-length distributed representations from texts of arbitrary length.The resulting model can be utilized for classification by providing paragraph labels during training.Furthermore, Kim (2014) has shown that convolutional neural networks (CNNs) combined with pre-trained taskspecific word embeddings achieve highest scores for various text classification tasks.
Combined approaches of topic segmentation and classification are rare to find.Agarwal and Yu (2009)  Central articles into four structural classes (introduction, methods, results and discussion).However, their manually-labeled dataset only contains a sample of sentences from the documents, so they evaluated sentence classification as an isolated task.Chen et al. ( 2009) introduced two Wikipedia-based datasets for segmentation, one about large cities, the second about chemical elements.While these datasets have been used to evaluate word-level and sentence-level segmentation (Koshorek et al., 2018), we are not aware of any topic classification approach on this dataset.Tepper et al. (2012) approached segmentation and classification in a clinical domain as supervised sequence labeling problem.The documents were segmented using a Maximum Entropy model and then classified into 11 or 33 categories.A similar approach by Ajjour et al. (2017) used sequence labeling with a small number of 3-6 classes.Their model is extractive, so it does not produce a continuous segmentation over the entire document.Finally, Piccardi et al. (2018) did not approach segmentation, but recommended an ordered set of section labels based on Wikipedia articles.
Eventually, we are inspired by passage retrieval (Liu and Croft, 2002) as an important downstream task for topic segmentation and classification.For example, Hewlett et al. (2016) proposed WikiReading, a QA task to retrieve values from sections of long documents.The objective of TREC Complex Answer Retrieval is to retrieve a ranking of relevant passages for a given outline of hierarchical sections (Nanni et al., 2017).Both tasks highly depend on a building block for local topic embeddings such as our proposed model.

Task Overview and Dataset
We start with a definition of the WIKISECTION machine reading task shown in Figure 1.We take a document D = S, T consisting of N consecutive sentences S = [s 1 , . . ., s N ] and empty segmentation T = ∅ as input.In our example, this is the plain text of a Wikipedia article (e.g. about Trichomoniasis3 ) without any section information.For each sentence s k , we assume a distribution of local topics e k that gradually changes over the course of the document.
The task is to split D into a sequence of distinct topic sections T = [T 1 , . . ., T M ], so that each predicted section T j = S j , y j contains a sequence of coherent sentences S j ⊆ S and a topic label y j that describes the common topic in these sentences.For the document Trichomoniasis, the sequence of topic labels is y 1...M = [ symptom, cause, diagnosis, prevention, treatment, complication, epidemiology ].

WikiSection Dataset
For the evaluation of this task, we created WIKI-SECTION, a novel dataset containing a gold standard of 38k full-text documents from English and German Wikipedia comprehensively annotated with sections and topic labels (see Table 1).
The documents originate from recent dumps in English4 and German5 .We filtered the collection using SPARQL queries against Wikidata (Tanon et al., 2016).We retrieved instances of Wikidata categories disease (Q12136) and their subcategories, e.g.Trichomoniasis or Pertussis, or city (Q515), e.g.London or Madrid.
Our dataset contains the article abstracts, plain text of the body, positions of all sections given by the Wikipedia editors with their original headings (e.g."Causes | Genetic sequence") and a normalized topic label (e.g.disease.cause).We randomized the order of documents and split them into 70% training, 10% validation, 20% test sets.

Preprocessing
To obtain plain document text, we used Wikiextractor6 , split the abstract sections and stripped all section headings and other structure tags except newline characters and lists.Vocabulary mismatch in section headings.Table 2 shows examples of section headings from disease articles separated into head (most common), torso (frequently used) and tail (rare).Initially, we expected articles to share congruent structure in naming and order.Instead, we observe a high variance with 8.5k distinct headings in the diseases domain and over 23k for English cities.
A closer inspection reveals that Wikipedia authors utilize headings at different granularity levels, frequently copy and paste from other articles, but also introduce synonyms or hyponyms, which leads to a vocabulary mismatch problem (Furnas et al., 1987).As a result, the distribution of headings is heavy-tailed across all articles.Roughly 1% of headings appear more than 25 times while the vast majority (88%) appear 1 or 2 times only.

Synset Clustering
In order to use Wikipedia headlines as a source for topic labels, we contribute a normalization method to reduce the high variance of headings to few representative labels based on the clustering of Babel-Net synsets (Navigli and Ponzetto, 2012).We create a set H that contains all headings in the dataset and use the BabelNet API to match7 each heading h ∈ H to its corresponding synsets S h ⊂ S. For example, "Cognitive behavioral therapy" is assigned to synset bn:03387773n.Next, we insert all matched synsets into an undirected graph G with nodes s ∈ S and edges e.We create edges between all synsets that match among each other with a lemma h ∈ H. Finally, we apply a community detection algorithm (Newman, 2006) on G to find dense clusters of synsets.We use these clusters as normalized topics and assign the sense with most outgoing edges as representative label, in our example e.g.therapy.
From this normalization step we obtain 598 synsets which we prune using the head/tail division rule count(s) < 1 |S| s i ∈S count(s i ) (Jiang, 2012).This method covers over 94% of all headings and yields 26 normalized labels and one other class in the English disease dataset.Table 1 shows the corresponding numbers for the other datasets.We verify our normalization process by manual inspection of 400 randomly chosen heading-label assignments by two independent judges and report an accuracy of 97.2% with an average observed inter-annotator agreement of 96.0%.

SECTOR Model
We introduce SECTOR, a neural embedding model that predicts a latent topic distribution for every position in a document.Based on the task described in Section 3, we aim to detect M sections T 0...M in a document D and assign topic labels y j = topic(S j ), where j = 1, . . ., M .Because we do not know the expected number of sections, we formulate the objective of our model on sentence level and later segment based on the predictions.Therefore, we assign each sentence s k a sentence topic label ȳk = topic(s k ), where k = 1, . . ., N .Thus, we aim to predict coherent sections with respect to document context: We approach two variations of this task: for WIKISECTION-topics, we choose a single topic label y j ∈ Y out of a small number of normalized topic labels.However, from this simplified classification task arises an entailment problem, because topics might be hierarchically structured.For example, a section with heading "Treatment | Gene Therapy" might describe genetics as a subtopic of treatment.Therefore, we also approach an extended task WIKISECTION-headings to capture ambiguity in a heading, We follow the CBOW approach (Mikolov et al., 2013) and assign all words in the heading z j ⊂ Z as multi-label bag over the original heading vocabulary.This turns our problem into a ranked retrieval task with a large number of ambiguous labels, similar to Prabhu and Varma (2014).It further eliminates the need for normalized topic labels.For both tasks, we aim to maximize the log likelihood of model parameters Θ on section and sentence level: Our SECTOR architecture consists of four stages shown in Figure 2: sentence encoding, topic embedding, topic classification and topic segmentation.We now discuss each stage in more detail.

Sentence Encoding
The first stage of our SECTOR model transforms each sentence s k from plain text into a fixedsize sentence vector x k which serves as input into the neural network layers.Following Hill et al. (2016), word order is not critical for documentcentric evaluation settings such as our WIKISEC-TION task.Therefore, we mainly focus on unsupervised compositional sentence representations.
Bag-of-words encoding.As a baseline, we compose sentence vectors using a weighted bagof-words scheme.Let I(w) ∈ {0, 1} |V| be the indicator vector, such that I(w) (i) = 1 iff w is the i-th word in the fixed vocabulary V, and let tf-idf(w) be the TF-IDF weight of w in the corpus.We define the sparse bag-of-words encoding x bow ∈ R |V| as follows: Bloom filter embedding.For large V and long documents, input matrices grow too large to fit into GPU memory, especially with larger batch sizes.Therefore we apply a compression technique for sparse sentence vectors based on Bloom filters (Serrà and Karatzoglou, 2017).A Bloom filter projects every item of a set onto a bit array A(i) ∈ {0, 1} m using k independent hash functions.We use the sum of bit arrays per word as compressed Bloom embedding x bloom ∈ N m : We set parameters to m = 4096 and k = 5 to achieve a compression factor of 0.2, which showed good performance in the original paper.
Sentence embeddings.We use the strategy of Arora et al. (2017) to generate a distributional sentence representation based on pre-trained word2vec embeddings (Mikolov et al., 2013).This method composes a sentence vector v emb ∈ R d for all sentences using a probability-weighted sum of word embeddings v w ∈ R d with α = 10 −4 , and subtracts the first principal component u of the embedding matrix [ v s : s ∈ S ]:

Topic Embedding
We model the second stage in our architecture to produce a dense distributional representation of latent topics for each sentence in the document.We use two layers of LSTM (Hochreiter and Schmidhuber, 1997) with forget gates (Gers et al., 2000) connected to read the document in forward and backward direction (Graves, 2012).We feed the LSTM outputs to a 'bottleneck' layer with tanh activation as topic embedding.Figure 3 shows these layers in context of the complete architecture.We can see that context from left (k − 1) and right (k + 1) affects forward and backward layers independently.It is therefore important to separate these weights in the embedding layer to precisely capture the difference between sentences at section boundaries.We modify our objective given in Eq. 2 accordingly with long-range dependencies from forward and backward layers of the LSTM: Note that we separate network parameters Θ and Θ for forward and backward directions of the LSTM, and tie the remaining parameters Θ for the embedding and output layers.This strategy couples the optimization of both directions into the same vector space without the need for an additional loss function.The embeddings e 1...N are calculated from the context-adjusted hidden states h k of the LSTM cells (here simplified as f LSTM ) through the bottleneck layer: Now, a simple concatenation of the embeddings e k = e k ⊕ e k can be used as topic vector by downstream applications.

Topic Classification
The third stage in our architecture is the output layer that decodes the class labels.To learn model parameters Θ required by the embedding, we need to optimize the full model for a training target.For the WIKISECTION-topics task, we use a simple one-hot encoding ȳ ∈ {0, 1} |Y| of the topic labels constructed in Section 3.3 with a softmax activation output layer.For the WIKISECTION-headings task, we encode each heading as lowercase bag-ofwords vector z ∈ {0, 1} |Z| , such that z(i) = 1 iff the i-th word in Z is contained in the heading, e.g.zk ={gene, therapy, treatment}.We then use a sigmoid activation function: Ranking loss for multi-label optimization.
The multi-label objective is to maximize the likelihood of every word that appears in a heading: For training this model, we use a variation of the logistic pairwise ranking loss function proposed by dos Santos et al. (2015).It learns to maximize the distance between positive and negative labels: We calculate the positive term of the loss by taking all scores of correct labels y + into account.We average over all correct scores to avoid a too strong positive push on the energy surface of the loss function (LeCun et al., 2006).For the negative term, we only take the most offending example y − among all incorrect class labels.
Here, s θ (x) (y) denotes the score of label y for input x.We follow the authors and set scaling factor γ = 2, margins m + = 2.5 and m − = 0.5.

Topic Segmentation
In the final stage, we leverage the information encoded in the topic embedding and output layers to segment the document and classify each section.
Baseline segmentation methods.As a simple baseline method, we use prior information from the text and split sections at newline characters (NL).Additionally, we merge two adjacent sections if they are assigned the same topic label after classification.If there is no newline information available in the text, we use a maximum label (max) approach: We first split sections at every sentence break, i.e. S j = s k ; j = k = 1, . . ., N and then merge all sections which share at least one label in the top-2 predictions.
Using deviation of topic embeddings for segmentation.All information required to classify each sentence in a document is contained in our dense topic embedding matrix E = [e 1 , . . ., e N ].
We are now interested in the vector space movement of this embedding over the sequence of sentences.Therefore, we apply a number of transformations adapted from Laplacian-of-Gaussian edge detection on images (Ziou and Tabbone, 1998) to obtain the magnitude of embedding deviation (emd) per sentence.First, we reduce the dimensionality of E to D dimensions using PCA, i.e. we solve E = U ΣW T using singular value decomposition and then project E on the D principal components E D = EW D .Next, we apply Gaussian smoothing to obtain a smoothed matrix E D by convolution with a Gaussian kernel with variance σ 2 .From the reduced and smoothed embedding vectors e 1...N we construct a sequence of deviations d 1...N by calculating the stepwise difference using cosine distance: Finally we apply the sequence d 1...N with parameters D = 16 and σ = 2.5 to locate the spots of fastest movement (see Figure 4), i.e. all k where d k−1 < d k > d k+1 ; k = 1 . . .N in our discrete case.We use these positions to start a new section.
Improving edge detection with bidirectional layers.We adopt the approach of Sehikh et al. (2017), who examine the difference between forward and backward layer of an LSTM for segmentation.However, our approach focuses on the difference of left and right topic context over time steps k, which allows for a sharper distinction between sections.Here, we obtain two smoothed embeddings e and e and define the bidirectional embedding deviation (bemd) as geometric mean of the forward and backward difference: After segmentation, we assign each segment the mean class distribution of all contained sentences: Finally, we show in the evaluation that our SEC-TOR model which was optimized for sentences ȳk can be applied to the WIKISECTION task to predict coherently labeled sections T j = S j , ŷj .We conduct three experiments to evaluate the segmentation and classification task introduced in Section 3. The WIKISECTION-topics experiment comprises segmentation and classification of each section with a single topic label out of a small number of clean labels (25-30 topics).The WIKI-SECTION-headings experiment extends the classification task to multi-label per section with a larger target vocabulary (1.0k-2.8kwords).This is important, because often there are no clean topic labels available for training or evaluation.Finally, we conduct a third experiment to see how SECTOR performs across existing segmentation datasets.
Evaluation datasets.For the first two experiments we use the WIKISECTION datasets introduced in Section 3.1, which contain documents about diseases and cities in both English and German.The subsections are retained with full granularity.For the third experiment, text segmentation results are often reported on artificial datasets (Choi, 2000).It was shown that this scenario is hardly applicable to topic-based segmentation (Koshorek et al., 2018), so we restrict our evaluation to real-world datasets that are publicly available.The Wiki-727k dataset by Koshorek et al. (2018) contains Wikipedia articles with a broad range topics and their top-level sections.However, it is too large to compare exhaustively, so we use the smaller Wiki-50 subset.We further use Cities and Elements datasets introduced by Chen et al. (2009), which also provide headings.These sets are typically used for word-level segmentation, so they don't contain any punctuation and are lowercased.Finally, we use the Clinical Textbook chapters introduced by Eisenstein and Barzilay (2008), which do not supply headings.
Classification models.We compare SECTOR to existing models for single and multi-label sentence classification.Because we are not aware of any existing method for combined segmentation and classification, we first compare all methods using given prior segmentation from newlines in the text (NL) and then additionally apply our own segmentation strategies for plain text input: maximum label (max), embedding deviation (emd) and bidirectional embedding deviation (bemd).
For the experiments, we train a Paragraph Vectors (PV) model (Le and Mikolov, 2014) using all sections of the training sets.We utilize this model for single-label topic classification (depicted as PV>T) by assigning the given topic labels as paragraph IDs.Multi-label classification is not possible with this model.We use the paragraph embedding for our own segmentation strategies.We set the layer size to 256, window size to 7 and trained for 10 epochs using a batch size of 512 sentences and a learning rate of 0.025.We further use an implementation of CNN (Kim, 2014) with our pretrained word vectors as input for single-label topics (CNN>T) and multi-label headings (CNN>H).We configured the models using the hyperparameters given in the paper and trained the model using a batch size of 256 sentences for 20 epochs with learning rate 0.01.SECTOR configurations.We evaluate the various configurations of our model discussed in prior sections.SEC>T depicts the single-label topic classification model which uses a softmax activation output layer, SEC>H is the multi-label variant with a larger output and sigmoid activations.Other options are: bag-of-words sentence encoding (+bow), bloom filter encoding (+bloom) and sentence embeddings (+emb); multi-class crossentropy loss (as default) and ranking loss (+rank).
We have chosen network hyperparameters using grid search on the en_disease validation set and keep them fixed over all evaluation runs.For all configurations, we set BLSTM layer size to 256, topic embeddings dimension to 128.Models are trained on the complete train splits with a batch size of 16 documents (reduced to 8 for bag-of-words), 0.01 learning rate, 0.5 dropout and ADAM optimization.We used early stopping after 10 epochs without MAP improvement on the validation data sets.We pre-trained word embeddings with 256 dimensions for the specific tasks using word2vec on lowercase English and German Wikipedia documents using a window size of 7. All tests are implemented in Deeplearning4j and run on a Tesla P100 GPU with 16GB memory.Training a SEC+bloom model on en_city takes roughly 5 hours, inference on CPU takes on average 0.36 seconds per document.In addition, we trained a SEC>H@fullwiki model with raw headings from a complete English Wikipedia dump 8 , and use this model for cross-dataset evaluation.
Quality measures.We measure text segmentation at sentence level using the probabilistic P k error score (Beeferman et al., 1999) which calculates the probability of a false boundary in a 8 excluding all documents contained in the test sets window of size k, lower numbers mean better segmentation.As relevant section boundaries we consider all section breaks where the topic label changes.We set k to half of the average segment length.We measure classification performance on section level by comparing the topic labels of all ground truth sections with predicted sections.We select the pairs by matching their positions using maximum boundary overlap.We report microaveraged F1 score for single-label or Precision@1 for multi-label classification.Additionally, we measure Mean Average Precision (MAP), which evaluates the average fraction of true labels ranked above a particular label (Tsoumakas et al., 2009).

Results
Table 3 shows the evaluation results of the WIKI-SECTION-topics single-label classification task, Table 4 contains the corresponding numbers for multi-label classification.Bloom filters on par with word embeddings.Bloom filter encoding achieves high scores among all datasets and outperforms our bag-of-words baseline, possibly because of larger training batch sizes and reduced model parameters.Surprisingly, word embeddings did not improve the model significantly.On average, German models gained 0.7 points F1 while English models declined by 0.4 points compared to bloom filters.However, model training and inference using pre-trained embeddings is faster by an average factor of 3.2.
Topic embeddings perform well on noisy data.In the multi-label setting with unprocessed Wikipedia headings, classification precision of SECTOR reaches up to 72.3% P@1 for 2.8k labels.This score is in average 9.5 points lower compared to the models trained on the small number of 25-30 normalized labels.Furthermore, segmentation performance is only missing 3.8 points P k compared to the topics task.Ranking loss could not improve our models significantly, but achieved better segmentation scores on the headings task.Finally, the cross-domain English fullwiki model performs only on baseline level for segmentation, but still achieves better classification performance than CNN on the English cities dataset.SECTOR captures latent topics from context.We clearly see from NL predictions (left side of Figure 5) that SECTOR produces coherent results with sentence granularity, with topics emerging and disappearing over the course of a document.In contrast, PV predictions are scattered across the  document.Both models successfully classify first (symptoms) and last sections (epidemiology).However, only SECTOR can capture diagnosis, prevention and treatment.Furthermore, we observe additional screening predictions in the center of the document.This section is actually labeled "Prevention | Screening" in the source document, which explains this overlap.Furthermore, we observe low confidence in the second section labeled cause.Our multiclass model predicts for this section {diagnosis, cause, genetics}.The ground truth heading for this section is "Causes | Genetic sequence", but even for a human reader this assignment is not clear.This shows that the multi-label approach fills an important gap and can even serve as an indicator for low-quality article structure.

Discussion and Model Insights
Finally, both models fail to segment the complication section near the end, because it consists of an enumeration.The embedding deviation segmentation strategy (right side of Figure 5) completely solves this issue for both models.Our SECTOR model is giving nearly perfect segmentation using the bidirectional strategy, it only misses the discussed part of cause and is off by one sentence for the start of prevention.Furthermore, averaging over sentence-level predictions reveals clearly distinguishable section class labels.

Conclusions and Future Work
We presented SECTOR, a novel model for coherent text segmentation and classification based on latent topics.We further contributed WIKISEC-TION, a collection of four large datasets in English and German for this task.Our end-to-end method builds on a neural topic embedding which is trained using Wikipedia headings to optimize a BLSTM classifier.We showed that our best performing model is based on sparse word features with bloom filter encoding and significantly improves classification precision for 25-30 topics on comprehensive documents by up to 29.5 points F1 compared to state-of-the-art sentence classifiers with baseline segmentation.We used the bidirectional deviation in our topic embedding to segment a document into coherent sections without additional training.Finally, our experiments showed that extending the task to multi-label classification of 2.8k ambiguous topic words still produces coherent results with 71.1% average precision.
We see an exciting future application of SEC-TOR as a building block to extract and retrieve topical passages from unlabeled corpora, such as medical research articles or technical papers.One possible task is WikiPassageQA (Cohen et al., 2018), a benchmark to retrieve passages as answers to non-factoid questions from long articles.

Figure 1 :
Figure 1: Overview of the WIKISECTION task: (1) The input is a plain text document D without structure information.(2) We assume the sentences s 1...N contain a coherent sequence of local topics e 1...N .(3) The task is to segment the document into coherent sections S 1...M and (4) to classify each section with a topic label y 1...M .

Figure 2 :
Figure 2: Training and inference phase of segmentation and topic classification (SECTOR).For training (A), we preprocess Wikipedia documents to supply a ground truth for segmentation T, headings Z and topic labels Y.During inference (B), we invoke SECTOR with unseen plain text to predict topic embeddings e k on sentence level.The embeddings are used to segment the document and classify headings ẑj and normalized topic labels ŷj .

Figure 3 :
Figure 3: Neural network architecture SECTOR.The recurrent model consists of stacked LSTM, embedding and output layers that are optimized on document level and later accessed during inference in stages 1-4.

Figure 4 :
Figure 4: Embedding deviations emd k and bemd k of the smoothed SECTOR topic embeddings for example document Trichomoniasis.The plot shows the first derivative of vector movement over sentences k = 1, . . .N from left to right.Predicted segmentation is shown as black lines, the axis labels indicate ground truth segmentation.

Figure 5
Figure 5 shows classification and segmentation of our SECTOR model compared to the PV baseline.

Figure 5 :
Figure5: Heatmaps of predicted topic labels ŷk for document Trichomoniasis from PV and SECTOR models with newline and embedding segmentation.Shading denotes probability for 10 out of 27 selected topic classes on Y axis, with sentences from left to right.Segmentation is shown as black lines, X axis shows expected gold labels.Note that segments with same class assignments are merged in both predictions and gold standard ('. . .').
approached to classify sections of BioMed

Table 2 :
Frequency and entropy (H) of top-3 head and randomly selected torso and tail headings for category diseases in the English Wikipedia.

Table 3 :
Results for topic segmentation and single-label classification on four WIKISECTION datasets.n = 718 / 464 / 3, 907 / 2, 507 documents.Numbers are given as P k on sentence level, micro-averaged F 1 and MAP at segment-level.For methods without segmentation, we used newlines as segment boundaries (NL) and merged sections of same classes after prediction.Models marked with * are based on pre-trained distributional embeddings.

Table 4 :
Results for segmentation and multi-label classification trained with raw Wikipedia headings.Here, the task is to segment the document and predict multi-word topics from a large ambiguous target vocabulary.

Table 5 :
Results for cross-dataset evaluation on existing datasets.Numbers marked with * are generated by models trained specifically for this dataset.A value of 'n/a' indicates that a model is not applicable to this problem.
Table 5 shows results for topic segmentation across different datasets.