Usage-based Grammar Induction from Minimal Cognitive Principles

Human language capacity stands out among species for its flexibility and expressive power. A fundamental unsolved question is what cognitive mechanisms enable humans to learn language. On the one side, there are theories on inborn language organization (Chomsky 1957; Pinker and Jackendoff 2005; Fodor 1983) or language specific learning processes (Nowak, Komarova, and Niyogi 2002; Reali and Griffiths 2009; Griffiths et al. 2010; Tenenbaum et al. 2011). On the other side, we find theories emphasizing usage-based, general learning principles (Bybee 1985; Tomasello 2003; Heyes 2018), combined with ideas on culturally emergent structure (Kirby, Cornish, and Smith 2008; Goldberg 2007; Langacker 1987; Croft 2001). Explanations relying on strong functional modularity often fail to explain variability across languages and disregard that human mental skills require social learning. Moreover, recent advancements in large language models (LLMs) challenge the notion that predetermined linguistic predispositions are necessary for language learning (Piantadosi 2023). In contrast, domain-general learning theories often lack explicit suggestions on the machinery that would enable such learning and leave the question of why other animals cannot acquire human-like language unanswered. In this article, our point of departure is the novel sequence hypothesis, postulating that faithful sequence representation is uniquely human and a central prerequisite for the emergence of language and complex culture (Ghirlanda, Lind, and Enquist 2017; Lind, Ghirlanda, and Enquist 2022; Enquist, Ghirlanda, and Lind 2023; Lind et al. 2023; Jon-And et al. 2023). Based on this, we propose a minimal cognitive architecture with a short and flexible sequence memory as its most central component and examine how far this design can reach in learning grammar from data.

Our proposed cognitive architecture follows a simple and traceable error-correction temporal difference learning algorithm (Sutton and Barto 2018). In order to make the model fully interpretable, we use an equation-based model rather than a neural network. Our minimal architecture allows us, on the one hand, to discuss what cognitive properties are necessary for language learning and, on the other hand, to follow the process of extracting grammatical information that supports the language learning process. Our approach is inspired by cognitive architecture research in artificial intelligence, that aims at building a unified theory of cognition where few general mechanisms should be able to explain diverse phenomena and characteristics of human cognition (Newell 1994). In line with recent domain-general artificial intelligence modeling (Wiggins 2020; van der Velde et al. 2017), we aim at accounting for human information processing while making as few assumptions as possible. Our initial hypothesis, drawing upon a pilot study for the present work (Jon-And and Michaud 2020), is that sequence memory, paired with chunking, are key domain-general cognitive mechanisms underlying the human ability to learn grammar from data. Our approach implements principles of usage-based language learning, namely, that language acquisition occurs through language use and through continuous updating of linguistic knowledge encoded as chunks or constructions (Bybee 2006; Tomasello 2003).

We explore reinforcement learning as a foundational framework for our learning model, seeking a cognitively plausible architecture and learning process, that can generate language structure. Reinforcement learning is grounded in robust empirical support for organisms learning through trial-and-error interactions with their environment (Pavlov 1949; Skinner 1965; Mackintosh 1983; Rescorla and Wagner 1972; Sutton and Barto 2018). Our objective is also to use simple, localized learning principles to model the acquisition of a complex system like grammar. The use of reinforcement learning aligns well with this aim, as reinforcement learning is suggested to be a sufficient mechanism to acquire many sophisticated and multifaceted behaviors that are found in humans, animals, and machines (Silver et al. 2021).

The task of our learning model is to identify sentences in a linguistic stream where cues to sentence borders like punctuation or capitalization are removed. In our model, the learner follows a chunk-and-pass learning principle (Christiansen and Chater 2016b) in which it decides whether to place a boundary or to chunk a newly encountered word to other words kept in the working memory. This is comparable to the shift-and-reduce procedure applied by Yogatama et al. (2016), where tree structures emerge and are optimized to improve performance on more advanced linguistic tasks like sentiment analysis and semantic judgments. In Yogatama et al. (2016), this is done by using word embeddings that emerge in a neural network, making both the tasks and the model considerably more complex than ours. In contrast, segmentation is a task that is present in early language learning, when structure is likely established (Peters 2013). Our chunking mechanism creates hierarchies with binary tree structures with the words as leaves. Since chunks retain the order in which words are encountered, they encode both the ability for sequence representation and chunking. Boundary placement triggers reinforcement, where correct sentence identification is rewarded and incorrect sentence identification is penalized. When referring to reward or penalization during language learning, we do not assume explicit feedback, that is often absent in language learning. Our assumption, rather, is that ultimately, successful communication triggers internal or external reinforcement, being the main goal of language use. Language understanding is a central part of successful communication, and the segmentation of a linguistic stream into meaningful units is in its turn necessary for understanding. While studies on segmentation in language learning have focused on the word level (Saffran, Aslin, and Newport 1996; Saffran 2001), higher levels of segmentation are also necessary for language understanding. Segmentation tasks are arguably more central for language learning than, for example, word or sound prediction tasks.

We choose the task of segmenting a stream of words into sentences because we are interested in grammar induction. We hypothesize that in a minimalist learning model that makes no assumptions on specific genetic guidance for language learning, and imposes cognitively plausible constraints on working memory and processing capacities, grammar emerges as a solution to the combinatorial challenge posed by language learning. The expectation is that hierarchical organization of words will emerge as a support for the task of sentence identification. There are, naturally, other tasks in language processing and production that are supported by grammar and may trigger its emergence. In this study, we have chosen one easily encoded and evaluated task, and test if this task is a sufficient driving force for grammar induction. This task may in future studies be combined with or compared to other tasks. We focus on syntax even though syntax and morphology are closely intertwined, for reasons of simplicity and for facilitating future testing of the model on written natural language corpora. Furthermore, while sentence boundaries are nowhere near as clear and consequent in spoken spontaneous language as in written texts, the identification of multi-word meaningful units is always a relevant task in language understanding. Here, for reasons of simplicity, we model sentences as a meaningful unit to be identified, to enable evaluation of the fundamental principles of our model.

Our model does not formally encode meaning, even though meaning is central to the concept of a linguistic construction, that is essentially a pairing of form and meaning (Langacker 1987; Goldberg 2007; Croft and Cruse 2004; Tomasello 2005). The notion of meaning is not, however, entirely absent from our model, as an underlying assumption is that identification of meaningful units is a prerequisite for a form-meaning mapping to occur. But the support for segmentation provided by meaning in natural language learning, as well as prosodic cues like stress or pauses (Kuhl 2004; Sanders, Neville, and Woldorff 2002) are absent in our model. These simplifications may impose difficulties for our learner to succeed in its task when linguistic complexity increases, but they also enable investigating how far the system can reach without this additional support.

Our work bears similarities to unsupervised grammar induction, which involves inferring a grammar from strings or sentences within a language. However, differently from our architecture, most unsupervised grammar induction models assume the existence of given formal syntactic relations even if there is no explicit training on how these should be encoded (Muralidaran, Spasić, and Knight 2021). While exceptions to this top–down perspective exist (Bod 2006; Solan et al. 2003; Shain et al. 2016), our goals also depart from those of unsupervised grammar induction in the sense that we are not searching for the model that best predict data. Instead, we are aiming at modeling the acquisition of grammar in a minimalist and cognitively plausible manner that is consistent with usage-based learning, and at testing whether sequence memory is sufficient to achieve this. In syntactic parsing the explicit task is to identify trees, dependencies, and/or constituents in sentences, while the task of our learner is to identify sentence boundaries, a goal more directly related to language understanding, and trees emerge as a support for this task.

The evaluation of the model is based on the one hand on its ability to fully learn to identify sentences in artificial languages with varying degrees of complexity, and on the other hand on its ability to identify and reuse chunks that are extracted from exposure to these languages. The second more qualitative measure is more central than the model’s quantitative performance, as our goal at this stage is primarily to study whether and how our design allows for extraction of functional structure. In order to understand how structure emerges during learning in our model, and to discuss whether this process in any aspects resembles natural language learning, we study the learning process and not only the final result. The evaluation is performed on relatively small artificial languages reflecting common natural language structures. To scale to larger input, the model needs a system that allows for abstraction, and currently it only contains a first step towards generalization, as the reuse of frequent chunks, partially reduces the combinatorial explosion of language learning. This partial solution can serve as a basis for generalization that may give rise to grammatical abstractions at higher levels. Such generalization could be implemented in, for example, a neural network, but if the transparency of the model is to be maintained, generalizations can be implemented within the error-correction temporal difference learning algorithm applied here. One way of doing this is using a category-based formalism that makes no pre-assumptions on categories, but only generalizes over the properties of the useful chunks the learner identifies. A pilot implementation of such a generalization system has been conducted by Jon-And and Michaud (2024).

As the system presented here does not include large-scale generalization, we do not evaluate the model’s performance on natural languages. In artificial languages, we find that the model is able to efficiently identify informative chunks and reuse them when they occur in new structures, thus in a simplified manner accounting for the emergence of grammar during learning. We also find that the model over time learns to ignore less informative chunks and gives priority to those that contribute more frequently to the correct identification of sentences. In this way, we see how a simple goal like identification of sentence borders gives rise to self-organization and consolidation in an emergent grammatical system.

This article is organized as follows: Section 2 provides some background for this study. Section 3 presents the computational framework used, that is, the framework of Markov Decision Processes (Sutton and Barto 2018). In particular, we discuss the learning task as well as the learning algorithms studied in this article. Section 4 presents the results of our study and discusses the performance of our model on a number of small human-like toy languages. We show that our model successfully learns these languages and provide some interesting insights into how learning is performed. Finally, Section 5 provides a discussion of the implication of our results and outlines how this work could be continued.

Associative learning is a fundamental mechanism that holds strong explanatory power for general learning in both humans and other animals (Pavlov 1949; Skinner 1965; Mackintosh 1983; Rescorla and Wagner 1972; Heyes 2018; Wasserman, Kain, and O’Donoghue 2023; Lind 2018; Heyes 2012b, 2012a; Enquist, Lind, and Ghirlanda 2016; Bouton 2016; Haselgrove 2016; Enquist, Ghirlanda, and Lind 2023). However, the inability of non-human animals to match the language capacities of humans calls for identifying unique properties that are present in humans, and not in other animals. Sequence learning has been pointed out as particularly important for human linguistic capacities (Heyes 2018; Christiansen and MacDonald 2009; Bybee 2002a; Christiansen and Kirby 2003; Christiansen and Arnon 2017; Frank, Bod, and Christiansen 2012; Cornish et al. 2017; Udden et al. 2012; Kolodny, Edelman, and Lotem 2015; Kolodny and Edelman 2018). At the same time, there is strong empirical support for non-human animals’ limited capacity to represent the exact order of stimuli (Roberts and Grant 1976; MacDonald 1993; Ghirlanda, Lind, and Enquist 2017; Read, Manrique, and Walker 2022; Lind et al. 2023). This suggests faithful sequence representation to be a basic defining feature of human cognition and linguistic abilities (Enquist, Ghirlanda, and Lind 2023; Jon-And et al. 2023). These findings call for testing the capacity for language learning of a minimalist associative learning model with a precise but limited sequence memory.

To enable processing of language or of any kind of sequential information, sequence memory likely needs to be combined with a capacity for chunking, that is, considering a recurrent sequence of stimuli of flexible length as a unit. Without chunking, it is impossible to faithfully represent a sequence. Chunking is known to be central for human language learning (Tomasello 2003; Bybee 2002a; Servan-Schreiber and Anderson 1990; McCauley and Christiansen 2019). Humans have strong memory constraints in online language processing and are, for example, typically only able to store somewhere between three and ten meaningful items in their working memory (Cowan 2001; Miller 1956). The chunk-and-pass principle has been identified as essential for overcoming such memory constraints (Christiansen and Chater 2016b), and is also theorized to underpin the multilevel representational structure of language (Christiansen and Chater 2016a).

Principles similar to associative learning, of local decision-making and memory updating, rooted in the utilization of limited recent information and associations derived from past experiences, have been implemented in models like the naming game, where categorizations self-organize driven by communicative success and alignment (Steels and Loetzsch 2012), or the utterance selection model, that accounts well for several aspects of the change and diffusion of linguistic forms (Baxter et al. 2006; Michaud 2019). In these models, only knowledge pertaining to the features present in each specific learning occasion is updated, thus avoiding costly computations. These models focus on single linguistic elements and not sequences, and cannot, therefore, infer any system that encodes relationships between elements, like grammar. We hypothesize that by applying similar principles to sequential linguistic input, a global system of rules may emerge during learning and use without the learner having to possess explicit knowledge or hypothesize about the entire system. Instead, agents will rely upon simple emergent dynamic rules to inform each decision locally.

In connectionist models, associative learning is operationalized through the adjustment of connection weights between artificial neurons. Such models, implemented in artificial neural networks, mimic the interconnected structure of neurons in the brain and tend to exhibit high performance in generalization of patterns from data. While connectionist models have successfully accounted for certain aspects of language acquisition (Elman 1990, 1996; Christiansen and Chater 2001; McClelland et al. 2010), their interpretability remains a challenge, and they struggle to capture symbolic representations.

Modern LLMs are connectionist models that stand out for their remarkable and often human-like linguistic performance. It is possible to gain some insight into what kind of information LLMs are able to store at the semantic, syntactic, and morphological levels by studying their attentional mechanisms (Manning et al. 2020; Vaswani et al. 2017; Piantadosi 2023; Piantasodi and Hill 2022), but the neural networks upon which LLMs rely contain billions to trillions of parameters (Piantadosi 2023; Brown et al. 2020; Amaratunga 2023) and it is not possible to extract precise information on whether or how any morpho-syntactic abstractions emerge in these models and, if so, what is their nature. In terms of cognitive plausiblity, there are also some divergences between the learning processes of LLMs and humans. LLMs typically leverage datasets approximately 1,000 times larger than the linguistic input available to a child (Warstadt and Bowman 2022). Moreover, humans learn and use language in parallel (Bybee 2006; Ellis, O’Donnell, and Römer 2015; Tomasello 2003), while LLMs are generally pre-trained and no longer learn when they are put in use. Both during training and use, LLMs can base their output on very long input sequences, sometimes containing thousands of tokens (Beltagy, Peters, and Cohan 2020; Liang et al. 2023; Brown et al. 2020), while human working memory is constrained to storing the aforementioned three to ten items (Cowan 2001; Miller 1956). Overall, LLMs have access to more resources in terms of both data, memory and processing power than humans during language learning. We hypothesize that imposing working memory and processing constraints in a learning model may trigger the emergence of grammatical structure as a way to overcome combinatorial problems inherent to language acquisition. It has previously been shown that limited working memory capacity may facilitate grammar induction (Shain et al. 2016). Possibly, models that are able to represent and process very long strings, paired with training on massive data, can achieve what humans judge as grammatically correct output, without establishing a system of grammatical categories and relations.

The training task of LLMs and many other connectionist language learning models is to predict words. In natural language learning, sound or word prediction undoubtedly provides support for efficient language processing, but it is theoretically possible to learn a language without predicting upcoming elements. On the other hand, the identification of units that can be mapped to meanings, is a step that needs to be taken to achieve both understanding and production. While segmentation is not the only or necessarily the most important task in language learning, it is a task that is relatively easy to implement in a model, as cues to rewards are present in text, and it may bring the modeled process closer to a real life learning process than word prediction.

Automatic syntactic parsers superficially share our goal of extracting grammatical structure from linguistic input. These models are, however, often not built with the aim of representing cognitively plausible learning processes. Supervised parsing depends on predefined syntactic rules and/or categories, human labeling, and training (Manning and Schutze 1999; Sanford and Sturt 2002; Nivre et al. 2020) and does in that sense not resemble informal language learning.

In contrast, unsupervised grammar induction aims at inferring grammatical structure from linguistic data without any explicit training on how to encode syntactic relationships. Many unsupervised grammar induction models still assume the existence of given formal syntactic relations, providing the model with initial information on categories related to grammatical dependencies or constituents (Muralidaran, Spasić, and Knight 2021). This top–down perspective is not compatible with domain-general learning. Unsupervised data-oriented parsing (DOP) models are more compatible with a usage-based perspective, as they apply a bottom–up methodology with no pre-defined categories. In this approach, the most probable binary trees are computed from the shortest derivations of sentences (Bod 2006, 2009; Post and Gildea 2013). Another approach that makes relatively few cognitive assumptions is the unsupervised hierarchical hidden Markov models (UHHMM) presented by Shain et al. (2016), which learns hierachical syntax statistically. While DOP and UHHMM demonstrate that grammar can be induced without relying on predefined categories, one drawback of these models lies in their need for simultaneous access to large amounts of data for probability computations. This requirement renders them both cognitively implausible and computationally costly. Another theory-neutral approach is the automatic distillation of structure (ADIOS), a model that extracts statistical patterns incrementally from a corpus of sentences and outputs a directed multigraph, whose patterns can be represented as a context-free grammar (Solan et al. 2003; Berant et al. 2007; Brodsky and Waterfall 2007). Differently from DOP and UHHMM, ADIOS carries out structural inferences locally. However, before extracting statistical patterns, pre-processing of the data is made where the corpus is loaded to a pseudograph with lexical entries as vertices. The local inference is thus not made on a raw stream of linguistic input.

Another paradigm of automatic grammar induction involves assessing hypothetical grammars of formal languages via Bayesian inference (Rule et al. 2018; Rule, Tenenbaum, and Piantadosi 2020; Piantadosi, Tenenbaum, and Goodman 2016; Amalric et al. 2017; Planton et al. 2021; Ullman, Goodman, and Tenenbaum 2012; Ellis et al. 2018). The formal languages are composed of sequences generated by rule systems known as artificial grammars organized within a mathematical hierarchy of escalating generative power, commonly referred to as the Chomsky hierarchy (Chomsky 2002, 1959). The Bayesian approach to formal language learning assumes the existence of a prior that is evaluated against data. This means that the learner needs to formulate and reformulate hypotheses about the entire system that is being learned, overlooking the possibility that grammatical rules may be inferred without a prior framework or without formulating hypotheses about the entire system. Furthermore, many formal languages in such studies do not resemble natural languages. Context-free languages like aⁿbⁿ, for example, are even impossible for humans to learn without explicit counting if n is high. Results from Hochmann, Azadpour, and Mehler (2008) suggest that human recognition of such strings relies on distributional regularities rather than on hypotheses about underlying grammar.

We propose a perspective that differs in some crucial aspects from all grammar induction models reviewed here. First, we initially present the learner with a stream of words without cues of sentence borders, whereas other grammar induction studies present sentences one by one. Presenting an unsegmented stream makes the input more similar to that of an early language learner and, at the same time, allows for using sentence borders as cues for reinforcement. Secondly, we apply local decision-making and updating of probabilities of actions, and never let the learner have access to more data than the current state and values of actions associated with it. This is not only computationally cheap, but also an empirically grounded approach to learning. Third and lastly, reinforcement learning allows for structures to be optimized for a task related to language learning instead of being identified purely on the basis of statistical regularities. We are primarily interested in exploring this new approach as a way of understanding how humans induce grammar from data, and not necessarily in optimizing performance in parsing. For this reason we do not compare our model’s performance with other grammar induction models at this stage, but rather study its learning process and evaluate its potential for understanding the cognitive prerequisites underlying human language learning.

The method used in this article is that of reinforcement learning applied to a Markov Decision Process (MDP) (Sutton and Barto 2018), that encodes a language learning task. The learner tries to identify sentences by placing boundaries between words and gets positive reinforcement when it succeeds and negative reinforcement when it fails. The model will then be evaluated by looking at learning curves, that is, the fraction of correct responses from a population of agents, and at the grammar induced by the model. This section provides an overview of the learning task, learning algorithms, and evaluation methodology.

We start by defining the learning task our cognitive architecture will use. Following Sutton and Barto (2018), we define the task as an MDP. MDPs provide a useful framework for studying learning from interactions to achieve a goal. In this framework, a learner is called an agent. In our case, the agent aims to identify sentences in a sequence of words constituting the environment. The sequence of words used here is generated using a probabilistic context free grammar (PCFG), but our model should work on any sequential input with well-defined units. To illustrate the type of input the model receives, it is useful to take a natural language example. Consider, for example, the following three sentences:

Because identifying sentences in this context is too easy due to punctuation and capitalization, we create a raw input by removing any hint as to the beginning of sentences—for example, capital letters and punctuation are removed from the list of words—yielding the following modified input:

The information about sentence boundaries is stored in a Boolean sequence encoding whether a word in the raw input is preceded by a boundary or not. This Boolean sequence is used to drive the reinforcement procedure. The aim of the learning agent is then to identify sentences, that is, sequences of words separated by boundaries and not containing any other boundaries. This task can be seen as a masking task, where sentence boundaries are masked and the aim of the model is to predict where boundaries occur.

In order to specify this task within the MDP framework, we need to specify

1. how agents represent information (i.e., their state space);

2. what they can do in a given state (i.e., their action spaces, which depends on their state), and

3. what consequences their actions have (i.e., the reward structure and transitions between states).

More specifically, the agent interacts with its environment at discrete time steps, t = 0,1,2,3,…. At each time step t, the agent is in a given state $St∈S$ and takes an action $At∈A(St)$⁠. As a result of that action, the agent receives a reward $Rt+1∈R⊂R$ and finds itself in a new state $St+1∈S$⁠. To encode the behavior of the agent, we use state-action values Q(s, a) encoding the value of taking action $a∈A(s)$⁠, when the agent finds herself in state s and define the policy of an agent π(a|s) as the probability to select action $a∈A(s)$ when in state s.

When a chunking action is selected, the resulting chunk becomes the first element of S_{t +1} and the next word is read. In this case, no reward is given.

Upon boundary placement, reinforcement is triggered. Whenever the first element of S_t is a correct sentence, a positive reward of R_{t +1} = r ₊ > 0 is given. Otherwise, a negative reward of R_{t +1} = r₋ < 0 is given. The next section describes how the rewards are used to update state-action values and the policy of the agents. After reinforcement, we considered two different ways of reinitializing the task:

• Continuous.

  The second element of S_t becomes the first element of S_{t +1} and the next word is read to instantiate the second element.

• Next sentence.

  The first element of S_{t +1} is set to the first word of the next sentence in the sequence of words, and the second element is the following word.

These types of reinitialization processes make the MDP episodic in the sense that the process terminates upon boundary placement and restarts in a new state. Thus, each attempt to identify a sentence constitutes a new episode. Note that we are mainly interested in the Continuous condition, since this is the only condition where the two borders need to be identified. The Next sentence condition corresponds to a simpler task where only the second boundary needs to be identified. We chose to study the Next sentence condition as an intermediary condition between the Continuous condition and the situation where the two borders are known, common in other grammar induction models.

During learning, an agent must choose between actions in order to increase its expected reward. In this article, we use a variant of the Q-learning algorithm (Sutton and Barto 2018), in which state-action values are updated according to a temporal difference algorithm. These state-action values can be considered as estimators of the expected reward from that specific state and learning is driven by reducing the error in prediction. In our case, states possess a hierarchic structure whenever d(S_t) > 1 and it is possible to modify the standard Q-learning algorithm to take advantage of this structure.

To take advantage of this structure, we define two composite versions of the state-action values:

1. The average composite state-action value$Q¯(s,ai)$ is defined as:

   $Q¯(s,ai)≔∑k=0d(s)−1Q(sk,ai−kk)H(i−k)∑k=0d(s)−1H(i−k),i=0,…,d(s)$
2. The additive composite state-action value$Q~(s,ai)$ is defined as:

   $Q~(s,ai)≔∑k=0d(s)−1Q(sk,ai−kk)H(i−k),i=0,…,d(s)$

The average composite state-action value $Q¯(s,ai)$ averages the state-action values of a state and its sub-states. The Heaviside function selects the actions that are valid for the sub-states. This quantity will be used in decision-making. The additive composite state-action value $Q~(s,ai)$ computes the sum of state-action values from a state and its associated sub-states. This quantity will be used in the learning mechanism to implement blocking in a similar way as Rescorla and Wagner (1972).

We are now in a position to explain how state-action values are updated. We consider two alternative learning rules:

• Q-learning
  In this case, state-action values associated with states and sub-states are updated independently. The updating rule is given by

  $Q(Stk,ai−kk)←Q(Stk,ai−kk)+αRT−Q(Stk,ai−kk)H(i−k)$
• Rescorla-Wagner Q-learning
  In this case, state-action values associated with states and sub-states are considered elements of a composite stimulus. The updating rule is given by

  $Q(Stk,ai−kk)←Q(Stk,ai−kk)+αRT−Q~(St,ai)H(i−k)$

In both cases, R_T is the reward obtained upon boundary placement and α is a learning parameter. The update is performed for all time steps t ∈ 0,…, T of the episode and for all sub-states k ∈ 0,…, d(S_t). These temporal difference algorithms are driven by the error in prediction of the reward R_T. The Heaviside function is there to avoid updating an action that does not exist in a given sub-state. The difference between Q-learning and Rescorla-Wagner Q-learning is that in the first case, the state-action values are used as predictors, whereas in the latter case, we use the additive composite state-action values.

To evaluate the performance of the model, we need to assess whether it successfully learns the input languages and, if so, how quickly. In addition, we are interested in what grammatical information it extracts and to study the process of grammar induction. To achieve these goals, we use learning curves to test the ability of the model to learn and quantify its speed. Grammatical information is extracted directly from the state-action values at different stages of the learning process and tested at the same time points.

In order to get a learning curve, we simulate N agents and record for each episode of the MDP whether the decision was correct or incorrect. We then compute the fraction of agents being successful at each episode and plot the fraction of correct responses as a function of the number of episodes of the MDP that we refer to as trials. If all agents eventually learn the language perfectly, then the fraction of correct responses should go to one.

Since we are interested in how learning unfolds, we need to test the performance of the model at various stages of the learning process. To this end, we use a test sequence consisting of n_test sentences produced with the same PCFG as the word sequences used for learning. Due to the small size of the languages studied, and the lack of strong generalization, sentences in the test sequence overlap with the sentences used in the training phase. We do not consider this to be problematic, since the beginning and the end of these sentences must be correctly identified for the system to perform well on the test sequence.

To evaluate how learning unfolds, we pause learning at different times of the learning process and record the performance of the model on the test sequence without learning. The learning phase then continues until the next snapshot time is reached. We measure the fraction of correct sentence identification and qualitatively discuss the tree structures used by the model for these sentences. In particular, we discuss the distribution of tree structures associated with a given sentence structure and the changes of that distribution over time.

In our model, all grammatical information is encoded in the state-action values of states and their associated sub-states, namely, in the Q values and the $Q¯$ values. In order to extract grammatical information, we need to extract the most likely action to be taken in each state. This information is encoded in the maximum value of $Q¯$⁠. Furthermore, the states used by the model are lexicalized, that is, individual words are not grouped into grammatical categories. However, we can group states sharing the same structure according to the underlying PCFG to display the results. As a proxy for the learned information, we choose a threshold 0 < t_G < r ₊ and extract all state-action values greater than t_G and report averages of Q and $Q¯$ over lexicalized states sharing the same structure. Displaying both Q and $Q¯$ is informative because if $Q>Q¯$⁠, then the larger structure controls the decision, whereas when $Q<Q¯$ the decision is controlled by sub-states. Of course, state-action with lower values also constitute grammatical information, but a threshold is necessary to select data to display. For each state (i.e., for each pair of chunks) we then have the most likely action to be taken and its associated values. This representation of grammatical information does not constitute a standard grammar, but is instructive for understanding how an agent learns. The extracted grammatical information is then saved in an Microsoft Excel spreadsheet. The code of this project as well as all Excel files used in this project are available online in the GitHub repository: https://github.com/michaudj/LanguageLearner/.

Because one of the aims of this article is to provide a transparent model of language acquisition, we use this technique of extracting grammatical information at various stages of the learning process and save them as different sheets in an Excel file. This allows us to look at how the grammatical information of an agent changes over time and whether it converges to a systematic representation.

For more complex languages in which the number of relevant state-action values is too high to be displayed, we will provide relevant examples and a more qualitative discussion of the learned grammatical information.

In this section, we present the results of our experiments on various artificial languages. We start by considering a simple language consisting of nouns (N) and verbs (V) organized in sentences with the structure noun-verb-noun (NVN), and then move on to consider more complex grammars. The NVN language is agnostic to the subject and object status of the nouns and may thus represent sentences occurring in English as well as many other languages. In the more complex grammars we use English as a model language but focus on phenomena that are typical of many languages with diverse typological profiles, like recursion and variable multi-word constituents.

Our baseline case has been chosen to illustrate how the agent learns the language and will be used to motivate the choice of learning algorithm used for more complex languages. We start by considering a simple NVN language. In this language, word transitions are enough to identify sentences, which, among other things, allows for testing whether the model is able to identify the most economic strategy of learning.

Our language is generated by PCFG $G=(M,T,R,S,P)$⁠, where $M$ is the set of non-terminal symbols, $T$ is the set of terminal symbols, $R$ is the set of production rules, $S$ is the start symbol, and $P$ is the set of probabilities on production rules.

With this NVN language defined, we test the ability of our models to learn this language using the proposed task. We consider the four cases combining the definition of Sections 3.1.2 and 3.2.2:

• QC.

  Q-learning with continuous border condition;

• QN.

  Q-learning with next sentence border condition;

• RWQC.

  Rescorla-Wagner Q-learning with continuous border condition;

• RWQN.

  Rescorla-Wagner Q-learning with next sentence condition.

The obtained learning curves are displayed in Figure 2. We see that all variants of the model successfully learn the language reaching a fraction of correct responses very close to 1. Using the fit to the logistic curve in Equation (2), we estimated the learning times of the four models in Table 2.

We observe that when the next sentence condition is used, learning is faster. This is not surprising, since the task is easier because only one of the two boundaries of a sentence has to be identified. With respect to learning times, there is very little difference between Q-learning and Rescorla-Wagner Q-learning, which is expected since the two learning algorithms are very similar. The main difference between these two algorithms is the information used to drive error-correction, namely, whether blocking is used or not. This will be examined below when we look at what is learned at different stages of the learning process.

As we have seen above, the performance of Q-learning and Rescorla-Wagner Q-learning are very similar in terms of learning curve. The question addressed in this section is what type of grammatical information is extracted and used by these two learning algorithms and whether the sentence condition affects what is learned. To achieve this, we report the grammatical information extracted in terms of the two elements of a state/sub-state, the action with the highest state-action value and the corresponding average composite state-action value. The results are shown in Tables 4 and 5 for the continuous border condition and are extracted from the two Excel spreadsheets QLearnerC2_revisedFinal.xlsx and RWQLearnerC2_revisedFinal.xlsx available in the GitHub repository: https://github.com/michaudj/LanguageLearner/. Snapshots are taken at the beginning of the S-shaped learning curve, when learning fraction is close to 0.5, when learning is just completed, and at two later points, when performances are very high. The columns labeled # refer to the number of lexicalized states (s₁, s₂) matching the pattern. For example, two states ([n₂, v₃], n₄) and ([n₁, v₅], n₂) have the same structure and constitute two instances of the ([N, V ], N) state. The results for the next sentence condition are similar and can be found in Excel spreadsheets QLearnerN2_revisedFinal.xlsx and RWQLearnerN2_revisedFinal.xlsx in the GitHub repository.

Our results show that the Rescorla-Wagner algorithm is much more parsimonious in the number of state-action values it relies upon. This is best seen by looking at the number of instances of a given state-action pair reported in column “#” of Tables 4 and 5. While these numbers steadily increase in Q-learning, only some of them increase in Rescorla-Wagner Q-learning and some of them even decrease after some time. For example, in the last three rows of Table 5. This shows that the model learns more specific information than the Q-learning algorithm. The same can be observed in the averaged state-action values, where all actions reported an increase in value for the Q-learning algorithm, while only some of them increase for Rescorla-Wagner Q-learning and some even decrease after some point. In addition, the effect of blocking is clearly visible when looking at the last three rows, since the value of the average composite state-action $Q¯$ does not decrease, showing that the sub-state are contributing more to the decision than the state itself.

The results for Q-learning suggest that all state-action pairs will eventually reach the maximum reinforcement value of 25 and the rule will not specify a preferable tree structure over the sentence. For instance, both [N,[V, N]] and [[N, V ], N] tree structure will occur with significant probability. This grammatical output is not very informative. In order to test this hypothesis, we conduct a performance test at the same snapshot times as for the Q values reported in Tables 4 and 5 using a test sequence of length n_test = 100. The results are reported in Table 3. We see that in the Rescorla-Wagner Q-learning, the number of sentences with the tree structure [N,[V, N]] is larger than for Q-learning. In addition, the next sentence condition produces a large bias towards that same structure. This is likely due to the fact that this specific structure has a higher right-depth and recruits information from more sub-states in the decision-making process. As argued above, we are providing results for the next sentence condition mainly for comparison, since we are interested in the full segmentation task. Together, these results suggest that the Rescorla-Wagner version of the algorithm is a better choice.

The results for Rescorla-Wagner Q-learning suggest that if learning continues only 4 state-action pairs are relevant for the decision-making process:

1. Place border between N and N;

2. Chunk together N and V;

3. Chunk together V and N;

4. When [N, V ] is followed by N, chunk at the root in 43/125 cases, yielding [[N, V ], N] and chunking deep otherwise, yielding [N,[V, N]], in good agreement with the test performances provided in Table 3.

This means that all information about sentence boundary placement is encoded in the N −N transitions and rules 2, 3, and 4 control the emergent tree structures. This model seems much more informative about the learning process and is, in some sense, minimalist with regard to the information it extracts. Therefore, we choose Rescorla-Wagner Q-learning for the analysis of more complex sentence structures.

In this section, we explore how the Rescorla-Wagner Q-learning algorithm with continuous border condition performs on more complex grammars. We consider a number of different grammars. First, we consider a grammar that has both mono- and ditransitive verbs to explore the model’s behavior and performance when introducing a minimal change that makes N −N transitions unreliable predictors of sentence transitions. We then increase the complexity of the grammars by either introducing relative clauses or a number of other adnominal elements. Relative clauses allow for including recursive repetition of structures within a sentence, a very common feature of the world’s languages. Complexifying the noun phrase allows for studying how the models behave when exposed to variable structures within a recurring constituent.

For the two latter more complex languages, the grammatical information extracted will often be too big to analyze in a table as we have done for the NVN language. Instead, we study the learning curve and break it down by sentence length to get some insight into which sentences are learned first. This analysis is complemented by performance tests. We then examine some examples of tree structures to show some systematic reuse of identified structures.

Here, we report results for a MD language with K_n = 5, and K_m = K_d = 1. We simulate the learning using the Rescorla-Wagner Q-learning with continuous border condition.

Figure 3 reports the learning curve for this language. The first panel displays the standard learning curve and the second panel shows the learning curve for different sentence lengths. Sentences of length 3 are monotransitive sentences and sentences of length 4 are ditransitive sentences. As can be seen, even when N −N transitions are not a reliable predictor of sentence boundaries, the learning algorithm still learns to identify sentences. The second panel shows that monotransitive sentences are learned first and that there is some interference between the mono- and ditransitive sentences. At the beginning, monotransitive sentences are quickly learned, but the learning slows down as the learner figures out how to deal with ditransitive sentences. After a transition period, both types of sentences are learned. In order to better understand how the processing is done, we look at snapshots taken during the learning process. All data is available online in the Excel spreadsheet RWQLearnerC_MD_revisedFinal.xlsx stored in the GitHub repository: https://github.com/michaudj/LanguageLearner/. The results are displayed in Table 6.

The first two snapshots show that at this stage of learning, only information about monotransitive verbs has been learned and decision making relies a lot on full sentences (⁠$Q≥Q¯$⁠). However, from trial 3,000, the learner starts to rely more on shorter chunks in the decision making process. At this point, the learner starts to have knowledge about ditransitive verbs. At the next snapshot, learning is complete and we see that now boundary placement essentially relies on N −N transitions as in the NVN case. The ditransitive case is solved by using longer chunks containing ditransitive verbs to inhibit border placement and favor a chunking action instead. But in both sentence structures, border placement relies on N −N transitions. This is best seen in the last snapshot where only the N −N transition supports border placements.

In order to see how the performance of the model evolves, we perform a performance test at each of the snapshot times. The test set contains 100 sentences divided into 52 monotransitive sentences and 48 ditransitive sentences. Results of these tests are reported in Table 7. We observe that the tree structures of monotransitive sentences are similar to those of the NVN language, with a bias towards the [N,[V, N]] structure. In contrast, the ditransitive sentences do not have a uniform distribution of tree structures; instead the structure [N,[[DV, N], N]] is overly represented and accounts for 48% of the cases, while the structure [[N, DV ],[N, N]] only occurs in 2% of the cases. This suggests that the chunk [DV, N] is used to inhibit border placement and promotes chunking instead, while allowing the final N to correctly predict border placement.

Here, we report results for a RelClause language with R_n = 5, R_m = R_d = R_r = 1. Once again, learning is done using the Rescorla-Wagner Q-learning algorithm with continuous border condition. The learning curves are displayed in Figure 4. Once again, we see that the model learns this language without problems. Interestingly, we observe a U-shaped curve at the beginning of the learning process and we can clearly see that the learning process is irregular and shows a staircase pattern with a plateau around trial 15,000. The breakdown into sentence lengths is showing that at first short sentences are learned well, but then performances decrease while the system learns to deal with ditransitive structures and relative clauses. After some delay all longer sentences get learned without significant delays, pointing out the generalization process over the end of sentences, for instance, 75% of the sentences end with MV, N, making [MV, N] −N transitions a good predictor of sentence boundaries.

We now take a closer look at the development of the grammatical information of a single learner by extracting snapshots of its state-action values. Selected results are displayed in Table 8. Not all constructions are displayed due to their high number. The display is limited to s₁ constructions of a maximum length of 3 to avoid a combinatorial explosion in the table. The display is also limited to constructions that are relatively frequent or that reach relatively high Q-values, but not necessarily high $Q¯$ values. The threshold of the Q-values of the displayed structure is set to t_G = 4, lower than previous thresholds, in order to capture what is going on in the initial U-curve. Full data on all the state-action values above for this learner are available in the Excel file RWQLearnerC_rel_revisedFinal.xlsx available online in the GitHub repository https://github.com/michaudj/LanguageLearner/.

In the snapshots, we first explore the initial U-shaped curve using both the Q and $Q¯$ values and performance tests. From the performance test, we see that monotransitive sentences have the structure [[N, MV ], N] and Table 8 suggests that this identification relies on the full structure, since only the border placement with that structure has a weight about the threshold. Note that $Q>Q¯$⁠, which suggests that the decision is not based on sub-states. The performance test shows that at this stage some ditransitive sentences are recognized, but without a preferred tree structure. Soon, at 1,000 trials, the model performances deteriorate and the number of sentences correctly identified drops to nearly 0. In Table 8, we observe that the model still relies on full structures for decision-making, but the weights are lower than in the first snapshot. Such a decrease in performances can be explained by the presence of ditransitive sentences in which the N −N transition is not a good predictor of sentence boundaries. After 7,000 trials the U-curve has turned and the model learns that monotransitive sentences should be analyzed [N,[MV, N]] as shown in Table 9 and the [MV, N] −N transition becomes a better boundary predictor. Table 8 also contains more information about ditransitives, but the learner still doesn’t know how to process relative clauses. The performances then steadily increase until the learner is proficient. In the specific learner analyzed in detail here, a high proficiency level is only reached at the last snapshot. This is illustrated by the much higher performance on the test set and the much more systematic structures used by the model.