Reproductive division of labor in a colony of artificial ants

We simulate an ant colony in which an ant’s genetics can determine behavioral, morphological and physiological differences between workers and queens. We show that depending on the benefits conferred to workers and queens different reproductive division of labor strategies evolve. In particular, we observe both generalist colonies and colonies with specialized worker and queen castes. Generalist colonies were subject to selection for optimal response thresholds. Colonies with castes evolved a discrete queen caste and either a discrete or continuous worker caste. As a secondary experiment we expose our evolved colonies to a changing environment to test their ability to adapt cooperative foraging strategies and we find all reproductive division of labor strategies were effective at cooperative foraging.


Introduction
Modern life has evolved through several major evolutionary transitions (Szathmáry and Smith, 1995;Smith and Szathmary, 1997) that resulted in the origin of protocells, prokaryotic cells, eukaryotic cells, plastids, multicellularity, eusocial animal behavior, and human societies. These major transitions are characterized by common features in which lower-level evolutionary units form cooperative groups through division of labor and specialization and then emerge as new higher-level evolutionary units with novel inheritance systems (Szathmáry, 2015). The transition from a population with no reproductive division of labor to one that does is critical in each of these transitions.
Reproductive division of labor (RDoL hereafter) describes colonies in which one or a small number of members in the colony are responsible for reproduction (i.e. queens). All other members (i.e. workers) carry out non-reproductive tasks like excavation, brood care, foraging, and defense. RDoL occurs only in eusocial animals like ants, bees, termites, and some mole rats (Crespi and Yanega, 1995;Burda et al., 2000;Wilson and Hölldobler, 2005;Nowak et al., 2010;Gadau et al., 2012). RDoL is one of three criteria usually used to identify eusocial animals. In addition eusocial animals have multi-generational communal cohabitation and shared responsibility for caring for their young.
We have chosen to focus our experiment on ants. There are over 14000 species of ants and social behavior is understandably varied among these species. Gadau et al. (2012) review the genetic impact of social evolution on seven different ant species for which the genome has been sequenced. These species show a variety of RDoL strategies. On one end of the spectrum is Harpegnathos saltator a species with very little RDoL and is considered to be the closest (of the seven species considered) to the ancestral ant species with no RDoL. On the other end of the spectrum you have species like Atta cephalotes in which the colony has a single queen that is larger than the workers and lives decades longer.
RDoL is characterized by behavioral specialization that may also correlate with morphological or physiological specialization. When morphological and physiological differences do exist the resulting castes can be discrete or continuous. A discrete caste has very little variation among the members of the caste. The species Atta cephalotes mentioned above has discrete castes. A continuous caste is one which has variation in some property within the caste and there are no gaps in the continuum. Solenopsis invicta is an example of an ant species in which worker size is on a continuum.
The mechanisms underlying RDoL vary between species and usually include factors in genetics, nutrition, hormones, and social contact between individuals (Beshers and Fewell, 2001;Korb et al., 2009;Gadau et al., 2012). In our experiment we are interested in the genetic factors that can impact the evolution of different RDoL strategies. One model of RDoL that takes genetic factors into considerations is the response threshold model (Bonabeau et al., 1998). The response threshold model assumes that for each task the ant has a threshold for a task-specific stimuli and the ant will select the task only when the stimuli is over the threshold. RDoL is achieved through variation among thresholds in the colony. Genetic factors can impact the ant's innate response threshold, its perception of the stimuli and its decision to carry out a task.
In Nowak et al. (2010) the authors argue that the evolution of eusocial behavior occurs through four vital stages.
The first stage is the formation of groups out of otherwise solitary individuals for the purpose of mutual defense, proximity to food and/or mates (i.e. multi-generational cohabitation). In the second stage preadaptations favorable to group life, like a propensity to cooperate and provisioning before birth, are selected for (i.e. shared responsibility for young). In the third stage the natural behavior of adult females is suppressed in many members of the population (i.e. RDoL). Finally, in the fourth stage emergent properties of the interactions between individuals can evolve leading to behaviors like cooperative foraging.
We have chosen to focus our experimentation on the third stage in which RDoL emerges and the fourth stage in which cooperative foraging strategies can evolve. We have previously studied the emergence of eusocial behavior in foraging agents capable of social learning (Marriott and Chebib, 2016) and division of labor in artificial ants in the fourth stage (Marriott and Gershenson, 2011).
We have reimplemented and extended the model of Marriott and Gershenson (2011) to create a simulation of an artificial ant colony. Ants in our model begin as cohabiting individuals sharing a single mound with preadaptations toward cooperation and provisioning. Thus, they are representative of a semisocial colony around stage two of the evolution of eusocial behavior and are intended to mimic an ancestral ant species with no RDoL or a modern species like Harpegnathos saltator. Every ant in the initial population is a generalist and ants forage via a solitary exploration strategy.
Our ants have a gene that in experimental settings determines the response thresholds for the breeding and foraging tasks and potentially morphological and physiological differences between queens and workers. In our control setting the gene has no effect on response thresholds and ants select the breeding task and forage task with equal probability. We see a variety of RDoL strategies evolve ranging from a colony of generalists to discrete and continuous worker and queen castes.
As a secondary experiment we have exposed our evolved colonies to a changing environment from one in which solitary exploration is adaptive into one in which cooperative foraging is adaptive. This is similar to the seasonal environments used in (Marriott and Gershenson, 2011). Our ants have a second gene which determines the response threshold for the two foraging strategies. We test the ability of our evolved colonies with different RDoL strategies to evolve cooperative foraging strategies when exposed to the second environment.

Model
Our ants live in a 2d discrete 80 × 60 toroid environment. The ant mound is placed in the "middle" cell and food is distributed among the cells depending on experimental settings. In our primary experiment we use an environment that

Season
Length d f r 1 (Sparse) 50000  3  200  0  2  25000  8  500  0  3 25000 13 1250 500 4 25000 18 1500 1500 5 (Dense) 75000 23 5000 5000 Our environments differ in how food is added to the environment initially and over time. Food can either spawn or regenerate. When food spawns every 50 ticks (so long as max food is not reached) a new food patch is created with f units of food and placed in the environment in a cell that is at least d cells away from any other patch. When food regenerates every 50 ticks a food cell grows by r units of food (so long as max food is not reached). Table 1 summarizes the values used by the five seasons in our secondary experiment. The first season's settings are identical to our sparse environment.
Each cell in the environment also records the level of incoming and outgoing pheromones. Outgoing pheromones are left by ants on their way out of the mound and can be followed back to the mound. Incoming pheromones are left by ants on their way back to the mound with food and can be followed back to the food source.
During every tick, each tile attempts to spread half of its pheromone level to its immediate neighbors in four directions. If the neighboring tile's pheromone levels are higher than half of the current tile's pheromone levels no spread occurs. Pheromones decay linearly at a rate of one twohundredth of the maximum energy level.
There are two tasks that ants in the colony must perform. Food must be gathered (forage task) and larvae must be created (breed task). We call these two tasks foraging and breeding and ants specializing in these tasks (or performing these tasks) workers and queens respectively. All ants begin as generalists capable of foraging and breeding. An idle ant (one who has just matured or has just finished a task) will select a new task at random according to its response thresholds. These thresholds are determined by the ant's genetics (see below).
When an ant selects the forage task it leaves the mound and searches for food according to a foraging strategy selected at random (response thresholds determined by the ant's genetics). Workers are either explorers or exploiters. Explorers leave the nest and follow a random walk in which 90% of the time they continue in the same direction and 10% of the time they turn left or right (never backwards).
Workers leaving the nest are allocated an energy level determined by genetics. This level determines how long the ant can forage outside the mound before returning home and the level of pheromones dropped while foraging. Every tick the energy level is decremented by 5 so pheromone levels drop the longer an ant is foraging. Workers normally return home when they are at full carrying capacity with food or if they deplete a tile of food. They follow the outgoing pheromones home by travelling to the cell with the highest outgoing pheromones. A worker that never finds food and runs out of energy will return home empty handed.
On the way back to the mound the ant leaves incoming pheromones (as long as it is carrying food) following the same rules as outgoing pheromones. Exploiters leave the mound and follow incoming pheromones back to food sources by travelling to the cell with the highest incoming pheromone level. Like explorers exploiters leave outgoing and incoming pheromones. If there are no incoming pheromones to follow exploiters act as explorers. When a worker returns to the nest, with food or not, it enters the idle state and selects a new task.
When an ant selects the breeding task it is placed in a pool of queens. As long as the current food level is greater than twice the sum of the current ant population and current larvae population a new larvae is created (otherwise queens idle until food level increases or they have idled too long). The new larvae is created by selecting a queen from the pool at random. This queen is the parent of the new larvae. It is not the genetic parent of the new larvae but does get fitness credit for successfully producing a new larvae. When a queen creates a larvae the queen must rest for a period determined by its genetics before entering the idle state and selecting a new task. Larvae are created until either food is depleted or there are no active queens.
Since reproduction in our simulation is asexual if the parent's genetics are selected for the child then all children of specialized queens would be queens too. Instead we select a genetic parent from the colony at random using a fitness function. This type of reproduction may seem artificial but it was chosen to mimic ant reproduction in species where queens and workers are genetically different (as it is in our ants). For instance, in Pogonomyrmex barbatus workers and queens are genetically different and workers and queens result from different matings (within lineage vs inter-lineage respectively) (Gadau et al., 2012;Cahan et al., 2004).
Our fitness function combines the ant's success at foraging with the ant's success at breeding. Let w be the amount of food returned to the nest (per tick) and let q be the number of larvae created (per tick) then the fitness of an ant is w w · w + w q · q where w w and w q are weights to balance the contribution of w and q. Because workers and queens have different advantages in our experimental setup we have had to vary the weights w w and w q in order to ensure that there are always queens and workers among the most fit individuals in cases where specialization occurs (otherwise a colony of all workers or all breeders will die out).
Instead of sorting our ants by fitness we use a more efficient procedure. We use the mean fitness to filter out all ants less than the mean. Then of the ants above the mean we repeat this process to isolate the top quartile of ants by fitness. A genetic parent is selected from this group uniformly at random. The two genes used in our experiments are real valued genes in the range [0, 1] and straightforward mutation mechanisms are used (increment by a random value in the range [−0.1, 0.1]). We use a mutation rate of 5%.
When a new larvae is created it eats a meal (10 units of food) from the food stored in the mound. A new larvae must mature for 100 ticks before becoming an adult ant. When it matures it eats a second meal and then selects its first task. If there is no food when it matures it dies.
After eating a meal an ant's hunger level is reset to 0. Every tick the hunger level increases by 1 and if the hunger level reaches the threshold (500) the ant becomes hungry (so ant's must eat after 500 ticks). If away from the mound the ant attempts to return. In any case when the ant is next idle it will eat a meal from the mound. If there is no food the ant dies. Ants over the age of 100 ticks can also die with a small chance every tick determined by the ant's genetics.

Experimental Setup
Our ants have a real valued gene breed ∈ [0, 1] that determines the response thresholds for the breeding and foraging tasks. That is, an idle ant will select the breeding task with probability breed and the foraging task with probability 1 − breed. The initial ant population in all experimental runs are generalists in that their breed genes are set to 0.5. This means an idle ant from the initial population decides to breed or forage with 50% probability.
In addition to determining the response thresholds for task selection the breed gene also impacts ant behavior in other ways depending on experimental settings. The breed gene also impacts the lifespan of the ant, the rate larvae can be created, the amount of food that the ant can carry, and the amount of energy an ant receives when leaving the mound.
When the breed gene impacts life span an ant will live 100 ticks and then have a 0.01 + (1 − breed) * 0.99% chance to die each tick. When the breed gene impacts the rate larvae can be created an ant selecting the breed task must wait 1 + (1 − breed) * 99 tick(s) until they can select a new task. When the breed gene impacts the amount of food the ant can carry an ant can carry a max of 1 + (1 − breed) * 99 units of food. When the breed gene impacts the amount of energy an ant receives any ant leaving the mound will have 5 + (1 − breed) * 495 energy. When these benefits are not activated the value is calculated with breed = 0.5.
Our primary experiment is to vary the strength and impact of the breed gene. In our control run the breed gene has absolutely no effect. In its weakest state the gene merely impacts response thresholds. We also can activate each of the four additional morphological and physiological effects above. This leads to 16 different experimental settings of which we will focus on only 8. For each setting we have an initial population of 100 and we conducted 200 runs for 100000 ticks measuring the distribution of values in the breed gene.
Our ants have a second real valued gene exploit ∈ [0, 1] that determines the response thresholds for the explorer and exploiter strategies for foragers. An idle ant that selects the foraging task will select the exploiter strategy with probability exploit and the explorer strategy with probability 1 − exploit. The initial ant population are explorers (exploit = 0) so that the initial ants mimic ancestral semisocial ants that use an explorer strategy.
In our primary experiment the colony is placed in a sparse environment in which explorers are adaptive. In our secondary experiment we take a population adapted to a sparse environment and expose it to a dense environment over a sequence of four seasons. In the dense environment exploiters are adaptive. We run our secondary experiment for 200,000 ticks to allow the colony first to evolve its RDoL strategy and then adapt to the new environment. Our second experiment reproduces some of the experiments in (Marriott and Gershenson, 2011) and tests the conditions under which a colony in stage three can move into stage four adapting from solo foragers to cooperative foragers.

Observations and Discussion
As a control run we have evolved a colony in which the breed gene has no effect on response thresholds at all. That is, idle ants in the control setting select the breed or forage task with equal probability. In this setting the breed gene is subject to genetic drift only so we should expect no selec-tion pressure at all. Figure 1 shows the distribution of breed genes in the control run when subject only to genetic drift.
Our experimental colonies have evolved a number of different RDoL strategies. Some colonies remain generalists but their response thresholds face selection towards foraging in some cases and breeding in others. Other colonies specialize into two discrete queen and worker castes with very low variation among members of the castes. Finally in some cases continuous castes evolves in which members have more varied response thresholds.
In the first experiment the breed gene impacts response thresholds and may impact morphological and physiological differences that might benefit one caste or the other. We consider four cases: where no caste benefits, where queens benefit, where workers benefit and where both benefit. Figure 2 shows the distribution of the breed gene over time in these four experimental runs.
We can see that when neither caste has benefits (Figure 2 top left) the colony does not specialize. Instead the colony consists of a single continuous caste of generalists. The breed gene is subject to genetic drift and selection towards more reasonable response thresholds. In these settings the response thresholds settle around an average of 0.3 with a wide variation. This appears to be the optimal value given these settings as the colony can reach a higher population level than in the control settings where the response thresholds are fixed at 50%. Despite genetic drift and selection the behavior of this colony does not change much over the simulation. The single caste of generalists is similar to an ant species like Harpegnathos saltator.
Two genetic effects impact queen morphology and physiology. The first increases the lifespan of queens while decreasing the lifespan of workers. The second increases the rate that larvae can be created by queens while decreasing it for workers. When the benefits to queens are activated the colony does not specialize (Figure 2 top right). However, we can see that when the queens gain a benefit the response thresholds are pulled towards the queen end of the spectrum and settles around an average of 0.7. This means that the average ant selects the breed task more often (about 70% of the time) but also that they live longer and recover from breeding tasks quicker. Since the genetics do not impact foraging behaviors these generalists are as efficient foraging as in the default conditions and thus evolve to take advantage of the queen morphology but still forage enough to maintain the population. When queens benefit the single generalist caste is continuous but not as wide as in the last case.
The other two genetic effects impact worker morphology and physiology. The first increases the amount of food a worker can carry while decreasing it for queens. The second increases the energy level of a worker while decreasing it for queens. When workers benefit the colony specializes into discrete castes with little variation among its members (Figure 2 bottom left). The colony settles on about 40% queens. These discrete castes have morphological and physiological differences and more than one queen. This colony is similar to ant species like Linepithema humile.
Finally when both workers and queens benefit we get specialization but not as strong as when only workers benefit (Figure 2 bottom right). Under these conditions the queens still specialize into a discrete caste with little variation among its members. Queens make up about 20% of the population. The workers however form a continuous caste that is very similar to the generalist colony evolved when ants have no benefits (but with much less variation). Thus workers will still select the breed task from time to time, but when they do they are less efficient at it than the dedicated queens. This colony also displays castes but since its worker caste is continuous this colony is closer to an ant species like Solenopsis invicta.
In order to take a closer look at the benefits we have added we also look at cases where only one morphological or physiological benefit has been added. Figure 3 shows the distribution of the breed gene over time in runs when queens live longer, queens create larvae faster, workers can carry more and workers travel further (but with only one benefit activated at a time).
When queens live longer no specialization occurs (Figure 3 top left). The colony evolves to a state very similar to when queens live longer and create larvae faster. The colony evolves to generalists with response thresholds favoring the breeding task. However when queens don't live longer but are able to create larvae faster the colony evolves to castes (Figure 3 top right). The highest concentration of ants are still specialized as queens or workers but the variation among the worker castes is greater than in the most specialized run (when workers benefit). Variation among the worker caste is greater than the queen caste so in this case we say that the queen caste is discrete while the worker caste has some continuity. The colony consists of approximately 30% queens.
When workers can carry more food specialization into castes occurs similar to when queens can create larvae quicker (Figure 3 bottom left). Variation among workers is greater than variation among queens so again we say the queen caste is discrete while the worker caste is continu- ous. It is interesting that both when queens breed faster and when workers carry more we get similar specialized castes with a discrete queen caste and a continuous worker caste. These benefits, though benefiting different castes, have similar overall benefits to the colony. That is, as workers carrying more increases the rate of food brought home to the colony both of these benefits allow for larvae to be created faster. In the later case because queens have less downtime but in the prior case because with more food entering the mound more larvae can be created even with the same downtime. In the prior case this also means a larger population can be achieved (see Table 2). When workers can spend more time out of the mound we get a strong specialization in queens. Workers still remain in a varied continuous caste but the variation is greater than in the last two cases. Table 2 summarizes the RDoL strategies evolved in each run. In addition it indicates the stable population level that the colony could achieve for each run. We can see that the control settings reaches a stable population of about 1100. The environment remains the same in each run (that is, the same amount of food is available) but the experimental colonies are subject to selection pressures that seek to  The run name corresponds to which benefits were activated in the run. The order of benefits is queens live longer, queens create larvae faster, workers carry more and workers travel further. We also indicate the stable population achieved in the run and the values of w q and w w used for the run.
increase population size. Even with no morphological or physiological benefits selection adjusts the response thresholds so that a population of about 1300 could be maintained. The most efficient run was when workers benefit but queens do not. Recall in this case we have discrete queen and worker castes and we also reach a stable population of 1700. Most other benefits allowed for a higher population to be reached than the default settings except when queens create larvae faster. When queens create larvae faster but there are no other benefits there is no way to increase food into the mound so the population cannot grow larger. When queens live longer a higher population can be maintained because queens remain in the population longer.
We notice that in all runs in which castes evolve the queen caste is always discrete. We explain this due to an asynchonisity between queens performing the forage task and workers performing the breed task. A worker performing the breed task is no worse than a queen at creating a larvae (it must only wait longer before selecting a new task and may not live as long as queens). On the other hand a queen performing the forage task will have a very low energy level and cannot carry much. This means most queens that select the forage task return home empty handed or with a very small amount of food. This is detrimental to their fitness and thus selection is much stronger for queens to avoid foraging.
Another trend we notice in runs where specialization occurs is that specialization occurs towards foraging first. Workers receive immediate fitness benefit from specialization in cases where workers benefit. Carrying more and spending more time out of the mound means less failed foraging attempts and more food returned per trip. On the other hand queens gain no fitness bonus from living longer (since our fitness function considers number of larvae created per tick not over their lifetime). Secondly creating larvae more rapidly does impact the fitness value but only if you are selecting the breeding task reliably (i.e. a specialized queen). However, once workers begin to specialize (and are less efficient at breeding) selection for specialized queens becomes stronger and benefits from breeding faster are amplified.
This seems most clear in the run where all benefits are activated. In this run workers specialize only into a continuous generalist forager castes similar to when no benefits are activated. However, this is enough to amplify the selection pressure of even a single or a few specialized queens and a discrete caste of queens still evolves.
In our secondary experiment we took our colonies that were adapted to the sparse environment and exposed them to a dense environment over four additional seasons. As the seasons transition from sparse to dense the adaptive foraging strategy changes from solitary exploration to cooperative foraging using pheromone trails. Figure 4 shows the distribution of the exploit gene over time for populations exposed to the dense environment. The dense environment arrives at tick 125,000 of 200,000.
We can see that adaptation to exploiters occurs in all experimental conditions. However, complete specialization only occurs in runs where no queen caste exists. The reason for this is that only in these population do all members face selection for their foraging strategy. When there is a specialized queen class there is no selection pressure on the exploit gene for members of the queen class. As they never forage they're foraging strategy is irrelevant and is subject to genetic drift. In cases of exploiter specialization the exploiters are still continuous and in cases where exploiters don't fully specialize we still see some explorers. This is because explorers are still needed even when exploitation is adaptive. This is why exploiters act as explorers in absence of incoming pheromones.

Conclusions
In our primary experiment we considered an ant colony in which an ant's genetics determines its response thresholds to the breeding and foraging tasks. Therefore these ants have the potential to evolve into genetically distinct behavioral castes. In our experimental settings the ant's genetics determine response thresholds and morphological and physiological differences between specialist workers or queens.
When the ant's genetics only impact response thresholds our ant colony evolves to be a colony of generalists with a preference for foraging. Generalists with a preference for breeding evolve when queens live longer than workers and workers have no benefits. If queens do not live longer than workers but can create larvae quicker the colony evolves a discrete queen caste and a continuous worker caste.
A discrete queen caste and a continuous worker caste also evolve when workers can carry more. When workers travel further and when both workers and queens benefit a discrete queen caste with little variation evolves along side a continuous caste of generalist workers. Finally, two discrete castes with little variation evolve when workers can carry more and can spend more time out of the mound. These are the only settings in which two discrete castes evolve.
These varied RDoL strategies evolved under different genetic circumstances. This parallels the wide range of RDoL strategies observed among different ant species. Our experiment has explored how the presence of morphological and physiological differences among ants performing different tasks can lead to the evolution of different RDoL strategies.
Our secondary experiment examined how colonies with different RDoL strategies can adapt to a changing environment and evolve cooperative foraging strategies. We have shown that all types of RDoL strategies we've studied can evolve cooperative foraging strategies.