Humans have developed a great variety of complex communicative systems (languages) without any centralized assistance. Therefore, evolution of human communication has often been modeled as a result of distributed learning among agents which are reinforced for successfully transmitting information to each other. These models, however, face two major challenges: 1) even in most successful cases, the agents can only develop a very small number of communicative conventions, whereas humans managed to successfully agree upon thousands of words; 2) after groups of artificial agents converge on a set of communicative conventions, they have no incentive to improve or expand it, whereas the development of human languages is open-ended. Here, I show how these two challenges could be resolved by dynamically changing the problem that the agents are learning to solve with communication. I suggest that the communicative problem that starts small and gradually increases in difficulty as the agents agree upon new communicative conventions is essential for achieving tractable evolution of rich communicative systems in decentralized multi-agent communities.

Learning to communicate in adaptive multi-agent populations introduces instability challenges at the individual and population levels. To develop an effective communication system, a population must converge on a shared and sufficiently stable vocabulary. We explore the factors that affect the symmetry and effectiveness of the communication protocols developed by deep reinforcement learning agents playing a coordination game. We looked at the effects of bottom-driven supervision, agent population size, and self-play (“inner speech”) on the properties of the developed communication systems. To analyse the resulting communication protocols and derive appropriate conclusions, we developed a set of information-theoretic metrics, which has been a major underdevelopment in the field. We found that all the manipulated factors greatly affect the decentralized learning outcomes of the adaptive agents. The populations with more than 2 agents or with a self-play learning mode converge on more shared and symmetric communication protocols than the 2-agent (no self-play) groups. Bottom-driven supervising feedback, in turn, augments the learning results of all groups, helping the agents learning in bigger populations or with self-play to coordinate and converge on maximally homogeneous and symmetric communication systems. We discuss the implications of our results for future work on modeling language evolution with multi-agent reinforcement learning.