We present a study demonstrating how random walk algorithms can be used for evolutionary image transition. We design different mutation operators based on uniform and biased random walks and study how their combination with a baseline mutation operator can lead to interesting image transition processes in terms of visual effects and artistic features. Using feature-based analysis we investigate the evolutionary image transition behaviour with respect to different features and evaluate the images constructed during the image transition process. Afterwards, we investigate how modifications of our biased random walk approaches can be used for evolutionary image painting. We introduce an evolutionary image painting approach whose underlying biased random walk can be controlled by a parameter influencing the bias of the random walk and thereby creating different artistic painting effects.
Evolutionary algorithms (EAs) have been widely and successfully applied in the areas of art (Romero and Machado, 2008; Hingston et al., 2008; McCormack and d'Inverno, 2012b; Lambert et al., 2013; Antunes et al., 2015; Neumann and Neumann, 2018a, 2019). In this application area, the primary aim is to evolve artistic and creative outputs through an evolutionary process (McCormack and d'Inverno, 2012a; Vinhas et al., 2016; al-Rifaie and Bishop, 2013; Greenfield, 2015; Neumann, Szpak et al., 2017; Neumann and Neumann, 2018b). The use of evolutionary algorithms for the generation of art has attracted strong research interest. Different representations have been used to create works of greater complexity in 2D and 3D (Todd and Latham, 1992; Greenfield and Machado, 2015; Machado and Correia, 2014), and in image animation (Sims, 1991; Hart, 2007; Trist et al., 2011; Neumann, Neumann et al., 2019). The great majority of this work relates to the use of evolution to produce a final artistic product in the form of a picture, sculpture, or animation.
1.1. Related Work
In the seminal work, The Blind Watchmaker, Dawkins (1986) investigated the process of evolution. He evolved figures called biomorphs in order to explore and to visualize the power of evolution. The biomorphs' appearance changes/diverges broadly from the original parent over time demonstrating a simple model of bio-inspired evolution. Inspired by Dawkins' work, Sims (1991) created complex and imaginative images. He used an expression-based approach to evolve images using a mathematical expression as the genotype and applied crossover and mutation. In the interactive media installation Galapagos, visitors were able to evolve virtual creatures based on Darwinian evolution principles whilst taking into account their own preferences (Sims, 1997). The abstract organisms were displayed on computer screens and users interactively chose virtual organisms for simulated growth by stepping onto selected pads. This exhibition was a visualization example of a collaboration process between the visitors and the machine.
Latham (1985) created a black and white lithographic artwork called Black Form Synth consisting of hand-drawn “evolutionary trees of complex forms” in the form of a flow diagram arranged with a set of transformation rules. Following the expression-based approach, Latham (1989) extended the rule-based evolutionary approach in order to invent new complex 3D forms from geometric primitives. Todd and Latham (1992) introduced the framework called Mutator to generate art and evolve new 3D biomorphic forms. The Mutator creates a series of complex branching organic forms and surreal virtual sculptures, and animated videos through the process of “surreal” evolution. At each iteration the artist selects phenotypes based on the idea of evolved organisms that are able to “breed and grow.”
Over time, the growing interest in an expression-based approach led to expansion of the area of evolutionary art, providing a coherent and solid basis for further research (Fenton et al., 2017a; Richter, 2018; Urbano, 2018; Paauw and van den Berg, 2019). Hart (2007) used an expression-based approach with the focus of evolving a set of images with a different appearance to the previous works. This system's interface allowed a greater range of control over the colors and forms of the evolved images. Complex and detailed images were created in Rooke (2006) using an expression-based approach guided by aesthetic selection, in particular towards the evolution of color space. Unemi (2004, 2012) continued to explore the expression-based approach to evolutionary art by evolving images and animations towards the progression of color volumes and novel forms with varied variables. The images are evolved based on aesthetic measures and have been displayed on the internet for decades using different versions of the SBART framework (Unemi, 1999). Machado and Cardoso (2002) introduced computer-aided software, called NEvAr. This is an evolutionary art tool which uses genetic programming, user-guided evolution, and automatic fitness assignment. The model uses a fitness function that permits aesthetically pleasing and visually complex images. Draves (2005) introduced Electric Sheep, a large and ongoing evolutionary art project using collective human evaluation. It was implemented as a distributed screen-saver allowing users to approve or disapprove phenotypes in order to evolve artificial life and is focused on the continuing behavior of the distributed system. Greenfield (2006) describes simulated robots embedded into an evolutionary framework for the purpose of painting a new artwork. The system implements optimization towards identifying robot paintings with higher aesthetic properties and takes into account the behavior of the simulated robots.
Aesthetic feature measures have been widely applied to generate new artistic images (Lewis, 2008; Neumann, Pyromallis et al., 2018; Heaton, 2019). For the evolutionary art system NEvAr, Machado and Cardoso (2002) used aesthetic measures and evolutionary computation methods, and thus developed automatic seeding procedures to generate images. Greenfield (2002) evolved images using computational aesthetic functions that are based on a color segmentation algorithm. Moreover, den Heijer and Eiben (2014) studied aesthetic measures in unsupervised evolutionary art using aesthetic measures as fitness functions.
In Interactive Evolutionary Computation (IEC), the traditional objective-function is replaced by a human subject who guides the selection (Takagi, 2001). In IEC, each solution is evaluated by a human judgment. Those subjective choices provide the basis for the selection of solutions in the evolutionary process. IEC founds its biggest application in the creative domains in which including humans in the loop can lead to novel solutions (Takagi, 2001; Romero and Machado, 2008; Hingston et al., 2008; Fenton et al., 2017b; Hollingsworth and Schrum, 2019). In this case, the fitness function is based on the individual user's experiences and preferences towards interesting or aesthetic results. This means that the fitness of the evolutionary algorithm (EA) is strongly subjective.
In contrast to this, a different approach evaluates artistic images by automating fitness evaluation in order to generate novel images. Baluja et al. (1994) attempted to automate the process of image evolution by using neural networks in order to produce images. The primary idea was to learn the user's preferences, and to apply this insight to generate aesthetically pleasing images. Moreover, den Heijer and Eiben (2014) presented an approach for evolving images without human interaction. Instead of human interaction, they used one or more aesthetic measures as fitness functions to guide the search. They investigated the correlation between aesthetic scores in order to calculate which aesthetic measures generate images that are evaluated positively by the other aesthetic measures.
Another application of evolutionary algorithms to art is the creation of image transitions. Graf and Banzhaf (1995) used interactive evolution to help determine parameters for image morphing. They combined interactive evolutionary computation with the concept of warping and morphing from computer graphics to evolve images. More precisely, they used the recombination of two bitmap images through image interpolation. Furthermore, Karungaru et al. (2007) used an evolutionary algorithm to automatically identify features for morphing faces.
More recently, deep neural networks have been used to create artistic images through the transfer of artistic style from one image to another, facilitating novel forms of image manipulation (Gatys, Ecker et al., 2016). A new neural style transfer approach preserves the colors of the original image using simple linear methods for transferring style. Gatys, Bethge et al. (2016) and Gatys et al. (2017) extended the existing method by introducing control over colour information, spatial scale, and spatial location. Johnson et al. (2016) combined the benefits of both approaches, trained feed-forward convolutional neural networks and perceptual loss functions based on high-level features extracted from pre-trained networks and proposed the use of perceptual loss functions for training feed-forward networks for image transformation. The transfer of style from one image to stable video sequences was proposed by new initializations and loss functions applicable to videos in Ruder et al. (2016).
Furthermore, inspired by the work of Gao et al. (2016), evolutionary diversity optimization has been applied to evolve images in one/two aesthetic feature dimensions (Alexander et al., 2017; Neumann, Gao et al., 2018; Alexander et al., 2019). Building on these studies, Neumann, Gao et al. (2019) introduced evolutionary diversity optimization for images using popular indicators from the area of evolutionary multiobjective optimisation.
Nonphotorealistic rendering (NPR) is an area of computer graphics that focuses on enabling a wide variety of expressive styles for digital art (Strothotte and Schlechtweg, 2002). In contrast to traditional computer graphics, which has focused on photorealism, NPR is inspired by artistic styles such as painting, drawing, technical illustration, and animated cartoons. Litwinowicz (1997) described transformations from ordinary video segments into animations that appear similar to hand-painted techniques. In particular, they used modified off-the-shelf image processing and rendering techniques in order to process images and videos for an impressionist effect. Kubelka (1948) introduced the optical properties model in order to simulate the optical effect of different layers of artwork. Hertzmann (1998) generated hand-painted images by using series of different brush stroke sizes and orientations to generate images. In recent years, several evolutionary approaches for the production of nonphotorealistic renderings of images have been introduced (Izadi et al., 2010; Barile et al., 2008; Kang et al., 2005; Machado and Pereira, 2012; Tao Wu, 2018).
Furthermore, extensive research has been carried out on swarm painting. Urbano (2006) investigated choices inside a group of agents in order to design swarm art with interesting random patterns. Monmarché et al. (1999) used the stochastic and exploratory principles of an ant colony to automatically discover clusters in data without prior knowledge of the structure of the data.
A swarm-based system was used as a method to create visualizations of data, and in order to combine information aesthetics with data visualization (Maçãs et al., 2015). Boyd et al. (2004) created a collaborative project called Swarm Art that incorporated the swarm-based simulation and projected the artwork onto a large screen. Greenfield (2005) used a colony optimization model to evolve ant paintings and investigated the effects of different fitness measures on the generation of different artistic styles. Inspired by natural phenomena, namely the use of pheromone substances for mass recruitment in ants' pattern behaviour, Urbano (2005) generated collective artistic work by embedding a pheromone medium pattern into the painters' behaviour.
1.2. Our Contribution
Neumann et al. (2016) described an image transition process where the key idea is to use the evolutionary process itself in an artistic way. The focus of our article is to study how random walk algorithms can be used in the evolutionary image transition process defined in Neumann et al. (2016) as mutation operators. We consider the well-studied (1+1) EA, popular random walk algorithms and provide a new approach to evolutionary art by using theoretical approaches for evolutionary image transition. The transition process consists of evolving a given starting image into a given target image by random decisions. Considering an error function which assigns to a given current image the number of pixels where it agrees with and maximizes this function boils down to the classical OneMax problem for which numerous theoretical results on the runtime behaviour of evolutionary algorithms are available (Jansen and Sudholt, 2010; Witt, 2013; Sudholt, 2013). An important topic related to the theory of evolutionary algorithms is random walks (Lovász, 1996; Dembo et al., 2004). We consider random walks on images where each time the walk visits a pixel, its value is set to the value of the given target image. By biasing the random walk towards pixels that are similar to the current pixel, we can study the effect of such biases which might be more interesting from an artistic perspective. After observing these two basic random processes for image transition, we study how they can be combined to give the evolutionary process interesting new properties. We study the effect of running random walks for short periods of time as part of a mutation operator in a (1+1) EA. Furthermore, we consider the effect of combining them with the asymmetric mutation operator for evolutionary image transition introduced in Neumann et al. (2016). Our results show that the area of evolutionary image transition based on random walks provides a rich source of artistic possibilities for creating video art. All our approaches are pixel-based, and created videos based on the evolutionary processes show frames corresponding to images that were created every few hundred generations. After introducing these different approaches to evolutionary image transition based on random walks, we study their behaviour with respect to different aesthetic features. Feature-based analysis of heuristic search methods has gained increasing interest in recent years (Mersmann et al., 2010, 2013; Nallaperuma et al., 2013). In other application areas feature-based analysis is an important method to increase the theoretical understanding algorithm performance and particularly useful for algorithm selection and configuration (Nallaperuma et al., 2015; Neumann and Poursoltan, 2016). For evolutionary image transition, we study how artistic features behave during the transition process. This allows the measurement of the evolutionary image transition process in a quantitative way and provides a basis to compare our different approaches with respect to artistic measures.1
This article extends its conference version (Neumann, Alexander et al., 2017) in several ways. Firstly, by investigations on the impact of random walk lengths for image transition in Section 4.2 and the impact of the bias of the random walks controlled by the parameter in Section 4.3. Large values of increase the probability of moving to similar color pixel and different values of lead to different image transition processes. Secondly, we investigate how random walk algorithms can be used to carry out painting of images in Section 7. The key idea is to use a biased random walk starting at a given pixel and color all pixels visited by the walk with the color of the starting pixel. We use the biased random walk mutation approach introduced for evolutionary image transition and combine it with the asymmetric mutation operator. Our approach starts a biased random walk for each pixel to be changed by asymmetric mutation. To achieve different effects of evolutionary image painting, we consider the parameter which allows to control of the bias of the random walk towards similar color pixels. Our experimental results show the effect of the setting of for images of various types.
The outline of the article is as follows. In Section 2, we introduce the evolutionary transition process. In Section 3, we study how variants of random walks can be used for the image transition process. We examine the use of random walks as part of mutation operators and study their combinations with asymmetric mutation during the evolutionary process in Section 4. Furthermore, we extend our investigation on the impact of different random walk length in biased random walk mutation and on the impact of the chosen in biased random walk mutation. In Section 5, we analyse the different approaches for evolutionary image transition with respect to aesthetic features. Our evolutionary painting algorithm using biased random walks is introduced and evaluated in Section 6. Finally, we finish with some conclusions and future work.
2. Evolutionary Image Transition
We consider the evolutionary image transition process introduced in Neumann et al. (2016). It transforms a given image of size into a given target image of size . This is done by producing images for which holds. Given a starting image a target image , and a current image , we say that pixel is in state if , and is in state if . Our goal is to study different ways of using random walk algorithms for evolutionary image transition.
We consider simple variants of the classical (1+1) EA in the context of image transition. The algorithm is using mutation only and accepts an offspring if it is at least as good as its parent according to the fitness function. The approach is given in Algorithm 1. Using this algorithm has the advantage that the parent and offspring only differ by a small number of pixels which allows a smooth transition process. This ensures a smooth process for transitioning the starting image into the target. Furthermore, we can interpret each step of the random walks flipping a visited pixel to the target outlined in Section 3 as a mutation step which, according to the fitness function, is always accepted.
As the baseline mutation operator, we consider the asymmetric mutation operator which has been studied in the area of runtime analysis for special linear functions (Jansen and Sudholt, 2010) as well as the minimum spanning tree problems (Neumann and Wegener, 2007). Using this mutation operator instead of standard bit mutations for OneMax problems avoids the coupon collector's effect (Kobza et al., 2007; Mitzenmacher and Upfal, 2005). In the transition process, the coupon collector's effect implies that the last (even small) fraction of the pixels that need to be flipped to the target need more time to be flipped than all the pixels previously set to the target. More precisely, flipping each bit with probability as done in standard bit mutations, implies that the waiting time to flip the last pixel to the target is whereas increasing the number of target pixels is if the number of target pixels is still a constant fraction of all the pixels. Such a slow-down at the end of the transition process where there is no progress for many iterations is not desirable.
We use the generalization of this asymmetric mutation operator already proposed in Neumann et al. (2016) and shown in Algorithm 2. Let be the number of pixels where and agree. Similarly, let be the number of pixels where and agree. Each pixel in starting state is flipped with probability and each pixel in target state is flipped with probability . The special case of has been mathematically analyzed with respect to the runtime behaviour on other pseudo-Boolean functions.
We set the parameters as follows: and . This allows both a decent and sufficient speed for the image transition process and enough exchanges of pixels for an interesting evolutionary process. We should mention that obtaining the last pixels of the target image may take a long time compared with the other progress steps when using large values of . However, for image transition, this only effects steps when there are at most source pixels remaining in the image. From a practical perspective, this means that the evolutionary process has almost converged towards the target image and setting the remaining missing target pixels to their target values provides an easy solution.
All experimental results for evolutionary image transition in this article are shown for the process of moving from the starting image to the target image given in Figure 1 where the images are of size pixels. The algorithms have been implemented in MATLAB. In order to visualize the process, we show the images obtained when the evolutionary process reaches , , , and pixels of target image for the first time. We should mention that all processes except the use of the biased random walks are independent of the starting and target image which implies that the use of other starting and target images would show the same effects in respect to the way that target pixels are displayed during the transition process.
3. Random Walks for Image Transition
Our evolutionary algorithms for image transition build on random walk algorithms and use them later on as part of a mutation step. Specifically, we investigate the use of random walk algorithms for image transition which move, at each step, from a current pixel to one of 4-connected pixels in its neighbourhood.
The classical random walk shown in Algorithm 3 chooses an element uniformly at random (Pearson, 1905). We call this the uniform random walk in the following. The cover time of the uniform random walk on a torus is upper bounded by (Dembo et al., 2004) which implies that the expected number of steps of the uniform random walk until the target image is obtained (assuming ) is upper bounded by .
3.1. Biased Random Walk
In our random walk, we prefer if is small compared to the other elements in . In order to compute the probability of moving to a new neighbour we consider and prefer elements in where this value is large.
Our standard biased random walk works with and we extend our investigations to other choices of as part of our random walk mutation operator in Section 4.3. Furthermore, we use different choices of for image painting in Section 7. It should be noted that the uniform random walk given in Algorithm 3 is the special case of Algorithm 4 where .
The biased random walk is dependent on the target image when carrying out mutation or random walk steps and the importance of moving to a pixel with similar color. By introducing the bias in respect to pixels that are similar, the bias can cause the evolutionary image transition process to take exponentially long as the walk might encounter effects similar to the gambler's ruin process (Mitzenmacher and Upfal, 2005). For our combined approaches described in the next section, we use the random walks as mutation components which ensures that the evolutionary image transition is carried out efficiently. We will use the biased random walk for evolutionary image transition in Section 4.
The four images of the biased random walk are clearly different to the images of the uniform random walk. During the course of the transition, the difference between these processes becomes more prominent, especially in the background where at pixels of the target image there is nearly an absolute transition to the target image . In strong contrast, the darker abstract figure of the images stays nearly untouched, so that we see a layer of the yellow face in starting image in the center of the abstract black figure in target image . In this image, the figure itself is also very incomplete with much of the source picture showing through. These effects arise from biased probabilities in the random walk which makes it difficult for the walk to penetrate areas of high contrast to the current pixel location.
4. Random Walk Mutation and Combined Approaches
The asymmetric mutation operator and the random walk algorithms display quite different behaviour when applied to image transition. We now study additional ways of carrying out mutations as part of the image transition process. Our goal is to obtain a new evolutionary process using mutations based on random walks and biased random walks. Furthermore, we investigate the effect of combining them with the asymmetric mutation operator.
4.1. Random Walk Mutation
Firstly, we explore the use of random walks as mutation operators and call this a random walk mutation.
The uniform random walk mutation selects the position of a pixel uniformly at random and runs the uniform random walk for steps (iterations of the while-loop). We call the resulting algorithm EA-UniformWalk. Similarly, the biased random walk mutation selects the position of a pixel uniformly at random and runs the biased random walk for steps. This algorithm is called EA-BiasedWalk. For our experiments, we set which implies that each mutation carries out a random walk consisting of 100 steps.
The images from EA-BiasedWalk appear visually similar to those of the EA-UniformWalk in the beginning but differences emerge during the final stages of the transition. In EA-BiasedWalk, elements of the source image still show through in areas of high contrast in the target image, which the biased random walk finds difficult to traverse. At a more local scale, this mirrors the effects of bias in the earlier random walk experiments. At a global scale, it can be seen that the blue background, which is a low contrast area, is slightly more complete in the final frame of EA-BiasedWalk than in the same frame in EA-UniformWalk.
4.2. Impact of Random Walk Length
Furthermore, we investigate the impact of the random walk length in biased random walk mutation. We want to explore how different lengths of biased random walk influence the creative process of evolving images. We also aim to understand the influence that the length of the biased random walk has on the results.
In general, we want to investigate the importance of different random walk lengths in the evolutionary process. We also want to understand how this parameter shapes the final result and how we are able to control those effects in a systematic manner. The artistic techniques that appear from evolutionary image transition using different random walks can be compared with a French style of abstract painting, namely Tachisme (synonymous with art informel). Tachisme features the intuitive and spontaneous gestures of the artist's brushstrokes that involve the use of dabs, drips, or splotches of colour.
At the beginning, we observe that the use of parameter and results in images with smaller patches. In the later stages, the darker patches from the background dominate the image. Apparently differences between image transition with small and middle occur. It can be observed that the small and middle random walk lengths produce various distinctive images.
When the process of the evolutionary image transition reaches and pixels of the target image, we can observe more chaotic random walk patches that are present over the image with a tendency to an isolated structure. At the same stage of the evolutionary image processes, the images produced with middle are less patchy and have a less irregular appearance. The reason for this is that the biased random walk builds longer pads during the evolutionary transition.
Furthermore, we notice differences in the images during the evolutionary processes when and pixels of the target image occur. Here, the image transition for biased random walk with middle has more patches incorporated into the target image . Thus, the images displayed obtain a nearly chaotic state. In the examples described above, we can see that the different lengths of the biased random walk have a clear effect on the image transition process.
At the beginning, when the process of the evolutionary image transition reaches and pixels of the target image, we can observe only a few larger patches of the target image. As consequence of that, we see start image dominates during the evolutionary image transition. In the later stage, when and pixels of the target image occur, we see that the darker abstract figure is overall untouched. This is the result of the bias occurring during the transition with biased random walk mutation. In contrast to the experiments conducted with small and medium , the images are mostly completed with more recognizable elements from the starting and target image. Using biased random walk with a larger into evolutionary image transition can create images that maintain content of both images without losing the style of the images.
4.3. Impact of the Choice of in the Biased Random Walk
Additionally, we investigate the impact of the choice of the in biased random walk mutation. We want to examine how the different of biased random walk influence the generation of artistic images. We aim to understand the influence that the choice of different has on the results.
In our previous experiments described in Section 2, we used the biased random walk for evolutionary image transition and set the to 0.
The most differences can be observed in images during the evolutionary processes when pixels of the target image occur. The evolutionary image transition for our algorithm with smaller value of imply to have more patches incorporated into target image . Thus, the image occurs more finished. In the examples described above, we can assume that the different values of has predominantly influence in rather smaller degree with the chosen parameter , during the final stage of the evolutionary image transition algorithm.
4.4. Combination of Asymmetric and Random Walk Mutation
Furthermore, we explore the combination of the asymmetric mutation operator and random walk mutation. Here, we run the asymmetric mutation operator as described in Algorithm 2 and a random walk mutation every generations. We explore two combinations, namely the combination of the asymmetric mutation operator with the uniform random walk mutation (leading to the algorithm EA-AsymUniformWalk) as well as the combination of the asymmetric mutation operator with the biased random walk mutation (leading to Algorithm EA-AsymBiasedWalk). We set and which means that the process is alternating between asymmetric mutation and random walk mutation where each random walk mutation carries out 2000 steps.
In Figure 7, we show the results of EA-AsymUniformWalk and EA-AsymBiasedWalk. From a visual perspective both experiments combine the stippled effect of the asymmetric mutation with the patches of the random walk. In EA-AsymBiasedWalk there is a lower tendency for patches generated by random walks to deviate into areas of high contrast. As the experiment progresses, the pixel transitions caused by the asymmetric mutation have a tendency to degrade contrast barriers.
However, even in the final frames there is clearly more background from the target image in EA-AsymBiasedWalk than in EA-AsymUniformWalk. Moreover, there are more remaining patches of the source image near the edges of the base of the abstract figure, creating interesting effects.
As it can be observed through the experimental results, the user obtains a large variety of effects for image transition through the choice of the different parameters. Large random walk lengths in the mutation operator lead to large patches in the transition process whereas small values of imply that there are many smaller patches widely distributed across the image. This implies that the image transitions appear more evenly across the whole image. The combination with asymmetric mutation which flips single pixels allows for a further smoothening of the image transition process.
5. Feature-Based Analysis
We now analyze the different introduced approaches for evolutionary image transition with respect to some features that measure aesthetic behaviour. Our goal is twofold. First, we analyze how the aesthetic feature values change during the process of transition. Furthermore, we compare the different approaches against each other and show where they differ with respect to the examined features when used for evolutionary image transition. For our investigations, we examine the starting and target image of Figure 1, the transition of a black starting image into a white target image, and the transition of the starting image Color1 into the target image Color2 as shown in Figure 8. Taking the last two pairs of images allows us to get additional systematic insights into the process of evolutionary image transition. Note that the images of Figure 8 are only swapping the colored squares.
The set of features we use are, in order of appearance, Benford's Law (Jolion, 2001), Global Contrast Factor (Matkovic et al., 2005), Hue, and Colorfulness (Hasler and Suesstrunk, 2003). We describe each of them in the following.
The Benford's Law feature () is a measure of naturalness in an image . Jolion (2001) observed that the sorted histogram of luminosities in natural images followed the shape of Benford's Law distribution of first digits. Here we use the encoding of the Benford's Law feature based on the one used by den Heijer and Eiben (2014).
Global Contrast Factor (GCF) is a measure of mean contrast between neighbouring pixels at different image resolutions. To calculate we calculate the local contrast at each pixel at a given resolution : where is the perceptual luminosity of pixel and are the four neighbouring pixels of at resolution . The mean local contrast at the current resolution is defined . From these local contrasts, is calculated as
The pixel resolutions correspond to different superpixel sizes of 1, 2, 4, 8, 16, 25, 50, 100, and 200. Each superpixel is set to the average luminosity of the pixels it contains. The are empirically derived weights of resolutions from Matkovic et al. (2005) giving highest weight to moderate resolutions.
Figure 9 shows how the features evolve over time during the image transition process. The first column refers to the transition process of the starting and target image given in Figure 1. The second column shows the transition of a complete black image starting image to a complete white target image, and the third column shows the transition of the color starting image to the color target image of Figure 8. Each figure shows the results of 10 runs for each algorithm that we have considered for evolutionary image transition.
Considering the results for the images of Figure 1 (left column), it can be observed that the feature values for Benford's Law reduce at the first half of the transition process and increase afterwards. Furthermore, the value for the target image is quite low, but the evolutionary image transition process produces images where the value for Benford's Law is significantly higher than the one for the starting and the target image in the last third of the image transition process. In respect to global contrast, it can also be observed that the transition process creates images of higher feature value than the ones of the starting and target image. All considered algorithms follow the same pattern for these two features, but it can be observed that the pure random walk algorithms of Section 3 overall achieve higher values for Benford's Law and the combined approaches are able to obtain a trajectory of higher values for Global Contrast Factor.
Considering the features Hue and Colorfulness, the feature values are following a more direct trajectory from the value of the starting image to the one of the target. For Hue, this trajectory is also very concentrated around the linear function connecting these two values whereas for Colorfulness a strong deviation, especially for the pure random walk algorithms of Section 3, can be observed.
The transition process for the images of Figure 8 carries out a process where the feature values of the starting and target image are of the same value. Again, it can be observed that the algorithms obtain higher values for Benford's Law and Global Contrast Factor during the transition for most of the runs. An exception is the biased random walk algorithm of Section 3 that sometimes produces lower values for these two figures during the transition. Hue and Colorfulness again exhibit a more direct trajectory between the starting and target feature value with the random walk algorithms showing a stronger fluctuation and in particular lower values with respect to Colorfulness.
Considering the transition for Black to White images, it can be observed that Benford's Law and Global Contrast Factor increase during the transition process. The concentrated behaviour for Benford's Law is due to the calculation of this feature as the feature value is fully determined by the number of black and white pixels. Furthermore, there are no changes during the transition process for Hue and Colorfulness.
The feature-base analysis allowed us to explore the different approaches for evolutionary image transition over time. Based on this, the user can obtain detailed insights into the behaviour of the seven approaches with respect to a given feature. As the different feature measurers are investigated on three different images, we can see the tendency of variations in the feature values for Benford's Law and Global Contrast Factor for all approaches throughout the transition process over time. In particular, the maximum value for Global Contrast Factor obtained during the transition is significantly higher than the one for the starting and the target image. Interestingly, the feature values for Mean Hue and Colorfulness exhibit a direct trajectory from the value of the starting image to the one of the target image. In contrast, Colorfulness shows a very strong fluctuation for the random walk algorithm during the whole transition process compared to the other approaches. In summary, the different aesthetic feature analysis gives the user valuable insights into the wide range of possibilities of using different evolutionary approaches for evolutionary image transition over time.
6. Evolutionary Image Painting
We now consider how to use variations of the evolutionary image transition process for evolutionary image painting. The key idea is to make use of the biased random walk and use its behaviour of favouring similar colours. We use this property to change pixel values of a “blank” image such that it becomes similar to a given target image. A biased random walk mutation in the painting process uses for each step the same colour which is determined by the pixel of the target image where it started. Due to this, we call this process “painting” of the target image.
The evolutionary image painting algorithm is given in Algorithm 5. It is similar to the evolutionary transition algorithm and uses a starting image and the target image to be painted. Again, we assume for all pixels as pixels with can be viewed as already painted. The algorithm minimizes the number of pixels that the current image agrees with and the pixels of with can be viewed as pixels that have not yet been visted. For our investigations, we use an all white starting image as we are mainly interested in the final painted image.
The mutation operator uses the biased random walk for a given starting pixel (see Algorithm 6). As we are considering painting of an image, the starting pixel of the biased random walk determines the color in which the random walk paints the part of the image that it is visiting. The mutation operator in the evolutionary painting algorithm uses this biased painting random walk for each pixel that is still in the starting state with probability and therefore adapting the asymmetric mutation operator. It ensures that only biased random walks are started at pixels that have not changed their state yet. The idea is that pixels that have changed their state are considered as being painted and should therefore not change their color again.
If a biased random walk is started at a pixel , the color to be used during this random walk is given by the value of the pixel in the target image . During the biased random walk only pixels that have not changed their value yet are painted with the color which is again motivated by not painting pixels of the image that have already been painted. Each iteration of Algorithm 5 does not increase the number of pixels where and agree. If at least one biased random walk happens, an offspring with (see definition of in Section 2) is obtained. The implies that the algorithm minimizes the fitness function and an image with is considered to be fully painted.
We now introduce a novel mutation operator called painting mutation operator for image transition developed to one specific purpose, in particular to produce interesting artistic images imitating modernist movement in the art called expressionism. The position of the first pixel is chosen exactly the same as in our previous mutation operators, randomly. The painting mutation operator imposes a transition of the current starting image to an area of the target image . Note that the paint mutation operator given in Algorithm 6 only paints a pixel with the chosen colour if the considered pixel is in the state of the starting pixel. A different option would be to always paint with the colour irrespective of whether the pixel is in the starting or target state. This would allow that pixels already painted with a colour are able to change their colour during the process.
In our approach, we choose randomly one pixel from starting image and using the biased random walk described in detail in Section 3.2. The transition process occurs from starting image , in our case we decided the image is white, to the target image, our artistic image. Through the transition from white image to another artistic image we can clearly represent processes that occur during biased random walk mutation. It is also possible to choose two artistic images when the colors are opposite of the color theory and achieve interesting effects in respect to stronger contrast. Our investigation has additionally the goal to give adequate insight into the evolutionary processes during the transition processes.
6.1. Impact of Choice of the in Evolutionary Image Painting
We have developed the special case of biased random walk mutation algorithm with constant . The classical random walk is running with without deliberately currying about the . On the assumption biased random walk with the algorithm will with high probability perform along the edges of the similar colors as opposed to going over the edges to the colors with greater RGB values differences.
For our experimental investigations, we set which implies that several random walks painting different parts of the image are started in each generation. The goal of our experimental investigations is to study the effect of in (introduced in Section 3) for evolutionary image painting. This parameter allows to focus on similar color pixels during the biased random walk and one would expect a painted image close to the given target if is large enough.
We study our evolutionary painting approaches on two landscape pictures and two portraits. The four target images used for evolutionary painting (see Figure 10) are as follows: Seated Woman with Bent Knee, 1917 by Egon Schiele; Adele Bloch-Bauer, 1907 by Gustav Klimt; Winter Landscape, 1909 by Wassily Kandinsky; and Murnau Street with Women, 1908 by Wassily Kandinsky. These images have been selected for our studies due to the wide recognition of these images as examples of fine art in Western culture (Iskin, 2017). For our experimental investigations, we consider evolutionary painting with and . The results for evolutionary image painting with these different parameters of are shown in Figures 11 and 12.
Our experiments show that the elements emerge over the image during the transition stage resulting in differently occurring images at the end of the generation processes. Figures 11 and 12 show four various images with unique artistic value. Firstly, we see the target image , following the four adjustment for . We have executed mostly varied appearances of the target image . Each of these pictures represent different stages of the painting. Additionally, for comparison we have chosen 4 different pictures as portrait, landscape, abstract art and nature, in the stage of transition process when the painting process is completed. We can observe the impact on the evolutionary process caused by the different settings.
As controls the bias of the underlying random walks, it can be observed that small values of obtain paintings that are much less precise than the target image. Thus, it's creating coarse-grained painting effects. This effect decreases with increasing and the painted image becomes much closer to the given target image. For all considered images, already shows a painting being close to the target which is the reason why larger values of are not investigated. Comparing the different images, it can be observed that creates aesthetically pleasing paintings for the considered landscape pictures whereas a value of produces novel results when applied to the considered portraits.
6.2. Impact of Random Walk Length in Evolutionary Image Painting
Now we investigate the impact of random walk length on our evolutionary painting approaches. For our experimental investigations, consider an evolutionary painting algorithm with random walk length with , and . We choose one of the landscape pictures presented in Figure 10 and the abstract pictures presented in Figure 1, which gives us a clear comparison to the previous experiments described in Section 6.
Also, we set which again implies that several random walks painting different parts of the image are started in each generation. The goal of our experimental investigations is to study the effect of random walk length in introduced in Section 3 for evolutionary image painting. This parameter allows us to focus on similar color pixels during the biased random walk and one would expect a painted image close to the given target if is large enough.
The results for evolutionary image painting with four different values of and several lengths of random walk, which imply values of the range between large and small , are shown in Figures 13 and 14.
In the first row, we observe the image with and , following the three adjustments for . Each of these pictures represent different stages of the painting. Comparing the different images, it can be observed that , and creates more patch-based appearance of the image. We can observe the impact on the process caused by the different settings of random walk length in evolutionary image painting.
In summary, the random walk length and the choice of influences how “precisely” the target image is painted. The different parameter choices exhibit a wide range of possibilities for painting images with different effects.
7. Conclusions and Future Work
Evolutionary image transition uses the run of an evolutionary algorithm to transfer a starting image into a target image. In this article, we have investigated how random walk algorithms can be used in the evolutionary image transition process. We have shown that mutation operators using different ways of incorporating uniform and biased random walks lead to different effects during the transition process. Furthermore, we have studied the impact of the different approaches with respect to different artistic features and observed that the process creates images which significantly differ from the starting and target image with respect to these features. Using biased random walks for image painting, we have shown that this approach creates artistic paintings which can be varied by the parameter controlling the bias of the underlying random walk.
All our investigations are based on a fitness function that is equivalent to the well-known OneMax problem. For future research, it would be interesting to study more complex fitness functions and their impact on the artistic behaviour of evolutionary image transition.
This work has been supported through Australian Research Council (ARC) grant DP140103400.