Investigating the Effect of Jo-Ha-Kyū on Music Tempos and Kinematics across Cultures: Animation Design for 3D Characters Using Japanese Bunraku Theater

Many unanswered questions remain about how humans form emotions through interaction and communication. In human-robot interaction, there is a phenomenon called the uncanny valley. The idea suggests that objects such as robots can make people feel uneasy or repulsed as they become increasingly human-like [1]. Research has indicated that the uncanny valley effect also occurs in animation and games when 3D characters resemble humans [2]. One potential explanation for these feelings is that robots or animation characters’ lack of nuanced, naturalistic movement causes uneasiness or revulsion following unconvincing presentations likening them to actual humans [3]. This leads us to conclude that interaction design between humans and animated characters is challenging and relies heavily on trial and error.

Puppet performances take many forms around the world [4], and even if the puppet looks like a human, it moves in such a way so as not to make the audience feel uncomfortable. “Ningyo Jōruri Bunraku,” or simply “Bunraku,” is one of Japan’s traditional performing arts involving puppets. It is recognized in UNESCO’s Representative List of the Intangible Cultural Heritage of Humanity. In Bunraku, there are three types of performers: (1) three ningyozukai (puppeteers) who manipulate a single puppet, (2) a gidayū (chanter), and (3) a musician playing shamisen (a three-stringed Japanese guitar) on one stage in a special puppet theater, as shown in Fig. 1.

Bunraku has significant potential to prevent the uncanny valley effect, as suggested by robotics professor Masahiro Mori’s conceptual framing [5]. Mori argues that Bunraku puppet movements resemble human movements more than humanoid robots and avoid making people feel the discomfort common in the effect. Mori’s research also pointed out that audiences tend to become absorbed in Bunraku in puppet theater and thus might develop a considerable affinity for the puppet. Therefore, as discussed below, investigating the interaction mechanism of Bunraku puppets may help us apply interaction design techniques to approaches by animators and musicians in animated character design.

Bunraku plays often generate empathy and strong emotions among the audience using a principle called Jo-Ha-Kyū in Japanese—opening, middle, and climax (end) in English—which underlies the artistic modulation of music tempos in traditional Japanese performances generally [6]. Jo-Ha-Kyū is considered one of the critical contributors to Bunraku’s resistance to the uncanny valley effect by the previous study. In a 2020 article, Dong et al. proposed a method to generate Jo-Ha-Kyū motions in robots using a convolutional autoencoder to train them to use only Bunraku puppet motions [7]. However, in addition to motion (visual), sound (audio) also plays an essential role in animation [8]. Furthermore, there has been no technical study on the Jo-Ha-Kyū mechanism in Bunraku performing techniques using synchronized music and movements.

We analyzed the Jo-Ha-Kyū mechanism based on previous research [9,10]. As performance techniques in Bunraku theater create Jo-Ha-Kyū patterns recursively and fractally, we looked at puppet motions and music tempo in both “long-term” Jo-Ha-Kyū—corresponding to the story level or the development of the narrative—and “short-term” Jo-Ha-Kyū, corresponding to the key-frame motion level (the movements divided by beats (key-frame) of the gidayū and shamisen), to devise a method of creating Jo-Ha-Kyū in 3D character animation.

European and American music uses the golden ratio to design instruments and create melody [11]. Japanese music uses a similar frequency mechanism in instrument design as instruments as Western music [12]. However, while frequencies in Western music tend toward the pitch standard (i.e., A440, which is the standard pitch corresponding to 440 Hz), Asian music, like traditional Japanese music, has no pitch standard and is indeterministic [13]. In American music, frequency changes express emotions, while this is not the case in Eastern music, such as Carnatic music in Asia [14]. In contrast, traditional Japanese music usually uses a rhythm or tempo change (based on Jo-Ha-Kyū) instead of a pitch to generate emotions from the audience [15,16].

The present research focuses on rhythm and tempo because these are the two most integral components with regard to the art concept of Jo-Ha-Kyū and incorporate easily into computer algorithms. Dramatic speed changes in the music and corresponding choreography are defining features of Jo-Ha-Kyū in Bunraku [17]. As shown in Fig. 2, each Jo, Ha, and Kyū can be divided into smaller Jo, Ha, and Kyū. Short intervals or breaks between the tempo modulations are called Ma, which refers to the breaking of rhythm. Thus, we define Jo-Ha-Kyū here, in motion and music, at both the long-term level corresponding to the story and the short-term level corresponding to the key-frame, which is the beat point of the gidayū and shamisen.

Humans simultaneously use facial expressions, eye movements, and gestures to express their feelings. Instead of facial expressions, Bunraku puppeteers express the puppet’s emotions using their unique manipulation techniques utilizing Jo-Ha-Kyū [18]. One of the most complex components of the Bunraku puppet is its head. As shown in Fig. 1, the master puppeteer uses sophisticated controls to generate head motions to lead sub-puppeteers to synchronize their puppet’s movements, which express various emotions [19]. The hip joint is the center of the hierarchical body structure, which is the origin of the puppet’s body movements. Therefore, this paper primarily analyzes the head and hip movements in a Jo-Ha-Kyū-based Bunraku puppet performed by Kanjuro Kiritake, a top-ranking master puppeteer certified as a Living National Treasure in Japan.

The gidayū (chanter) primarily controls the head movement. To perform a Bunraku play using Jo-Ha-Kyū principles, the gidayū has their tonal center, which changes and breaks tempo according to story narration (jōruri). The gidayū modulates the tempo to lead the entire performance, synchronizing the puppets’ motions (choreography) and the shamisen music (rhythm) to the narrative while interacting with the audience.

This study focuses on a scene performed in Jo-Ha-Kyū called “Sugisakaya,” from the famous Jōruri performance Imoseyama Onna Teikin (gidayū: Tsukoma Takemoto; shamisen: Sosuke Takezawa; puppeteers: Kanjuro Kiritake and two others). We adopted both optical and magnetic motion-capture systems to obtain high-accuracy motion data. To capture the tempo of the play, we use spectrogram analysis to detect beats in the shamisen music and the gidayū’s chanting [20].

Since Jo-Ha-Kyū patterns are recursive, we first analyzed the full duration of the “Sugisakaya” chapter (totaling 140 seconds) to observe Jo-Ha-Kyū in the long term. Figure 3a shows the time sequence of angular velocities, or joint rotation speed, of the puppet’s head and hip joints. Figure 3b shows the variations in tempo over time. We use a five-second window (reflecting the average length of the gidayū’s chanting) moving average filter for both selected motion sequence and music to obtain average tempos.

As seen in Fig. 3, the angular velocities of the motion and the tempo of music can broadly be divided into three Jo-Ha-Kyū parts, with two short breaks (ma) in between them. The correlation coefficient, an index that measures the strength (from weak 0.0 to strong 1.0) of a linear relationship between two data, between the motion speed of the puppet and the tempo of the music, was r = 0.53. This indicates that puppet movement in the scene is synchronized with the long-term tempo, so the puppet motions reflect the Jo-Ha-Kyū principle.

Just as American and Japanese traditions historically have different ways of evoking emotion through music [21], they also have other mechanisms of choreography. Although there are puppet theaters in European and American society [22], in this study, to help identify the Jo-Ha-Kyū principles that are typically expressed in Bunraku plays, we compared the motion speeds (the combined angular velocities of both head and hip) of the Bunraku puppet with those of a European and American mechanism-based dance by the Japanese pop trio Perfume. Perfume is known for its futuristic, robot-like dancing in line with their musical style [23]. This comparison presents a potential method of instituting Jo-Ha-Kyū in character animation.

Figure 4 shows the angular velocities of the two main points (head and hip joints) in the choreography (160 seconds in duration) for Perfume’s song Enter the Sphere, which is composed by Yasutaka Nakata and has a consistent tempo measuring 130 beats per minute [24, 25].

As seen in Fig. 4, Perfume’s dancers are moving almost constantly. Comparing Fig. 3 with Fig. 4 reveals that Perfume’s rate of motion is not significantly modulated, as are Bunraku puppets. In Bunraku, the puppet motions largely follow the tempo of the gidayū’s chants or narrations, which are continuously modulated [26]. European and American mechanism-based dance performances do not change tempos in short timeframes. In comparing Fig. 3 with Fig. 4, it can be concluded that the Bunraku puppet choreography follows Jo-Ha-Kyū throughout the “Sugisakaya” chapter. At the same time, no Jo-Ha-Kyū can be found in the motion-speed analysis of Perfume’s dance.

We then analyzed the keyframed (gidayū and shamisen’s beat segmented) Bunraku choreography data and compared them with those of Perfume’s dance in the short term, searching for indications of Bunraku techniques. Figure 5a shows the four keyframed Bunraku movements for approximately two seconds. Figure 5b shows the four keyframed speed variations of the head, divided up by beats (vertical lines). Figs 5c,d show the trajectory of these motions in 3D. The different segmentations divided by beats correspond to different keyframed movements. The size of the arrow is proportional to the speed of action. As indicated in Figs 5b–d, the head speed changes at every keyframe. Figures 5c and 5d show figure-8 trajectories, making it quite surprising that the directions and velocities of the arrow change smoothly and continuously at every keyframe, even in a two-second duration. It is also remarkable that each keyframe length varies.

To analyze Jo-Ha-Kyū in the short term, we analyzed the keyframed motions of Perfume’s choreography over 1.5 seconds. Figure 6a shows three characters designed using the 3D animation software MikuMikuDance and based on Perfume’s motion capture data. Motion capture often provides a poor anatomical fit (unable to precisely model the human body, such as bone shape and skeletal rotation angle) with the corresponding models [27]. In Asia, European and American mechanism-based dance motions are usually applied to these cartoon characters known as “Japanese pop culture” instead of realistic characters to avoid the uncanny. As the tempo of European and American music does not often change over time, the length of time for each keyframe is mostly consistent. As shown in Fig. 6b-d, the beats (vertical lines in Fig. 6b) segmented the motion, and the average speed remained nearly constant across keyframes.

Since the tempo of Perfume’s dance is relatively consistent, the average speed of head movements does not change across the keyframed motions (segmentations divided by beats in Fig. 6), making the difference between Jo-Ha-Kyū and non-Jo-Ha-Kyū motions noticeable. It is also surprising that Bunraku puppeteers can produce finely controlled affective puppet’s movements. Therefore, the Jo-Ha-Kyū principles can be central to overcoming the uncanny valley effect with representational character animation.

To show that these principles can be used today, we apply Jo-Ha-Kyū to a 3D character animation. First, we demonstrated the earlier results of professional musician Hiromi Shinoda playing European and American mechanism-based percussion. Next, to verify that our results can help animators and musicians to create more accurate character animations, we asked the musician to create two new versions of the music of “Sugisakaya,” one employing Jo-Ha-Kyū and another without.

Figure 7 shows the spectrograms of the “Sugisakaya” chapter recreated by Hiromi Shinoda. Figure 7a shows the spectrogram for the composition based on Jo-Ha-Kyū principles. Since the tempo changes in both the short and the long term, we can see that the beats are recursively distributed from the Jo part to the Kyū part. In contrast, Fig. 7b shows the new “Sugisakaya” spectrogram without Jo-Ha-Kyū principles. Thus, the beat is constant compared to Fig. 7a.

Next, we applied the puppet motion-capture data to an animation character. Since the heroine in the “Sugisakaya” chapter is a young girl, we choose Yuno Takasaki [28], a similarly aged animation character designed in present-day CG style, as shown in Fig. 8a. Figure 8b shows the character’s average head motion speed divided by beats shown in Fig. 7a, where Jo-Ha-Kyū was integrated and synchronized with the music. As seen in Fig. 8b, head speed changes in each beat interval, representing the development of keyframe motions, and arrows also change, representing the development of narratives. As a result, short-term Jo-Ha-Kyū in keyframe motion formed long-term Jo-Ha-Kyū at the story level. Then, to verify the Jo-Ha-Kyū, we created two animations using the same character movements (Fig. 8) with two different pieces of recreated (same Jo-Ha-Kyu pattern but different melody, instrument, and narrative) present day music style (Fig. 7). Comparing the character animation with Jo-Ha-Kyū and without Jo-Ha-Kyū reveals that the perception of specific emotions being expressed between the two is entirely different. These results are significant for animators and musicians creating character animation.

This study identifies Jo-Ha-Kyū principles that could be implemented for 3D animation characters to avoid the uncanny valley effect. We analyzed Jo-Ha-Kyū-based motions of Japanese traditional puppet theatre, called Bunraku. Then we compared them with contemporary Japanese trio Perfume’s music and dance. Jo-Ha-Kyū principles modulate the tempo structure of the performance and thereby shape its affective power according to the story. Our conclusions can be summarized as follows:

• While European and American mechanism-based dance motions maintain consistent tempo and motion speeds, Bunraku performances shift continuously and smoothly to align the emotions with the narrative.

• Bunraku puppets’ motions change speed, synchronizing with the gidayū chanting tempo and shamisen music following Jo-Ha-Kyū. This allows the puppeteers to adjust their movement continuously and recursively following the story. This makes possible expressive puppet movements throughout the entire play.

• Our 3D character animation results using Jo-Ha-Kyū principles reveal that our study could help designers create Jo-Ha-Kyū in 3D character animation.

As the Jo-Ha-Kyū patterns of tempo and motion in Bunraku techniques are only based on experience, it is challenging to generate said techniques without the many years of training that Bunraku puppet masters receive. However, our results demonstrate that a professional designer could establish Jo-Ha-Kyū by consulting our results. Although our present research requires designers to create Jo-Ha-Kyū motions and music manually, it may be possible to analyze Bunraku and European or American puppet theaters to provide more nuanced comparisons making possible AI automation that implements Jo-Ha-Kyū principles.

This work was supported by JSPS KAKENHI Grant Numbers JP16H01804, JP20K12525, JP20K23352, and JP21K17833.