Learn to Animate

Lip syncing

Lip syncing is the process of synchronizing audio with the animation so that the sound fits with the visual movement. When lipsyncing isn't just right, the effect is jarring on the audience.
Rhubarb Lip-Sync is a command-line program that automatically generates a data(text) file with the timing for a single audio file. That .dat file can be loaded into OpenToonz and applied to a mouth level to very quickly and accurately sync the movement of the mouth to the audio.
  1. Download Rhubarb
    1. Get Rhubarb from Google Classroom or from github
    2. Using Finder, move Rhubarb to the Desktop
    3. If Rhubarb did not unzip automatically, double-click on the zipped folder
  2. Setup the audio files:
    1. Place your audio file on the Desktop if it is not there already
    2. If your audio file has any spaces or symbols in the name, rename it now to remove any spaces or symbols (dashes - or underscores _ are fine)
    1. Open Terminal (either searching in Spotlight or the F4 shortcut (inside the "Other" folder)
    2. Open the Desktop directory by typing:
      cd ~/Desktop
      and press enter
    3. Now type the following without pressing enter:
      1. rhubarb-lip-sync-1.10.0-osx/rhubarb
        - this is the path(location) of rhubarb within the open directory (if using a later version of rhubarb the version number will change)
      2. add a space and then type:
        -o youroutputname.dat
        - this is the name/path of the data file to output (make sure to change the name, especially when syncing multiple files)
      3. add a space and then type:
        -f dat
        - this is the format for the output
      4. add a space and then type:
        --datUsePrestonBlair
        - this specifices the phoneme/mouth set (right now OpenToonz only supports Preston Blair)
      5. add a space and then type the name of your audio file - ie:
        youraudiofilename.wav
        - this assumes that the file is located in the Desktop directory
    4. Once the entire line looks like this (with your own output name and audio file name):
      rhubarb-lip-sync-1.10.0-osx/rhubarb -o youroutputname.dat -f dat --datUsePrestonBlair youraudiofilename.wav
      press enter
    5. The generated .dat file will be located on the Desktop - go to the OpenToonz tab for further directions
    Papagayo is a free program that can give you more control over the process of lip syncing. Text is lined up with audio and then the program generates a text file that can be used in animation programs. If you are not comfortable using a command-line program like Rhubarb, Papagayo is a great free option.
    Visit the Edit Audio page to learn about loading and making basic edits to the audio track.
    One of the major approaches to creating mouth shapes to pair with spoken dialogue was created by Preston Blair, a noted animator that worked at a few animation studios including Disney over the course of his career. OpenToonz currently requires the Preston Blair mouth shapes for automated lip-syncing. Preston Blair's page on Dialogue is missing
    While the mouth shapes could be drawn in any order, it saves time to draw them in a precise order.
    1. Create a level with a single frame
      • While any level type could be used, Vector or Toonz Raster levels are recommended for labeling purposes
    2. Draw the mouth shape for AI
      1. Create an extra style just for labeling - something bright and easy to see (ie. red) that will not be used elsewhere is recommended
      2. Using the type tool - or just writing normally, use that color to write a label that says "AI" large and off to the side
    3. Add 9 more frames in the level strip (it is easier to sync if most of the mouth drawings are not exposed in the Xsheet)
      1. Right-click in the empty space at the bottom of the Level Strip
      2. Select "Add Frames"
      3. Set the new frames from 2 to 10 and select "Add"
    4. Draw the mouth shapes for the new frames - make sure to label each one with your special label style selected
      1. Turn on onion skin to keep the mouths positioned properly
      2. Each mouth shape should be drawn on it's own frame in the following order:
    The example image is missing
    1. AI
    The example image is missing
    2. E
    The example image is missing
    3. O
    The example image is missing
    4. U
    The example image is missing
    5. FV
    The example image is missing
    6. L
    The example image is missing
    7. MBP
    The example image is missing
    8. WQ
    The example image is missing
    9. Etc. (C,D,G,K,N,R,S,TH,Y,Z)
    The example image is missing
    10. Rest (the default shape when not speaking)

    Do not rely too much on examples of other artist/animator's mouths - look in the mirror (or use Photo Booth on the Mac) to see what your own mouth looks like as it makes each sound.

    Before starting this process, make sure that only one frame for your mouth level is exposed in the Xsheet/Timeline (delete the other frames from the Xsheet if necessary).
    1. Load your audio column if it is not already exposed in the Xsheet/Timeline (right-click on the cell the audio should begin and select "Load Level")
    2. Select the mouth column (click on the column/layer header)
    3. In the program menu, go to Xsheet → Apply Lip Sync Data to Column
    4. Use the arrows underneath each icon to change drawings if necessary to correct any errors with the mouth shape pairings
    5. Select the lip sync data file (.dat) from Papagayo at the bottom of the lip-sync pop-up
    6. Select "Apply"
    For optimal lip syncing, audio should be slightly delayed from the mouth movement.
    1. As necessary, adjust the position of the audio level or the mouth level so that the sound begins a frame or so after the mouth movement (using the top or left dragbar)
    2. Scrub and/or play the audio to make sure the timing is correct
    3. Make adjustments to timing/placement as necessary
    4. If any mouth shapes are incorrect:
      1. Select the cells for the incorrect drawing
      2. Use the hotkey "Q" to switch out the drawing for the correct drawing (moving backwards)
      3. Use the hotkey "W" to switch out the drawing for the correct drawing (moving forwards)
    5. Repeat as necessary
    6. When the syncing is complete, in the Style Editor, turn the opacity (A/alpha) down to 0 for the special label style to hide the labels

    Create an animation with at least 2 lines of synced dialogue. The animation should include:

    1. An audio track:
      1. At least 2 lines of original, recorded dialogue
      2. Audio track edited & background noise filtered out in Audacity
    2. 1 or more characters
    3. At least the face of each character
    4. At minimum - the character's mouth should move in time with the audio
    5. Add movement to the character's jawline and eyes to add additional believability (optional)
    6. Color & background (optional)

    If you have no dialogue planned as part of the scene you are animating, this can be a standalone animation.