2D Ball Bounce Eval

Motion, LLM Evaluation
Timeline: 2 weeks
↗ GitHub

Background

Why ball bounces?

A ball bounce is a beginner animation exercise that teaches students how to use arcs, timing and spacing, and squash and stretch to simplify how objects move in the real world. Unlike a physics simulation, curves are flexible and can be manipulated to make motion realistic or exaggerated. A skilled animator can make a form read like a heavy bowling ball, a floaty beach ball, or a bouncing snowman with extra airtime for dramatic effect.

Setup

Each rollout gives the model a task outlining the viewport size, ball's start and end position, and animation duration. The model is asked to create a CSS ball bounce animation with keyframes and easing. The model has access to tools read_file/write_file/list_dir/bash/finish and can check their work by calling render_graph, which shows a graph plot of their work and a filmstrip.

Rubric

The rubric items are adapted from this tutorial created by Moshe Mahler. Each rubric item is graded pass/fail.

  1. Contact points have broken tangents and apexes are smooth
    Rubric 2
  2. Attempts at least 3 bounces
  3. At a given contact point, the ball's path does not bulge past its handlebars
    Rubric 1
  4. Translate X is linear or subtle ease-out
    Rubric 3
  5. Every Y key is either a contact or apex, no in-between keys allowed
    Rubric 4
  6. At most two X keys, indicating a smooth path that is easily modifiable
  7. No stray keys on unused channels such as scale or rotation
  8. The final X key lands either on the last contact of the last bounce, or one last-bounce length past that point, indicating the ball rolling to a full stop
  9. The time of each bounce decreases at a constant rate
    Rubric 7
  10. Contact points have handles of length greater than zero

Results

I ran 4 models x 5 seeds via OpenRouter. The results generally fell into two camps. Either the model created created a convincing looking bounce that had too many keyframes, or the model used the correct number of keyframes for the Y channel and the X-movement wasn't smooth.

Rubric 7
Rubric 7

To my surprise, all of the models followed rule #3, which specifies that the ball path must stay inside the contact point handlebars. Students often struggle with this concept at first because it's possible to make a convincing ball bounce without following this rule.

All of the models failed rule #9, which specifies that the time of each bounce decreases at a constant rate. In real life, the time succeeding ball bounces usually decreases by a multiplicative factor, but in animation, this is approximated with subtracting a fixed number. This is because it is more intuitive for a human animator to check their work with subtraction as opposed to multiplication, and subtraction ensures that the numbers always stay whole integers associated with frames.

All of the models' work had the same number of X keys and Y keys, breaking the rule that for all channels, the number of keys should stay relatively minimal. This makes human collaboration difficult because if a human wanted to modify the end-position of the ball's bounce but keep the overall travel shape, with respect to time, the same, instead of moving one keyframe, they would not only have to move about triple the number of bounces' worth of keyframes, but also fit them to their desired shape by hand.

This may be because CSS keyframes make separating channels inconvenient. Thus, it would be a good idea to run the same experiment across different frameworks.

Next Steps

The vanilla ball bounce exercise is effective because it is easy to spot when the animation looks wrong and there are few enough moving parts that the exercise stays focused on understanding the graph editor. Its simplicity lends itself well to be extended towards more complex tasks. Some ideas of future evals include:

  • Testing other animation frameworks such as GSAP and Lottie
  • Animating different ball typies, such as a ping pong ball, bowling ball, or beach ball
  • Model is given a scene where they fix or restyle an existing ball bounce
  • A ball obstacle course with a single ball, multiple identical balls, heavy/light balls, etc.
  • Attempting any of what's listed above in a 3D scene in three.js

Zooming Out

Online discourse treats the problem of creating beautiful things as an impossible task, but like many other fields that AI has been capable of learning, successful motion is made up of small components that can be verified. Given the ephemeral nature of design trends, it's really interesting to think about what the lowest common denominators that have persisted throughout design history are, and speculate whether or not they will hold in the future.

Automating motion design is just one example of how AI is changing what it will mean to work as a designer. The creative world is headed in a direction where it will be difficult for us to go back to a time without artificial intelligence, but we can steer progress so that it does not just benefit a small group of people.