Ceci Capstone GUI

In Golan Levin's Fall 2023 Creative Coding course, I used p5.js to create a gesture expander that rendered in real-time. The project used the via the MIT ML5 Bodypose Keypoints library. Ceci Sun, a friend and dancer at Johns Hopkins University, performed choreography to one of my favorite songs at the time, "Motion Picture Soundtrack" by Radiohead.

When Ceci and I caught up in Winter 2025, she asked if I would be interested in creating a new version of the project for her senior thesis performance. Given that it had been two years since we last collaborated, and newer technologies had become available, I was really excited to create an improved version.

I had two options for connecting MediaPipe to TouchDesigner. TouchDesigner's Web Render node can display a webpage as a texture, which would show the MediaPipe visualization directly. With this approach, I would receive an image as an input. The alternative was using OSC to send the numerical pose data. I decided to use OSC because it has a lower latency and the programming workflow of using variables as opposed to encoding and decoding an image felt more intuive to me. Additionally, I could create effects using placeholder values in TouchDesigner and then replace them with the OSC values later.

MediaPipe is really powerful because it can retreive a lot of information super quickly. The framework provided three parameters for all 33 joints and deciding which ones to use so that the mapping would look intuitive to the audience was an interesting challenge. When connecting the data to TouchDesigner, I observed that the all of the data had a natural jitter, so it turned out that only one to two parameters was enough to communicate the overall movement of the choreography.

I decided to use the distance between two fixed points to drive the animation. For example, the distance between the left hand and the left shoulder is easy for the dancer to control. The slider in the p5 GUI would control how sensitive the mapping was.