ITP Blog

Dancing Drawing Robot — The Process

ICM + Physical Computing — December 10, 2020

The Idea of Drawing with Your Body

The Dancing Drawing Robot was created to help people feel more comfortable about their bodies, about their movements — about being weird sometimes ;)

Some people love expressing themselves while dancing; some people do it while drawing. We thought, why don't we combine those two and create something fun?

Dancing Drawing Robot

To control the robot, try to recreate the given poses. The robot can move from left to right and has two arms that can rotate 180 degrees.

Drawing Robot on the Table Drawing Robot Top Part with Pens


Pose classification and user interface by Yon. Physical computing by Stuti. Fabrication by Yona and Stuti.

Fabrication and Training the Model


We brainstormed a lot on what kind of housing we had to make to fit all the parts of the robot. The robot consisted of 4 DC motors with the wheels and 2 servo motors holding the arms with the pens so we decided to go for the boxy shape with wooden arms.

Drawing Robot Drawing Robot Drawing Robot


It was also our first experience using the laser cutter and with fabrication in general.

Drawing Robot Drawing Robot

We managed to fit all the motors but ruined at least 3 DC motors before that with soldering. It was also a long process of designing the "tower" that would hold the pens.

Drawing Robot Drawing Robot Drawing Robot

Training the Model

To train the model to classify poses we used ml5js and PoseNet to collect data, train it and deploy. We concluded that as it was PoseNet, it was enough to use only one person to gather the data. So the first model was trained on multiple poses, made by one single person.

Drawing Robot Drawing Robot

But turned out the model was not precise enough because we did not consider the difference in heights as well while doing the pose classification.

Here is the code for the data collecting process that collects 34 points for 10 classes.

We ended up retraining the model on 6 different people with different heights and then combining all the data files into one to train the model.

Drawing Robot

The files were so large that I also ended up writing a python script that would parse them.

import json

def write_json(data, filename='data.json'):
    with open(filename,'w') as f:
        json.dump(data, f)

with open('data.json') as f:
  data = json.load(f)["data"]

newData = data

for i in range(1, 32):
    with open(str(i) + '.json') as f:
      appendedData = json.load(f)["data"]

    newData += appendedData

jsonData = {"data": newData}

After getting a single json file we put them through the training with 200 epochs with ml5js. You can find the code for the model training here.

Drawing Robot

With more data we managed to get more precise results.

Drawing Robot Drawing Robot Poses Classifier in Action

We used 10 dance moves from famous TikTok dances as references for the poses.

Dancing Drawing Robot

In Progress Work

Currently we are moving the robot parts into a bigger box (which means more fabrication, yey!) and also working on the UI part for a better user interaction.

Drawing Robot

© built during the year of masks and social distancing by yonaymoris with Gatsby