Dancing Drawing Robot — The Process

The Idea of Drawing with Your Body

The Dancing Drawing Robot was created to help people feel more comfortable about their bodies, about their movements — about being weird sometimes ;)

Some people love expressing themselves while dancing; some people do it while drawing. We thought, why don't we combine those two and create something fun?

To control the robot, try to recreate the given poses. The robot can move from left to right and has two arms that can rotate 180 degrees.

Roles

Pose classification and user interface by Yon. Physical computing by Stuti. Fabrication by Yona and Stuti.

Fabrication and Training the Model

Sketching

We brainstormed a lot on what kind of housing we had to make to fit all the parts of the robot. The robot consisted of 4 DC motors with the wheels and 2 servo motors holding the arms with the pens so we decided to go for the boxy shape with wooden arms.

Fabricating

It was also our first experience using the laser cutter and with fabrication in general.

We managed to fit all the motors but ruined at least 3 DC motors before that with soldering. It was also a long process of designing the "tower" that would hold the pens.

Training the Model

To train the model to classify poses we used ml5js and PoseNet to collect data, train it and deploy. We concluded that as it was PoseNet, it was enough to use only one person to gather the data. So the first model was trained on multiple poses, made by one single person.

But turned out the model was not precise enough because we did not consider the difference in heights as well while doing the pose classification.

Here is the code for the data collecting process that collects 34 points for 10 classes.

We ended up retraining the model on 6 different people with different heights and then combining all the data files into one to train the model.

The files were so large that I also ended up writing a python script that would parse them.

import json

def write_json(data, filename='data.json'):
    with open(filename,'w') as f:
        json.dump(data, f)

with open('data.json') as f:
  data = json.load(f)["data"]

newData = data

for i in range(1, 32):
    with open(str(i) + '.json') as f:
      appendedData = json.load(f)["data"]

    newData += appendedData

jsonData = {"data": newData}
write_json(jsonData)

After getting a single json file we put them through the training with 200 epochs with ml5js. You can find the code for the model training here.

With more data we managed to get more precise results.

We used 10 dance moves from famous TikTok dances as references for the poses.

In Progress Work

Currently we are moving the robot parts into a bigger box (which means more fabrication, yey!) and also working on the UI part for a better user interaction.