Draw With Your Body App

Idea

For this part of the project I decided to work on combining physical computing and machine learning with ml5js and simple controller. The idea was to create a drawing app where you would have to draw with different parts of your body and switch between those parts with a physical controller.

How It Works

For the software part, I used ml5js, specifically PoseNet to recognize different points on your body.

You can find the source code to the software part with ml5js in my GitHub repository.

The p5 Sketch

I started out with imitating the controller's buttons using my keyboard and made a quick sketch with posenet and dots to see whether you would be able to draw with your body at all. You can see the initial result in the video below.

The drawing effect was achieved by adding every dot into an array of brush strokes, which contains the dot's position, size of the brush on the position, as well as the RGB values.

// save the strokes position and color here
let strokes = [];
let brushSize;

...

function draw() {
  ...

  if (poses.length > 0) {
    let pose = poses[0].pose;

    let nose = pose['nose'];
    strokes.push([nose, brushSize, [nose.x/3, 0, nose.y/2]]);
  }

  ...
}

You can see the full code in the GitHub repo.

Physical Controller

For the physical controller I used three switches and a circle potentiometer to control the points on the body and the size of the brush respectively.

I chose the circle potentiometer as I wanted the smooth change in the brush size while drawing. After multiple trials, it also turned out more convenient to use the middle finger for the brush size while se your thumb to press on the switches to change the part you want to draw with.

The code for the Arduino was relatively simple. As p5.serialcontrol can only accept bytes as data from the Arduino, all the values had to be converted into a string and then passed to Serial.println(s). I converted the values of the switch to a and b for HIGH and LOW values respectively. The potentiometer's value was passed as it is to the array of characters.

void setup() {
  Serial.begin(9600);
  pinMode(14, INPUT);
  pinMode(15, INPUT);
  pinMode(16, INPUT);
}

void loop() {
  int switches[3];
  switches[0] = digitalRead(14);
  switches[1] = digitalRead(15);
  switches[2] = digitalRead(16);
  char s[] = "";
  char ch;

  for (int i = 0; i < 3; i++) {
    if (switches[i] == HIGH) {
      ch = 'a';
    } else {
      ch = 'b';
    }

    strncat(s, &ch, 1);
  }

  int potentiometer = analogRead(A3);   
  int mappedPot = map(potentiometer, 0, 1023, 0, 255);
  char potchar[12];
  sprintf(potchar, "%d", mappedPot);
  strncat(s, potchar, sizeof(potchar));

  Serial.println(s);    
  delay(1);
}

The string would then be parsed in a separate function in the p5 sketch.

let anchorsSwitch = [false, false, false];
...

function serialEvent() {
  // inData = String.fromCharCode(Number(serial.read()));
  var inString = serial.readStringUntil('\r\n');

  //check to see that there's actually a string there:
  if (inString.length > 0 ) {
    anchors = inString.slice(0, 3).split('');
    brushSize = inString.slice(3, inString.length);
  }
}

function checkAnchors() {
  if(anchors != undefined) {
    if (anchors[0] == 'a' && anchorsSwitch[0] == false) anchorsSwitch = [true, false, false];
    if (anchors[1] == 'a' && anchorsSwitch[1] == false) anchorsSwitch = [false, true, false];
    if (anchors[2] == 'a' && anchorsSwitch[2] == false) anchorsSwitch = [false, false, true];
  }
}
...

Where anchorsSwitch represents three switches and parts of the body. For this version I decided to use only nose, left and right wrists to draw with PoseNet.

This project is being extended as the part of the final project.