Pixelated People

The Idea

Dave and I wanted to create something pixelated using machine learning and as ml5 got the new trained features with Yolo and Coco SSD we decided to go for it.

How It Works

We created a p5 sketch and used the example with Coco SSD code to see whether it works first. The coordinates of the rectangle are used to create the pixelated effect. Initially it was supposed to be the same pixelated image of an object but because Coco SSD and WebGL in particular work pretty slowly with object recognition, the pixelated code would work faster which causes the "broken TV" effect like you can see on the demo.

References

1. Coding Train video on Coco SSD
2. ml5js documentation
3. Coco SSD code example

Demo

We also found out the p5js get() worked inconsistently and most of the time would cause the browser to freeze. Therefore instead of get() we manually used an individual pixel value, extracted from the pixel array (s.o. to my friend Jonathan Park for the help 🙏🏻).

function pixelStuff(x, y, w, h, n) {
  for (let i = x; i < x + w; i += 15) {
    for (let j = y; j < y + h; j += 15) {
      let index = (i + j * w) * 4;
      fill(
        video.pixels[index],
        video.pixels[index + 1],
        video.pixels[index + 2],
        video.pixels[indlex + 3]
      );
      noStroke();
      rect(i, j, 15, 15);
    }
  }
}

You can find the source code of the project on Github and try the demo yourself here.