Working with video

← back to Creative Computing

by Allison Parrish

The p5.js framework makes it easy to control video playback from your sketch. Let’s take a look!

Video and web browsers

When we talk about video on computers, we’re usually talking about video files: a file that contains the encoded data of a moving image. There are many different formats for “encoding” video, and still more “container” formats (essentially metadata “wrappers” around encoded video).

Creating software that can successfully recognize the myriad formats and containers for video is a big job, and in particular writing software that can efficiently turn a video file into actual images on the screen (a process known as “decoding”) is a difficult task. For this reason, p5.js doesn’t provide its own video playback code. Instead, it “piggy-backs” on your web browser’s built-in video decoding capabilities.

In a previous tutorial, we used functions from the p5.dom library to add controls like sliders, buttons and input areas to the page containing a p5.js sketch. The process for adding simple video playback to a sketch looks very similar: we use the createVideo() function from the p5.dom library to add a video “control” to the page. Here’s a simple example:

let vid;
function setup() {
  createCanvas(0, 0);
  vid = createVideo("iwaswrong.mp4");
  vid.loop()
}

To get this example to work, you’ll need a few things. The first is this video, which is a tiny clip from Sutherland Production’s classic capitalist propaganda film, “Destination Earth”. You’ll also need to have a copy of the p5.dom library. (This should be included by default when you make a sketch in the p5 app, but you’ll have to include it separately if you’re working on bare HTML/JS files.)

You’ll notice that I called the createCanvas() command with dimensions of 0, 0. That’s because the video you’re seeing isn’t being played inside the sketch—it’s being played in an HTML <video> element on the same page, next to the sketch. In fact, our sketch code isn’t doing anything here other than creating the video tag (with createVideo) and then calling the video tag’s .loop() method, which causes the video to loop indefinitely. We’ll discuss other methods of the video object below.

A note on formats

As mentioned above, the state of video formats and their support in web browsers is kind of a huge mess; in my opinion, it’s an more confusing by an order of magnitude than the audio format support boondoggle that we discussed in a previous tutorial. In this tutorial, I’m using .mp4 files, because they are widely supported (begrudgingly, in some cases) by most web browsers. Mozilla has a good rundown of which browsers support which formats. For the widest possible compatibility, p5.js supports an alternate form of the createVideo() command that allows you to specify more than one file format, in case the first format doesn’t work:

`let vid = createVideo(['vid.mp4', 'vid.webm']);`

This function will attempt to load vid.mp4, and then will attempt to load vid.webm in the event that the browser doesn’t support videos in MP4 format.

If you want to make your own video to include with your sketch, you’ll need to save it as an MP4 and possibly in WebM format as well. This article is a bit old but has some good rules of thumb for creating video media for HTML5 and troubleshooting common problems. If you’re more technically inclined, you might look into learning a bit about ffmpeg, which is essentially a “Swiss Army knife” for converting video between different formats.

Controlling and querying video objects

You can do more with a video element than just include it on the page. In the following example, I show how to create a sketch that gets information about a video as it’s playing and how to pause and unpause the video by clicking the mouse:

let vid;
let playing = false;
let completion;
function setup() {
  createCanvas(400, 100);
  vid = createVideo("iwaswrong.mp4");
  vid.size(400, 300);
}
function draw() {
  background(50);
  completion = vid.time() / vid.duration();
  ellipse(completion*width, 50, 20, 20);
}
function mousePressed() {
  if (!playing) {
    vid.play();
    vid.time((mouseX/width) * vid.duration());
    playing = true;
  }
  else {
    vid.pause();
    playing = false;
  }
}

There are a number of new things in this example, and I’ll explain them in turn!

• The .size() method of the video object controls the dimensions of the video object. I made it a bit wider than it was tall.
• The .time() method returns the number of seconds elapsed in the video, and the .duration() method returns the total number of seconds in the video. Dividing time by duration gives us a percentage of completion (i.e., a number from 0 to 1, where 0 is the beginning of the video and 1 is the end).
• The .play() and .pause() methods of the video object play and pause the video (respectively). Video objects don’t come with a method that tells us whether or not the video is currently playing, so you need to keep track of that yourself. (In this sketch, I used a boolean variable called playing for this purpose.)
• The .time() method, when called with a parameter, causes the video to seek to the given position, as given in seconds. In this sketch, we multiply the video’s duration by the ratio of the mouse position to the width of the sketch, which has the effect of moving the video’s playback to proportionally with the X position of the mouse.

The following sketch seeks to a random place in the video every few seconds, and adjusts the volume at random:

let vid;
let playing = false;
let completion;
function setup() {
  createCanvas(400, 100);
  vid = createVideo("iwaswrong.mp4");
  vid.size(400, 300);
  vid.play();
}
function draw() {
  background(vid.volume()*255);
  if (frameCount % 120 == 0) {
    vid.time(random() * vid.duration() - 2);
    vid.volume(random());
  }
  ellipse((vid.time()/vid.duration())*width,
    50, 20, 20);
}

The .volume() method of the video object sets its volume (0 is silent, 1 is full volume); you can set the volume by calling .volume() with a parameter (ranging, likewise, from 0 to 1).

The reference for p5.MediaElement has a complete list of the methods you can call on video objects.

Working with pixel data from videos

Playing videos is boring, like I can do that on my phone. Whatevs. I understand this, so I want to show you something a bit more interesting: how to get pixel data from a video. In the draw() function, you can call a video’s .getPixels() function, which makes it possible to use the .pixels attribute of the video to get the color at a particular position in the current frame of the video. Here’s an example:

let vid;
function setup() {
  createCanvas(320, 240);
  vid = createVideo("iwaswrong.mp4");
  vid.loop();
  vid.hide();
  noStroke();
}
function draw() {
  background(0);
  vid.loadPixels();
  for (let y = 0; y < height; y += 8) {
    for (let x = 0; x < width; x += 8) {
      let offset = ((y*width)+x)*4;
      fill(vid.pixels[offset],
        vid.pixels[offset+1],
        vid.pixels[offset+2]);
      rect(x, y, 8, 8); 
    }
  }
}

This example is super tricky, mostly because of that weird .pixels thing. What is it? What does ((y*width)+x*4 mean? The .pixels array is designed for maximum performance, and so it’s a one-dimensional array. (1D arrays are faster and more memory-efficient in Javascript than arrays, which are actually just arrays that contain other arrays.)

Here’s how it works: Pixel data in .pixels is arranged such that the red, green, blue and alpha values of each pixel are stored as separate items. The first four items are the RGBA values for the pixel at 0,0; the second set of four items are the RGBA values for the pixel 1,0; the next four are for the pixel at 2,0; etc. When a row of pixels on the screen ends, the pixel data starts over again at 0,1 (and then 1,1 and 2,1, etc.). Essentially, it looks like this:

In the example above, the two for loops iterate from zero to the width and height in both dimensions. The expression ((y*width)+x)*4 gives the offset of the four values that correspond to the color of the pixel at x and y. Then, I set the fill color to the red value for that pixel by getting the value at vid.pixels[offset], the green value for that pixel by evaluating vid.pixels[offset+1], and the blue value by evaluating vid.pixels[offset+2]. (Also note that I changed the size of the sketch to the size of the underlying video in order to make all this math easier.)

Here’s the same example, but this time using separate “steps” in the X and Y directions, and using the green value of the pixel to control the size of each rectangle:

let vid;
function setup() {
  createCanvas(320, 240);
  vid = createVideo("iwaswrong.mp4");
  vid.loop();
  vid.hide();
  vid.volume(0);
  noStroke();
  rectMode(CENTER);
}
function draw() {
  background(50);
  fill(255);
  vid.loadPixels();
  for (let y = 0; y < height; y += 10) {
    for (let x = 0; x < width; x += 5) {
      let offset = ((y*width)+x)*4;
      rect(x, y, 10,
        10 * (vid.pixels[offset+1]/255));
    }
  }
}

Capturing video from a webcam

Thanks to HTML5, it’s possible to easily capture video from the user’s webcam and include it in the sketch. Here’s a simple example:

let cap;
function setup() {
  createCanvas(400, 400);
  cap = createCapture(VIDEO);
  cap.hide();
  imageMode(CENTER);
}
function draw() {
  background(50);
  image(cap, mouseX, mouseY, 160, 120);
}

(When you run this sketch, you’ll be asked to allow the server to use the camera. Click “yes” or the examples won’t work!)

The createCapture(VIDEO) function returns a “capture” object, which works a lot like a video object. You can use it in the image() function to draw the image from the video capture to the screen.

The capture object also supports .loadPixels(), which you can use to create weird video mirrors:

let cap;
function setup() {
  createCanvas(400, 400);
  cap = createCapture(VIDEO);
  cap.hide();
  rectMode(CENTER);
  noStroke();
}
function draw() {
  background(50);
  fill(255);
  cap.loadPixels();
  for (let cy = 0; cy < cap.height; cy += 10) {
    for (let cx = 0; cx < cap.width; cx += 5) {
      let offset = ((cy*cap.width)+cx)*4;
      let xpos = (cx / cap.width) * width;
      let ypos = (cy / cap.height) * height;
      rect(xpos, ypos, 10,
        10 * (cap.pixels[offset+1]/255));
    }
  }
}

(Note that in this example, the size of the video capture is greater than the size of the sketch, so we have to do some fancy math to get pixel data from the capture but draw that data to the screen at the right position.)