Synthesizing and analyzing sound

In the Media section, I showed you how to load external sound files and play them. And hey, that was fun! But the p5.sound library has a few more tricks up its sleeve that are worth investigating.

Oscillators

Sound (to review) consists of fluctuations in air pressure; when those fluctuations occur at regular frequencies, we perceive a tone. The shape of the fluctuations affects the quality of the sound (or its “timbre,” on which more below). A microphone works by recording air pressure over time. A speaker works by moving a membrane in order to produce bursts of air pressure.

In fact, we already know how to make things happen at regular frequencies in p5.js! For example, to make the background of the sketch pulse every few frames (warning: flashing lights):

► run sketch ◼ stop sketch

function setup() {
  createCanvas(400, 400);
}
function draw() {
  background(
    map(sin(frameCount * 0.25), -1, 1, 0, 255)
  );
}

This is, in essence, a simple oscillator: a bit of code that produces a repeating series of values. If the sketch is running at 60 frames a second, this oscillator has a frequency of (60 * 0.25) / TWO_PI, or approximately 2.39repetitions every second. “Repetitions per second” is often referred to as “Hertz” (or Hz), named after this guy.

In an ideal world, we could use this same code to make sound by sending the value from the expression in the background() function directly to the speaker. The problem is that humans only begin to perceive tones when the repetitions in air pressure reach a particular frequency: forty times a second. The draw() loop runs at most 60 times per second, meaning the fastest repeating pattern you can make in draw() would work out to thirty repetitions per second. That’s not quite enough!

To get around this limitation, we use p5.sound, which in turn makes use of a low-level browser API called Web Audio. Your browser’s Web Audio capabilities take care of performing the fast math needed to produce audible tones, and other niceties.

Making tones

In particular, p5.sound has an oscillator class that you can use to generate a sound signal.

Let’s make a pure sine tone at 220 Hz:

► run sketch ◼ stop sketch

let osc;
function setup() {
  noCanvas();
  osc = new p5.Oscillator();
  osc.freq(220);
  osc.setType('sine');
  osc.amp(0.5);
  osc.start();
}
function draw() {
}

The new p5.Oscillator() call produces a new oscillator object, which has a number of methods:

.freq() sets the frequency (in Hertz) of the oscillator.
.setType() sets the oscillator type—the shape of the values in the repeating cycle.
.amp() sets the amplitude of the oscillator. Normally the values of the oscillator vary between -1 and 1; setting the amplitude to 0.5 means that the values of the oscillator will go from -0.5 to 0.5 instead.

See the reference for even more methods.

When you create an oscillator, it doesn’t start automatically; you have to call .start(). By default, the sound of the oscillator goes straight to the speakers (though you can send it elsewhere—see below). You can also call .stop() in order to stop the oscillator altogether, and check to see whether the oscillator is currently running by checking the value of the .started attribute:

► run sketch ◼ stop sketch

let osc;
function setup() {
  createCanvas(400, 400);
  osc = new p5.Oscillator();
  osc.freq(220);
  osc.setType('sine');
  osc.amp(0.5);
  osc.start();
}
function draw() {
}
function mousePressed() {
  if (osc.started) {
    osc.stop();
  }
  else {
    osc.start();
  }
}

Here’s another example, using a different oscillator type, whose frequency and amplitude you can control with the mouse. This will drive your cats crazy.

► run sketch ◼ stop sketch

let osc;
function setup() {
  createCanvas(400, 400);
  osc = new p5.Oscillator();
  osc.setType('sawtooth');
  osc.freq(220);
  osc.amp(0.5);
  osc.start();
}
function draw() {
  background(128);
  osc.freq(map(mouseX, 0, width, 100, 1000));
  osc.amp(map(mouseY, 0, height, 0, 1));
}

If you’re familiar with the Western diatonic scale and you prefer to make actual “notes” instead of arbitrary frequencies, you can use p5.sound’s midiToFreq() to convert MIDI note numbers to their corresponding frequencies. Here’s a sketch that changes the frequency of an oscillator to a random MIDI note every handful of frames. Give me some of that ol’ atonal music like daddy used to play:

► run sketch ◼ stop sketch

let osc;
function setup() {
  noCanvas();
  osc = new p5.Oscillator();
  osc.setType('triangle');
  osc.freq(220);
  osc.amp(0.5);
  osc.start();
}
function draw() {
  if (frameCount % 15 == 0) {
    osc.freq(midiToFreq(int(random(21, 96))));
  }
}

Controlling oscillators with oscillators

An oscillator is, at the end of a day, just an oscillator, and you do things with them other than send them to the speakers. For example, p5.sound makes it easy to use one oscillator to control parameters of another. In the following example, I create two oscillators: one to create a tone, and another to control the amplitude of the tone in order to create a tremolo effect. Try it out:

► run sketch ◼ stop sketch

let tone;
let tremolo;
function setup() {
  noCanvas();
  tremolo = new p5.Oscillator();
  tremolo.freq(4);
  tremolo.add(1);
  tremolo.amp(0.5);
  tremolo.setType('sine');
  tremolo.disconnect();
  tremolo.start();
  tone = new p5.Oscillator();
  tone.setType('sine');
  tone.amp(tremolo);
  tone.freq(220);
  tone.start();
}
function draw() {
}

There are a couple of tricks here. The oscillator assigned to the variable tone is producing the tone you hear; the oscillator in tremolo is passed in as a parameter to tone’s .amp() method. Now, instead of tone having a constant amplitude, its amplitude is determined by the value of tremolo. I used the .add() and .amp() methods to adjust the range of the oscillator to be between 0 and 1. Note that I also used the oscillator’s .disconnect() method to ensure that the output of the oscillator is not sent to the speakers.

You can use an oscillator to control another oscillator’s frequency as well, in order to create a vibrato effect:

► run sketch ◼ stop sketch

let tone;
let vibrato;
function setup() {
  noCanvas();
  vibrato = new p5.Oscillator();
  vibrato.freq(5);
  vibrato.amp(15);
  vibrato.disconnect();
  vibrato.start();
  tone = new p5.Oscillator();
  tone.setType('sine');
  tone.amp(0.5);
  tone.freq(220);
  tone.freq(vibrato);
  tone.start();
}
function draw() {
}

The oscillator assigned to vibrato has a frequency of 5Hz and an amplitude of 15, meaning that it oscillates from -15 to 15 five times every second. After setting an initial value for tone’s frequency, I pass vibrato as a parameter to the .freq() method, causing the frequency of tone to vary according to the value of vibrato.

Here’s an example that combines tremolo and vibrato, controlling their frequencies with the mouse:

► run sketch ◼ stop sketch

let tone;
let vibrato;
function setup() {
  noCanvas();
  vibrato = new p5.Oscillator();
  vibrato.freq(5);
  vibrato.amp(15);
  vibrato.disconnect();
  vibrato.start();
  tremolo = new p5.Oscillator();
  tremolo.freq(4);
  tremolo.add(1);
  tremolo.amp(0.5);
  tremolo.setType('sine');
  tremolo.disconnect();
  tremolo.start();
  tone = new p5.Oscillator();
  tone.setType('sine');
  tone.amp(tremolo);
  tone.freq(220);
  tone.freq(vibrato);
  tone.start();
}
function draw() {
  vibrato.freq(map(mouseX, 0, width, 12, 1));
  tremolo.freq(map(mouseY, 0, height, 12, 1));
}

I dunno. It’s something. Things to try: change the parameter you pass to .setType() from sine to one of the more exotic waveforms that p5.sound makes available to you.

Arrays of oscillators

Times like these I go back to the text of my valedictorian* speech. “Friends, the world is yours. Nothing is stopping you from having more than one audible oscillator.” The following sketch has two oscillators, whose frequency you control with the x/y position of the mouse. (* I have never been a valedictorian.)

► run sketch ◼ stop sketch

let osc1;
let osc2;
function setup() {
  createCanvas(400, 400);
  osc1 = new p5.Oscillator();
  osc1.setType('square');
  osc1.freq(220);
  osc1.amp(0.5);
  osc1.start();
  osc2 = new p5.Oscillator();
  osc2.setType('sawtooth');
  osc2.freq(220);
  osc2.amp(0.5);
  osc2.start();
}
function draw() {
  osc1.freq(map(mouseX, 0, width, 100, 1000));
  osc2.freq(map(mouseY, 0, height, 100, 1000));
}

Or we can create a whole batch of oscillators and put them in an array. The following sketch adds a new oscillator every time you click. To demonstrate how you’d make changes to these oscillators after creating them, I change the panning of each oscillator to a random value every few frames with the .pan() method:

► run sketch ◼ stop sketch

let allOscs = [];
function setup() {
  createCanvas(400, 400);
  textAlign(CENTER, CENTER);
}
function draw() {
  background(255);
  noStroke();
  fill(0);
  text("click to create an oscillator", width/2, height/2);
  if (frameCount % 30 == 0) {
    for (let i = 0; i < allOscs.length; i++) {
      allOscs[i].pan(random(-1, 1));
    }
  }
}
function mousePressed() {
  let osc = new p5.Oscillator();
  osc.setType('square');
  osc.freq(random(100, 1000));
  osc.amp(0.05);
  osc.start();
  allOscs.push(osc);
}

And the following sketch just straight up creates fifty oscillators at random frequencies. The sketch also shows (with green lines), the frequency of those randomly-chosen oscillators, mapped to the height of the sketch. (The frequency of an oscillator can be accessed with the .f attribute.) Pick new random values for each oscillator’s frequencies by clicking the mouse.

► run sketch ◼ stop sketch

let oscCount = 50;
let allOscs = [];
let minFreq = 100;
let maxFreq = 1000;
function setup() {
  createCanvas(400, 400);
  for (let i = 0; i < oscCount; i++) {
    let osc = new p5.Oscillator();
    osc.setType('sine');
    osc.freq(random(minFreq, maxFreq));
    // scale amplitude to number of oscillators
    osc.amp(1.0 / oscCount); 
    osc.start();
    allOscs.push(osc);
  }
}
function draw() {
  background(255);
  stroke(0, 255, 0);
  for (let i = 0; i < allOscs.length; i++) {
    let drawY = map(
      allOscs[i].f, minFreq, maxFreq, 0, height);
    line(0, drawY, width, drawY);
  }
}
function mousePressed() {
  for (let i = 0; i < oscCount; i++) {
    allOscs[i].freq(random(100, 1000));    
  }
}

Picking apart frequencies

That’s nice! We’re well on our way to a profitable career in noise music. Indulge me for a second and consider the following scenario: Let’s say that you released an album of recordings from that last sketch. The album is a hit and months later your agent comes to you and says, “What were the frequencies of the sine waves you used on that track? People are going crazy for them. If we make a new album with just those frequencies, you’re sure to make a million dollars.” The problem: the oscillators’ frequencies were set randomly, and you didn’t bother to use console.log() or something similar to save the frequencies so you could reproduce them later. What to do?

We can state the problem more generally like this: given some arbitrary audio signal, how can we recover the frequencies of the oscillators that produced that audio?

This might seem impossible on its face, like unmixing paint. But it can actually be done, more or less, using the power of the Fourier transform, one of the best things humans have done with math. (The “unmixing paint” metaphor comes from the 3Brown1Blue video about Fourier transforms, which I highly recommend viewing.)

Fast Fourier transform

I won’t explain here the nitty gritty of how the Fourier transform works. For our purposes, here’s what you need to know: you put an audio signal into the Fourier transform, and get back an array that tells you the frequencies and amplitudes of the oscillators which, combined, produced that signal.

Let’s look at an example. We’ll use p5.sound’s implementation of the Fourier transform (p5.FFT) to detect the frequencies a simplified version of the sketch above.

(This is a really complex example, so don’t sweat it if it seems opaque at first, and make sure that you understand the examples that came before this one before really digging in.)

► run sketch ◼ stop sketch

let oscCount = 5;
let allOscs = [];
let fft;
let minFreq = 100;
let maxFreq = 1000;
function setup() {
  createCanvas(400, 400);
  for (let i = 0; i < oscCount; i++) {
    let osc = new p5.Oscillator();
    osc.setType('sine');
    osc.freq(random(minFreq, maxFreq));
    // scale amplitude to number of oscillators
    osc.amp(1.0 / oscCount); 
    osc.start();
    allOscs.push(osc);
  }
  fft = new p5.FFT();
}
function draw() {
  background(255);
  stroke(40);
  // analyze the audio signal and draw frequency
  // intensities
  let bins = fft.analyze();
  for (let i = minFreq; i < maxFreq; i++) {
    let drawY = map(i, minFreq, maxFreq, 0, height);
    let val = fft.getEnergy(i);
    let lineWidth = map(val, 0, 255, 0, width);
    line(0, drawY, lineWidth, drawY);
  }
  // draw original frequencies
  stroke(0, 255, 0);
  for (let i = 0; i < allOscs.length; i++) {
    let drawY = map(
      allOscs[i].f, minFreq, maxFreq, 0, height);
    line(0, drawY, width, drawY);
  }
}
function mousePressed() {
  for (let i = 0; i < oscCount; i++) {
    allOscs[i].freq(random(100, 1000));    
  }
}

The new part of this sketch in draw(), starting with the call to fft.analyze(), which performs the actual analysis of the audio data (in this case, the audio that is being produced by the oscillators). The fft.getEnergy() function takes a numerical parameter that specifies a frequency and evaluates to the amplitude of the oscillator producing a tone at that frequency. In the code above, I loop over every integer frequency from minFreq (the minimum value used in the random() function to select random oscillator frequencies) and maxFreq (the maximum value for the same), and then draw a grey line from the left side of the sketch to the right side according to the value returned from fft.getEnergy() for that frequency. The loop right below draws the actual frequency of all of the oscillators.

You’ll see that they mostly match up! Where the green lines are (i.e., the actual frequency of the oscillators), you’ll also see the longest grey lines. So it worked! Sorta. You’ll notice that the results of fft.getEnergy() are kind of chunky—that’s because p5.FFT is performing what’s called a Fast Fourier transform (which is why it has that extra F at the beginning). The Fast Fourier transform algorithm is fast enough to run on every frame, but
it can’t tell you the specific frequency of component oscillators. Instead, it breaks up the audible spectrum (up to 44.1kHz) into an array of evenly-spaced bins, giving you the approximate amplitude of component oscillators whose frequencies fall in that range.

So our noise music agent is out of luck, but at least we can make some good guesses.

FFT on other signals

It turns out that any audio signal can be analyzed as the result of playing a bunch of sine wave oscillators on top of each other at various frequencies and amplitudes—even complex noises like voices, drums, meows, etc. It also turns out that Fourier transforms of audio are useful for all kinds of computational tasks with audio, from visualization to classification. (It furthermore turns out that Fourier transforms and other related algorithms can be used to analyze all kinds of data, not just audio—the JPEG compression algorithm is based on discrete cosine transform, a close relative of Fourier transforms.)

As an example, let’s visualize the audio coming from the microphone using a fast Fourier transform. Getting audio data from the microphone with p5.sound is easy: simply create a new p5.AudioIn object. (Note that the microphone’s output won’t be sent to your speakers by default.) Then pass that object to p5.FFT’s .setInput() method, so it performs its calculations on the microphone data and not on the audio output of the sketch (as would otherwise be the case):

► run sketch ◼ stop sketch

let mic;
let fft;
function setup() {
  createCanvas(400, 400);
  mic = new p5.AudioIn();
  mic.start();
  // second param sets number of bins
  fft = new p5.FFT(0.5, 64);
  fft.setInput(mic);
}
function draw() {
  background(255);
  fill(40);
  noStroke();
  let bins = fft.analyze();
  for (let i = 0; i < bins.length; i++) {
    let drawY = map(i, 0, bins.length, 0, height);
    let val = bins[i];
    let rectWidth = map(val, 0, 255, 0, width);
    let rectHeight = height / bins.length;
    rect(0, drawY, rectWidth, rectHeight);
  }
}

In the sketch above, I’m using the values from p5.FFT’s bins directly, instead of calling .getEnergy(). I also used the parameters to p5.FFT’s constructor to change the number of bins. (There are 1024 bins by default; because of the implementation of FFT, the number of bins has to be less than or equal to 1024 and also a power of 2, e.g., 64, 128, 256, 512, 1024).

A similar sketch, but now “smearing” the FFT values across the screen, slitscan-wise, using the bin value to set the color:

► run sketch ◼ stop sketch

let mic;
let fft;
let xoff = 0;
let xstep = 3;
function setup() {
  createCanvas(400, 400);
  mic = new p5.AudioIn();
  mic.start();
  // second param sets number of bins
  fft = new p5.FFT(0.5, 256);
  fft.setInput(mic);
  background(0);
}
function draw() {
  noStroke();
  let bins = fft.analyze();
  for (let i = 0; i < bins.length; i++) {
    let drawY = map(i, 0, bins.length, 0, height);
    let rectHeight = height / bins.length;
    fill(bins[i]);
    rect(xoff, drawY, xstep, rectHeight);
  }
  xoff += xstep;
  if (xoff > width) {
    xoff = 0;
  }
}

What we just made is called a spectrogram: a visualization of frequency intensities of a signal over time. Nice! In this tutorial, I’ve tried to explain how Fourier transforms work, and shown a few examples of how to visualize sound with the results of a Fourier transform. Further applications are a bit beyond the scope of this tutorial, but having an understanding of how Fourier transforms work is super important! They find applications in all kinds of audio processing tasks, including classification tasks like peak detection and pitch detection.

Exercise: See if you can visually identify vowel formants in an audio signal. You may need to reduce the number of bins that you draw, focusing on lower frequency sounds. (Remember, FFT analyzes frequencies even up in the 44.1kHz, where it’s difficult for the human ear to make fine distinctions.)

Analyzing amplitude

An even more basic way to analyze audio is to calculate how loud it is. There are several ways to do this, and of course what counts as “loud” is subjective. The p5.Amplitude object provides a simple and easy to understand measure of loudness: call the .getLevel() method to average the amplitude of a few hundred samples of the signal, then return the square root (yielding a value between zero and one). Here’s an example using microphone input.

► run sketch ◼ stop sketch

let mic;
let amp;
let xoff = 0;
let xstep = 3;
function setup() {
  createCanvas(400, 400);
  mic = new p5.AudioIn();
  mic.start();
  amp = new p5.Amplitude();
  amp.setInput(mic);
}
function draw() {
  noStroke();
  fill(40);
  let level = amp.getLevel();
  // adjust map values to taste; actual levels
  // tend to be between 0 and 0.5
  let barHeight = map(level, 0, 0.25, 0, height);
  rect(xoff, height - barHeight, xstep, barHeight);
  xoff += xstep;
  if (xoff > width) {
    xoff = 0;
  }
}

Resources

p5.sound is a feature-packed library and we have barely scratched the surface of what’s possible. Browse around and find something fun to play with (envelopes! distortion! convolution! sequencing!).
Jason Sigal’s notes for Visualizing Music with p5.js are amazing and have many helpful code examples.