Cartoon Beats Reality

The Web Audio API is an incredibly fun audio playground, widely supported in most browsers. You can quite easily do things like build synthesisers and sound effects, or slice up audio samples – in JavaScript code, in your web browser.

There’s also a companion API for sending and receiving MIDI events. This allows you to play your Web Audio synthesiser using a physical (piano-style) keyboard, or send midi notes and controller information out to play other instruments or devices, either hardware or other software.

I’ve been experimenting with these APIs because I want a more flexible way to sequence and trigger musical parts live.

In my app I have various patterns of music. Some contain audio events – for example a drum beat, made of different drum sounds triggered in Web Audio in the browser. Others contain MIDI events – for example, a bassline. These MIDI patterns are sent out to other synth software – they don’t make sound in the browser.

What’s the problem?

When I’m playing my different patterns, I want the notes to line up in sync.

The Web Audio API and Web MIDI API use different scheduling, so things don’t line up by default.

This post explains how to get these to play back in sync so I can combine audio events and MIDI events in a performance or piece.

How to sync audio and MIDI

Web Audio events are scheduled relative to when the AudioContext started, in seconds.

Web MIDI events are scheduled relative to when the page started loading, in milliseconds.

So we have two differences to account for:

• Seconds vs. milliseconds – e.g. x 1000
• When the page started loading vs when the AudioContext started

This second item is the tricky one – the AudioContext doesn’t necessarily start when the page loads, could be much later.

We can measure this difference by using the high resolution time API, and comparing that to the current AudioContext time.

const perfNow = window.performance.now()
const audioNow = audioContext.currentTime;
const audioContextOffsetSec = ( perfNow / 1000.0 ) - audioNow;

This tells us how late audio events are relative to MIDI or real time. (MIDI events are sent in close to real time.)

So to sync we need to offset (delay) MIDI events by this latency:

const timestamp = ( startSeconds * 1000 );
const offset = ( audioContextOffsetSec * 1000 );
midiOutPorts[0].send(
  [ 0x90 + 0, midiNote, 100 ],
  timestamp + offset
);

For a long time I had this backwards – I’d schedule my audio events earlier by audioContextOffsetSec, trying to account for the latency, but this breaks down when you are scheduling close to now. The AudioContext can only schedule so far in advance.

I’ve put up a complete example on GitHub as a demo – take a look.

For a deeper dive on how to build a reliable, accurate sequencer in Web Audio and JavaScript, check out A Tale of Two Clocks on HTML5 Rocks. Spoiler alert: there are more than two clocks.

Hopefully this article helps someone – it took me a while to get my head around this. Although the Web MIDI API is still experimental, I’m really excited to see what apps and tools will emerge in its wake.

🚤 💻 🎛

a loop-based generative audio-visual experiment

A while ago I decided to relax a bit more with my music production. The idea was to remove the pressure to finish and release tracks, and just noodle around and make loops. If I had fun jamming, I’d render the parts as a loop, archive it, and move on.

It was really liberating to focus in on short sections of music, and not concern myself with arrangement.

I might set out to make an “east coast whine” sound, and end up with a little trap/hip hop loop (Kufca). Or I might want to make a classic “synth strings” sound, and then turn that into a vocal-sample driven 90s big-beat loop (Mivova).

Around the same time, I was experimenting with the Web Audio API. It’s really powerful, making it easy to slice up audio files, sequence audio and midi, and even build custom synthesis and effects processing.

I’ve also been dabbling in ways to combine audio and (primitive) animation. Wouldn’t it be cool to jam out on a sequencer or midi controller, and synthesise visual and audio content at the same time?

How the audio works

There are currently nine songs/loops of material set up. Each song is divided into four stem parts – sometimes a single instrument (drums, bass, or vocals), sometimes a combination.

I’ve named the different mix layers using words that you might use to navigate a mountainous landscape:

• alpine – typically drums
• ridge – typically bass
• uplands – lead, chords, synth
• hills – vocals, texture, arpeggio, pad

These aren’t hard & fast categories; for example, some songs have different drum patterns in alpine and ridge. The idea here is that if you combine say a ridge and alpine from one song and hills from another, it won’t sound terrible 🙂

The playback is automated and has some gentle randomisation so it’s not exactly the same each time.

• Playback happens in 64 beat cycles.
• The order of the songs is shuffled. 🗂
• For each song, a part is selected at random in each cycle, until all parts are playing.
• Then the parts are removed in a random order.
• When there’s only one or two parts of the outgoing song playing, a new song kicks in. This means parts from two different songs will overlap – like a DJ blend.
• Each part might fade in or filter in slowly over the cycle, or start fully turned up. Similarly, parts may fade or filter out. 🎛
• When we run out of songs, we shuffle the order and start again. 🎲

To add a little punctuation & spice to this, I’ve added in a handful of sound effects. Some are recordings of cutting card or ripping paper, but there’s also an airhorn. These are randomly sprinkled on cycle changes, especially if a song is changing. These are fed into a dub-delay effect.

There are also some texture samples – maybe birds, a dripping tap, the ocean, or sea scouts around a campfire. When the song changes, a beat-synched chunk is sampled and looped from one of these textures, to add another layer to the transition.

The mixing and effects processing happens live in your web browser, using Web Audio nodes – filters, delays.

How the animation works

The mountains are React components containing SVG. There are four mountain ranges, to match the four mix layers. If a ridge (e.g. a bassline) part is playing, you’ll see a moderately tall mountain range. When parts fade in or out over the cycle, the mountains fade in too.

Each mountain is an SVG polygon, and the points are animated, parallax-scrolling style. There’s a codepen prototype here.

When a song comes in, random colours are picked for the front (hills) & back (alpine) range. The intermediate two colours are interpolated.

What next?

Occasionally I may add new songs to Padded Landscape. One song will get released on Cartoon Beats later this year – Redline Train, based on the Maenyb dub / reggae loop. Keep an ear out for it – this will include a remix from nsu (of Newclear Music).

If you need a low-maintenance soundscape…
fire up the site and click play!

A little while ago we had some work done on our house to make it better.

We swapped the windowless, dank bathroom out to the large sun-filled laundry (with a vaulted/angled ceiling), extended the hallway to the back of the house, and added a toothbrush nook.

Next to our kitchen there was a nice chunk of space, with a view out to the back yard (and forthcoming deck). I was really keen on turning this into a dining booth – a bit like a cafe/diner booth, but a bit bigger.

So I borrowed a circular saw, started measuring, and built it!