Updated 2014.04.25: a new version of the waterfall display is available, give me feedback! Link in the article. A new updated article will follow.

Waterfalls: not the river kind, but the audio kind. Also known as sonograms, those are 2D+1 representations of the spectrum of a signal: the X axis usually represents the frequency, the Y axis the time (or the other way around), and the power of each frequency is drawn on a color gradient.

In the ham radio world, waterfalls are a great way to visualise part of the frequency spectrum. Depending on the type of radio you have – SDR or not, really – , you can either display a fairly wide part of the spectrum, or just a few kilohertz. But in any case, this visualisation will give you a new way of understanding what is going on around you, RF-wise.

Below is a good example on a busy band, over a 96kHz bandwidth: you can see lots of CW towards the left, PSK31 around the center, RTTY just on the right side, packet radio higher up, and so on.

Screen Shot 2014-02-03 at 16.50.26

You can learn more on how to create such displays on your computer in this article.

fldigi and digital modes

Another great use for waterfalls, is to help out identify digital signals on narrower parts of the spectrum. A good example is how fldigi uses this to both visualise and select signals for decoding. Below are a few examples, my article on digital modes contains more screenshots.

PSK31 Other example

PSK31

Olivia 16-500 on 20m

Olivia 16-500 on 20m

Creating my own waterfall

While the ‘fosphor’ display looks awesome, it requires gnuradio and a good OpenGL graphics card. The fldigi waterfall is good too, but really, I wanted to create my own version.

Since I do most of my development in javascript and HTML5 these days, I thought it would be interesting to see what can be done in a standard web browser: with technologies such as WebAudio and the latest advances in the HTML5 canvas, I was curious to see if I could create something decent in a web page.

My goal here was to create my own configurable waterfall display using the radio audio output in a first phase – I have not implemented IQ output yet. The radio is my Elecraft KX3. The audio bandwidth on USB/LSB is about 4kHz max, so the waterfall should display that audio range.

Web Audio

The key technology I leverage here is Web Audio. Pretty much every decent browser today implements it – Internet Explorer of course doesn’t, but seriously, who care? Web Audio gives us a lot of interesting capabilities:

  • Access to your audio input (strictly speaking, this is the media stream interface, not the web audio API)
  • A mechanism to implement processing of audio samples in javascript
  • A modular audio routing system
  • Quite a few off the shelf filters and goodies, including cool stuff like native FFTs.

The Canvas API, on the other side is, all things considered, a fairly high performance interface for doing 2D graphics in a browser. There is also a way of doing 3D using OpenGL – called, WebGL – which I have not worked on yet, but which is also powerful.

getUserMedia

The first step is to get the audio from the sound card: pretty easy, the getUserMedia API takes care of this for us:

Like most javascript APIs, everything is callback-based. The “audioConstraints” object lets us specify exactly what sort of media we want: in our case, we just say “Audio”, but we could just as well be more precise and request a specific device. Here, we are just getting the default input device – defined at the operating system level.

Audio processing graph

Now that we have an audio stream, we can start doing things with it: the objective is to get a good waterfall of the 0 to 4kHz or so range, since this is the bandwidth of our KX3 audio.

The Web Audio API interface enables us to do a maximum 2048 point FFT, so ideally, we would need to get a 4kHz bandwidth signal using a 8kHz sampling rate, then do a Fourier transform on it. This way we would spread the 1024 “bins” of the FFT over just 4kHz of spectrum, which amounts to about 4Hz resolution.

This is where things get hairy: WebAudio, believe it or not, was not designed for amateur radio use, but mostly for high quality audio. One important thing we are lacking, is the ability to specify the sample rate of the audio stream, which is a bit of a problem since we only want a 4kHz bandwidth on our waterfall.

The solution I found was to manually resample the audio input. Comments on the chromium bug reporting interface about the lack of control on the sampling rate of the audio pointed towards https://github.com/grantgalitz/XAudioJS project which has implemented a very fast resampler. Using the Audio Processing node, it is fairly straightforward to pipe the audio input into the resampler:

Explanation of the above: we initialise a resampler inside of the closure context, then return a callback which gets an audio processing event: this event contains an input buffer, and an output buffer. Those two references are read-only, which means that we can only modify the contents of the buffers, but not the buffer references themselves, which explains why we manually copy  the resampled buffer into the output buffer.

One big issue: now that the audio is resampled from 44.1kHz to 8.82kHz, the 8192 bytes of the input buffer are going to shrink to only 1638 bytes – the sampling rate was divided by five. This means that the output buffer will contain bursts of audio followed with silence – and this is going to disturb the FFT which we will do right afterwards:

Now our analyser is doing its transforms over chunks of 2048 bytes, which over a period of four chunks will be mostly 1638 bytes of sound then silence for the next three chunks.

We want to make sure we display the chunks with sound and not those with only silence, so we now need to find a way to synchronise the display of the waterfall lines. We will use another script processor node for doing this:

The script processor will get data synchronously after the analyser,  in chunks of 2048 bytes, so given the 1/3 ratio of sound to silence in the data after the resampler, we will only get actual data to draw every fourth call. Unfortunately, this is not so simple: depending on the browser/computer load, some samples can be missed – ie some callbacks don’t fire during high load, which would translate to an audio glitch if we played the audio back – , so any attempt to systematically trigger an FFT analysis every fourth call will end up getting desynchronized and you will draw only silence every once in a while.

For this reason, a bit of overhead goes into the ‘drawSpectrogram’ routine below:

Drawing the spectrogram

I used resources in the article called Exploring the HTML5 Web Audio to get my inspiration on how to draw the spectrogram below:

Very straightforward: iterate over the frequency values of ‘array’, draw them on a canvas line, and translate down. I have not covered a few things here: I used the “chroma.js” library to create a nice color gradient from the frequency power:

I have left a few gradient variations in the comments. The one which is active simulates the fldigi gradient I currently use.

Results

The code extracts above are just that, extracts. You can check out the whole proof of concept on Github for a working version. Tested on Chrome only, again don’t try with IE, but Safari and Firefox should be fine. The HTML5 APIs used are fairly cutting edge, so things are bound to break at some point… You need to serve the web pages through a browser, by the way, a local file won’t work for security reasons.

Do you want to try it right away? Go to the test page here! Please report whether this works for you. A recent version of Chrome is a must. It should also work on Android tablets, by the way.

Initial results are pretty good: below is a simple example of a PSK31 signal on FLDigi on top, and the waterfall.html script at bottom:

FLDigi on top, javascript version at bottom

FLDigi on top, javascript version at bottom

I will insert other screenshots when I do further work on the script:

Update: one new screenshot taken a few minutes ago on 20m where the band seems to be quite open:

Nice waterfall display on 20m

Nice waterfall display on 20m

There are some issues with this waterfall though, the first of which being that I have not been able to find a satisfactory way of handling resampling in a graceful way along the processing graph. As far as I can tell, Web Audio implementations not only are unable to manage sample rate in general, but the closed object which can define a sample rate can generally only go down do 22kHz, not below.

It should be possible to process a fourier transform entirely in javascript, of course, but one of the good things with the implementation above, is that the FFT is done natively by the browser using Web Audio, which gives us better performance. In fact, the waterfall display does not require much more than 10% CPU on Chrome on my Macbook pro.

Next steps

I have embedded this waterfall display in a web app I wrote, and it works fairly well. The real lingering issue being the lousy handling of resampling. My next steps will be:

  • Find a better way to handle resampled data – either by processing FFT in pure javascript, or other.
  • Make the waterfall resolution/size configurable more easily.
  • Add a graticule display – potentially make it a flot graphs plugin?
  • Use WebGL to improve performance and lower battery usage when running on embedded devices.
  • Add hooks for digital mode decoding!

A few evenings/nights of fun in perspective, in the mean time I hope you enjoyed this ham-oriented hack with Web Audio. If you have any idea of how I can improve this implementation, please let me know!

12 comments on “A pure javascript audio waterfall / panadapter

  • April 15, 2014 at 19:21
    Permalink

    Really well done and well explained. Love it!
    Many compliments and keep on with those interesting projects.
    Andrea

    Reply
  • April 15, 2014 at 19:25
    Permalink

    Feedback for test page, running on Firefox on Mac: Did not run

    webkitGetUserMedia threw exception :TypeError: navigator.webkitGetUserMedia is not a function

    Thanks
    Niel
    WA7SSA

    Reply
    • April 15, 2014 at 22:04
      Permalink

      Excellent, thanks for the link! That’s the next step: implement signal processing directly in Javascript.

      And yes, I realised I refer to “webkit” explicitely for the getUserMedia call, so it is bound to fail on Firefox. I’ll update the script when I find a minute, thanks to all who pointed out the issue.

      Reply
  • April 15, 2014 at 23:06
    Permalink

    Well, you can write shaders in WebGL. I really like the fosphor display and have been wishing for it in other applications. Fosphor easily handles many megasamples/sec on GPU hardware. Surely in js/webgl it would handle 8ksps.

    Reply
    • April 16, 2014 at 18:44
      Permalink

      Absolutely! I know nothing about WebGL so far, it is mainly a matter of having time to ramp up… If you want to help, the code is on github, it would be excellent to create a Fosphor-like webgl display for the I/Q output of the KX3.

      Reply
  • April 24, 2014 at 20:51
    Permalink

    Ed,

    Another way to supply a full buffer of data from the resampler to the FFT would be to keep a circular buffer of the last 8192 decimated results, and copy the entire buffer to the output.

    var resamplerNode = keep(context.createScriptProcessor(8192,1,1));
    var circularBuf = new Float32Array(8192);
    var bufptr = 0;

    resamplerNode.onaudioprocess = (function() {
    // We're getting a 4500Hz bandwidth which is enough for the bandwidth
    // of the audio of our receiver (4kHz max).
    // The output data will be 1638 or 1939 samples long (8192 samples divided by 5)
    var rss = new Resampler(44100, 8820, 1, 1639, true);
    return function(event) {
    var inp, out;
    //console.log(event);
    inp = event.inputBuffer.getChannelData(0);
    out = event.outputBuffer.getChannelData(0);
    var l = rss.resampler(inp);
    // the "out" variable is a reference to a Float32Array.
    // We can edit the values of the array, but not change the reference -
    // if we do, it won't do anything, the audioBuffer will keep its internal
    // reference: therefore we need to manually copy all samples:
    for (var i=0; i < l; ++i) {
    circularBuf[bufptr++] = rss.outputBuffer[i];
    bufPtr &= 8191;
    }
    var tptr = bufptr;
    for (var i=0 ; i < 8192 ; i++) {
    out[i] = circularBuf[tptr++];
    tptr &= 8191;
    }
    };
    }());

    With this method, your FFT is for the current frame, plus the four previous frames.

    Reply
    • April 25, 2014 at 23:39
      Permalink

      Thanks Bob! You are quite right. I have actually done quite a bit more work on this proof of concept, stay tuned for an update: I have not only used a ring buffer as you describe, but I have moved away from the analyser node which did not give me enough control, so my latest iteration lets you change resampling on demand and select various FFT windowing functions, smoothing, etc… Stay tuned!

      Reply
  • December 6, 2014 at 11:17
    Permalink

    Hey man, awesome post! I found this really helpful. Any suggestions on how to make this more configurable for frequency resolution (sample rate) and frequency range? I’m interested in creating a higher resolution graph over a narrower range. I’ve managed to manipulate the code to zoom in on an arbitrary frequency range, but I’m not sure how all this resampling stuff works.

    If you could just give me an example of how to do a different resolution and range than the one in this tutorial, I could probably figure it out.

    Thanks anyways though. This is really cool and really fun to play with! Great work.

    Chet

    Reply
  • June 13, 2015 at 02:28
    Permalink

    I have built a transceiver front panel using Websockets and the Web Audio API. This will go on the Katena (Whitebox) radio produced by Algoram, and we’ll also offer it to other manufacturers. Two-way audio and transceiver control work fine. I didn’t use WebRTC because there’s no small embedded library to run it yet, only node.js, and I have to fit this _in_ the transceiver. I currently have a demo at server1.perens.com . For my version, the FFT is going to be done in the radio, and the data will be passed back to the web client via websockets. I’ll look over your code

    Thanks

    Bruce Perens K6BP

    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *