Testing Jamulus Latency on Raspberry Pi

Similar to JackTrip, Jamulus is an open source project that uses the Jack Audio Connection Kit. Clients stream their audio to a server, and the server mixes all the audio together and streams the result back to each of the clients. I built Jamulus on my Raspberry Pi by following these instructions, and tested it as a potential alternative to JackTrip.

JackTrip also supports a serverless peer-to-peer (P2P) model, where every client streams their audio directly to every other client. This is used by JamKazam and other online jamming solutions. If you have to open a port in your firewall, you are likely using P2P. While P2P doesn’t require a server, it pushes a lot more work to the clients. The compute processing and bandwidth (both upload and download) required for each client increases proportionally to the number of participants. Given today’s common laptop/desktop processing capabilities, and home Internet connection speeds, the P2P model does not scale well beyond 6 or so participants.

Jamulus has many advantages over JackTrip. It’s been around longer and seems to have a more robust community. I was surprised in particular to see how many public Jamulus servers are already running and available. I even found one close to me in San Francisco with a 10ms round-trip ping time. Perhaps most significantly, it seems far more stable. JackTrip crashed numerous times during the testing I’ve done so far. I’ve had no trouble with Jamulus, at least so far (knock on wood).

Jack streams lossless audio, even at high definition bitrates. Jamulus streams lossy audio, encoded using the Opus codec. This isn’t so much a drawback as a tradeoff. Opus is designed specifically to maximize quality while minimizing latency. Jamulus audio may be lossy, but at least to me, it still sounds very good.

Using “normal” quality, Jamulus consumes about 220 Kbps bandwidth (per client, both ways). Even using “high” quality, it only consumes about 300 Kbps. JackTrip consumes about 890 Kbps bandwidth for 48k, or 1.75 Mbps at 96k. My clients had a CPU load of about 0.4 (40% of 1 CPU core), and my server had a very low load of about 0.05 (5% of 1 CPU core). Compared to JackTrip, that puts a little more load on the clients (where we have a lot of excess) for a lot less load on the server. This suggests it could scale to handle many more concurrent participants than JackTrip, given the same server resources.

Of course, it would have to distribute work evenly out across threads for this to be true. My top analysis for 2 concurrent clients didn’t give a strong indication either way. Jamulus uses multiple threads, but it’s not clear to me how well it’s distributing the work per client:

11197 root 20 0 78632 33092 17212 S 7.3 1.7 0:27.15 Jamulus
11224 root 20 0 78632 33092 17212 R 1.7 1.7 0:06.01 QThread
11229 root 20 0 78632 33092 17212 S 0.7 1.7 0:02.84 CHighPrecisionT

I tested Jamulus latency using jack_delay with the same EC2 virtual machine and Raspberry Pis (with HiFiBerry DAC+ ADC) as my previous tests. I found configuring a loopback client for this test to be extremely challenging. On my first attempt, all the audio was being excessively garbled. After a lot of troubleshooting, I discovered this was caused by Jamulus’ local mixer. By default, Jamulus plays back all audio coming from yourself. You can use the local mixer in the user interface (UI) to mute your own audio, but it’s not obvious or consistent who “you” are. Sometimes “you” are the left-most channel, and other times the opposite. I had to use the UI to manually mute local capture on each client to clear up feedback from the loopback audio.

Unlike JackTrip, Jamulus provides a very nice UI. It does have a “–nogui” command line option that enables it to run headless. There is also a “–inifile” option that allows you to point to a configuration file. One can use the UI to create a configuration file with the desired settings, and load these back in when running headless. Unfortunately, I haven’t been able to find a “don’t monitor my own audio” setting. Its default behavior of enabling local monitoring, combined with not having a way to disable this other than via the UI, may unfortunately make it unusable for headless applications. Monitoring your own audio is widely known to give a bad experience. I found it difficult to even carry on a conversation over Jamulus while this was happening. Even the Jamulus documentation warns against doing this, so it’s quite surprising to me that the default behavior is setup this way.

Jamulus only seems to work if Jack is configured with a frequency of 48k. I tried using a frames per second of 64, but this caused excessive overruns. All my testing of Jamulus therefore uses a frequency of 48k with frames per second of 128.

Jamulus lets you configure the size of server-side and client-side jitter buffer for each participant, and has a feature to automatically adjust it for you to minimize dropouts. I used jack_delay to test the latency when using different jitter buffer sizes, with identical values set for both clients). Here are the results:

Client Jitter BufferServer Jitter BufferLatency
AutoAuto2206 frames (46ms)
221950 frames (41ms) *dropouts
442462 frames (51ms)
882718 frames (57ms)
Round-trip loopback latencies over the Internet

All in all, these results are very similar to JackTrip latency with the same frequency & frames per second. But Jamulus uses far less bandwidth, has far fewer dropouts, and seems to be a lot more stable. JackTrip offers lossless audio at lower latencies when using higher frequency rates (96k), but this comes at a cost of 6 times higher bandwidth utilization and frequent dropouts. If I can figure out a way to disable the local audio monitoring, it would be a great option for headless Raspberry Pis.

8 thoughts on “Testing Jamulus Latency on Raspberry Pi

  1. As far as I know, The HiFi Berry DAC is not designed for low latency. It’s designed for streaming. I have one of these that I used to use for streaming music in my house. It’s good for that (though I moved on to the ALLO Boss DAC which is better, albeit a bit more expensive). However, for real-time music, I wouldn’t use either of these for Jamulus playing. I could be wrong, though.

    For a better test of the potential of using Raspberry Pi’s for Jamulus playing, you should try Pi Sound, which is an ADC/DAC hat for the Raspberry Pi. It’s a bit pricey — it’s like 100 bucks or more. It’s designed for low latency, and with it, you can also use a low latency kernel, (in fact, it comes with an optional version of the Raspian OS using a low latency kernel — though I don’t using a low latency kernel is necessary, as even the normal Linux kernels are getting more and more low latency features built-in all the time.)

    I have tried using such a set up for a Raspberry Pi 4 Jamulus client (and have tried with 2GB ram, 4 GB ram, and 8 GB ram versions of the Rpi 4). The Pi Sound hat has a built 1/4″ plug audio for plugging in, say, a guitar or high impedance microphone, as well as an audio output jack for headphones (or to send to amp). You can use 64 or 128 samples for the jack buffer, and I’ve even used 32 samples and even 16 samples with some success, though really, below 64 is probably asking for trouble.

    But you can also hook up, say, a Behringer UMC 404HD audio device (Which is what I have) to it as well, via USB port as well. I get decent results with it but have not tested it thoroughly. But I’m inclined to think that such a setup would make a decent client machine — mainly because you can dedicate it just for the task of Jamulus playing.

    I have successfully used Rpi4 as a Jamulus server as well. But for that, you of course don’t need any ADC/DAC hat or sound card or anything. I have used it as a server for 4 players, though we are still learning how to set things up properly on everyone’s machine — and not everyone has a fast internet connection and may not have the best computers in the world, so we don’t have definitive results yet. I have tried the 2 GB, 4 GB, and 8 GB versions or Rpi 4, but don’t have enough experience yet to draw any conclusions about what’s needed. For client use, the 2 GB is probably just fine. For server use, any of them would work, but I’d be inclined to use at least 4 GB version, just for the sake of have lots of headroom. That’s all still to be determined. It could be 2 GB is just fine.

    Playing with my band remotely here in Phoenix with a Rpi 4 server, I of course have 0 ms ping, (since the server is in my house) and about 20 ms delay overall. Friends that are roughly 20 miles away are seeing 15-20 ms ping, and 40-50 ms overall — but I have fiber optic, they don’t. And I think that makes a lot of difference. I think their cable modem is a big culprit, and maybe even their router. I think with good internet connections (fiber optic if possible) the Rpi 4 is perfectly fine as a client, and perfectly fine as a server. One advantage over say, a Windows machine (which I used most of the time as a client since it’s very fast) is that ASIO is a lot more restrictive when it comes to stringing devices together. Jack on Linux, (and Core Audio on the Mac) are much more flexible in this regard. So it’s possible to get a bit more creative with how many instruments you can string together on a single client. Now, Jack, is available on Windows too, but I don’t know how reliable it is, and I believe it’s actually a layer that sits on top of ASIO.

    The big killer for the Raspberry Pi 4 is that it only runs at 1.5 GHz, whereas my Windows desktop (several years old now) runs at 3.5 Ghz. Yeah, you can overclock the Rpi 4 somewhat, but it already runs hot the way it is. So still remains to be seen how many players you can feasibly have when using the Rpi4 as a server, or how many instruments you can hook up when using it as a client. The saving grace of using Rpi 4 instead of my Windows desktop is having a dedicated computer just for Jamulus. I think, for example, getting a Rpi4 2 GB for $35 is a compelling choice as a Jamulus client. (Of course, you need an extra keyboard/mouse/monitor too).

    UPDATE: I’m beginning to understand that the computer you are using *does* make a lot of difference above and beyond your internet connection. For example, I created an Amazon AWS Jamulus server that runs out of Los Angeles. Here in Phoenix, via COX Gigablast internet connection, and running a Behringer UMC404HD audio device, I see ping times of 14 ms for both a Windows Jamulus client, and for a Rpi 4 Jamulus client as well. However, the overall delay on the Windows desktop is 24 ms, whereas on the Rpi4, it’s 30 to 34 ms. So the Rpi4 is 5 to 10 ms slower, for whatever reason. My friends 20 miles away, on the same server, are seeing 20-25 ms ping times, and 40 ms overall delay. Why more? I think it’s the age of their computers, and their cable modem, (they’re not on Gigablast.)

    One take away of this update, is that even running as a Phoenix client for a server in Los Angeles, it’s possible to get decent (30 ms) overall latency values on a Rpi 4 client. One caveat, I haven’t tested audio quality all that much yet, especially with multiple players. But it is showing the potential.

  2. HiFiBerry DAC is very different from the HiFiBerry ADC DAC+. My testing shows it’s excellent for low-latency audio, with overhead of about 1ms.

    PiSound does also seem like a great option, albeit at twice the price. I have one sitting boxed that I’ve been meaning to try for weeks. Just haven’t got to it yet. Once I do, I’ll post results here.

  3. Cool. I didn’t know that. I need to investigate the HiFiBerry ADC DAC+. I knew about it, but hadn’t really looked into it too much.

    Oh and yes, the PiSound is a bit pricey. Funny thing is, after posting my message the other day, I tried to use the PiSound box as a Jamulus client. I couldn’t get it to work again. It wouldn’t connect to any server I tried. The symptom you see, when say, looking at the settings of *another* Jamulus client that’s running on a different machine, and connected to the same server as the Rpi client, is that the ping and latency numbers on the lower right corner alternately flash (at roughly a 1/2 sec or 1 sec rate) between whatever the normal numbers were for that other client, and then numbers in red, saying “> 500 ms”. This goes on until you decide to give up and press the disconnect button on the Rpi jamulus client.

    I had seen these symptoms before, and somehow had managed to get the PiSound Rpi client to work after messing around with the Jack settings. But I could not get it to work again.

    Since then, I’ve tried running a “normal” Rpi4 box (8 GB) with a Behringer UMC 404HD connected to it. I was able to get it to connect to various Jamulus servers (as long as I started qjackctl first and setup the desired device of choice as being the Behringer). What I did find, however, is that the Jack service start up by qjackctl would fail if I tried any buffer setting less than 128 samples (like, say, 64). 128 and 256 were okay. That’s probably why the PiSound client failed, as I usually set the buffer size for it to 64 samples. But I swear I’ve gotten to work before at 64 samples, I just don’t know what the magic trick was. (It could have been my memory fooling me.)

    Back to the bare bones Rpi4 with UMC404HD hooked up: At 128 samples, on the particular servers I connected to (one was an Amazon cloud server that I set up located in LA with 15 ms ping time between there and Phoenx), would lead to overall latencies of anywhere from 40-60 ms. Which isn’t what I had hoped for.

    This all being said, I’ve been able to use Rpi4’s as Jamulus servers okay, though not tested with a lot of clients (no more than three or 4), and what testing I have done has not been thorough.

    1. An update about using Jamulus on a PiSound: I started from scratch and loaded a new PatchBox OS for PiSound, and then installed Jamulus. Got it working as a client. And it works as long as I load qjackctl first and tell it to use the PiSound for audio.

      I have no idea why I was having trouble getting this to work before.

      I haven’t actually played music with it yet, but I’m getting no xruns with a sample buffer of 64, with 2 periods. I’m connecting to my own AWS server with a 16 ms ping time, and overall delay of 25 ms. My jitter buffer on auto runs at 2 + 2. It appears rock-solid, with no buffer dropouts. That’s pretty darn good! That’s as good as or even better than my desktop PC which is a much faster machine. Still, though I need to try playing music to see what happens. It could be that “real world playing” is a whole nuther ball of wax.

      The difference between using the PiSound and running an “ordinary” Rpi4 withg an external USB audio device is for sure the latency. That’s because, like the HifiBerry ADC DAC, the PiSound is a hat, using I2s, so it’s much faster and less jittery than any USB device is going to be. Also, PiSound can be used with its own version of Raspian (PatchBox OS) that uses a real time kernel. Though I don’t think that’s really going to make any difference.

      The Hifiberry is for sure less expensive, (the PiSound is 99 euros, which is whatever in dollars — call it $125?) Compared to the $55 or $67 or whatever the HifiBerry is. However, the PiSound does have a 5 pin MIDI port — and I’m finding that using 5 pin MIDI instead of USB Midi has some advantages. It seems less “jittery”, though it has more delay (1 ms) than USB midi would. But these observations haven’t been investigated thoroughly. They are more “impressions” than anything else.

      Mike, good luck with your explorations of using the HifiBerry with Jamulus. I’ll be watching this blog for any new observations, and will keep you posted on any things I discover with the PiSound.

  4. Your comment about disabling local monitoring surprises me, as well as your latency results.

    The first rule of jamulus is to listen through the sound back from the server, never your own local sound. Even though this takes a tiny bit of getting used to, it ensures your timing is the same as other people’s timing. We tried both extensively, as muting yourself is just pressing the mute button, and listening to ourselves through the headphones produced a much better experience.

    Regarding your latency: with 64 samples at asynchronous mode in jack and a hifiberry, I get very stable results, and due to how jamulus works, slightly lower latency than at 128 samples synchronous mode.
    I am still trying to figure out how to get 64 samples in synchronous mode without pops and cracks, I will try a 1000hz clock tick kernel next.

  5. Note that these latency tests were designed to be 2x what you would get from person A to person B (i.e. 23ms for “auto” with 128 samples).

    Jamulus has since introduced a “–mutemyown” option which allows the client to not hear their own audio coming back from the server. Based on my experience having since worked with LOTS of people, this seems to be a personal preference, and perhaps different depending on what type of music you are performing. I imagine another variable is what their own latency is (the higher the number, the more this becomes a distraction versus helpful). I have no personal opinion, just prefer to offer people the option when there seems to be broad disagreement on it.

Leave a Reply