Up until a few months ago, my home network setup was pretty terrible. The entire house ran off the WiFi provided by an OpenWRT-powered TP-Link WDR3600 from 2013 in the far corner of the house. So when I fired up a game on my laptop via Steam in-home streaming and got unplayable lag and stuttering, I just figured that the issue was the network and left it at that. After all, the gaming PC was connected via WiFI from the opposite corner of the house with no shortage of walls in between.
In November 2018, I finally got serious about the network upgrade I had been planning. I'll probably do a more in-depth post about this at some point, but to make a long story short: Most things are wired and what isn't is connected via a modern, centrally-located access point. Performance is noticeably better in general and amazingly better for high-bandwidth tasks like transferring files.
So when I excitedly fired up the same game on my laptop, I was pretty surprised that it was still unplayable. At this point I should clarify that the game was not something that requires a CRT and wired controller for super-low response times. So when I say unplayable, I mean legitimately unplayable. I'm talking periodic lag spikes that caused the video to drop a few seconds or more behind the inputs. Even turn-based games would have issues with the amount of lag.
Narrowing the scope
To make sure it wasn't an issue with the streaming software, the gaming PC, or the network, I used
ping to test the latency to the gateway from a few hosts. All the computers except my laptop had
normal ping times. Running it on my laptop looked like this:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 38 39
(sweet graph at the end courtesy of spark)
Notice the periodic spikes. Clearly there's something going on with the network connection on my laptop. Connecting via a wired connection and running it again confirmed that it was only happening on WiFi.
Now that I had narrowed the issue down to an issue with the WiFi, I could start trying to debug it.
After coming across this very helpful post explaining how, I enabled WiFi debug logging by running:
(remember to turn this off after,
airport is pretty chatty with debug logs enabled)
I then watched the wifi log (
sudo tail -f /var/log/wifi.log) as I ran the same
Whenever I saw ping times spike, I also saw the following entries in the WiFi log:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24
This pretty clearly indicates that the QSyncthingTray application (a tray icon to monitor
Syncthing) was periodically requesting WiFi scans which, in turn, caused the bursts of latency.
After quitting the application and running
ping again, the times were all <2ms, even after running
for a few minutes. Also, when I started up the game again to test it, the lag was completely gone!
Since QSyncthingTray is open source, I filed a bug report on the project and dug into the code to see if I could pinpoint the issue. After finding nothing out of the ordinary and doing a bunch of searching the internet, it turns out that it's not an issue with QSyncthingTray at all, but with Qt, a cross-platform SDK that QSyncthingTray and many other programs use. Some classmates and I actually used it to make a game called ParticleStorm back in university.
There are open bug reports against this issue from 2013, 2014, and 2015. Additionally, there are loads of blog posts and per-program workarounds. It's a little disappointing that just running a network-enabled Qt application can seriously disrupt real-time applications on the same system. Granted, I have no knowledge of the complexities of this issue, but at the very least it seems the documentation could be clearer on the issue so developers stop getting surprised by it.
Anyway, the "fix" is to set
QT_BEARER_POLL_TIMEOUT=-1 as an environment variable on your system.
Honestly, I've just stopped using QSyncthingTray instead. Not that it's not useful, I just realized
that I didn't really need it. Syncthing works well enough that I don't find myself ever worrying
about it. If a Qt application ever becomes an essential part of my setup I guess I'll have to
When macOS scans for networks (as it does when requested by an application, when you click the WiFi icon, or when you request your location) it causes high ping times and dropped packets. According to bug reports, scanning for WiFi networks on other OS's causes the same issues. This is something I didn't really expect, but good to know going forward.
Qt has a bug where instantiating a
QNetworkConfigurationManagercauses it to scan for WiFi networks every 10 seconds by default. Some applications set the
QT_BEARER_POLL_TIMEOUTenvironment variable themselves to disable this behaviour, but the majority probably have no idea that it's even an issue until a user reports it.
So if you're seeing regular latency spikes, audit what programs you have running. Specifically look
for programs built with Qt. Setting the
QT_BEARER_POLL_TIMEOUT environment variable to
help with these. Applications that use your location could also be the culprit. Who knew keeping
your ping down to an acceptable level was such a minefield...