Diagnosing periodic high ping times on macOS

Background

Up until a few months ago, my home network setup was pretty terrible. The entire house ran off the WiFi provided by an OpenWRT-powered TP-Link WDR3600 from 2013 in the far corner of the house. So when I fired up a game on my laptop via Steam in-home streaming and got unplayable lag and stuttering, I just figured that the issue was the network and left it at that. After all, the gaming PC was connected via WiFI from the opposite corner of the house with no shortage of walls in between.

In November 2018, I finally got serious about the network upgrade I had been planning. I'll probably do a more in-depth post about this at some point, but to make a long story short: Most things are wired and what isn't is connected via a modern, centrally-located access point. Performance is noticeably better in general and amazingly better for high-bandwidth tasks like transferring files.

So when I excitedly fired up the same game on my laptop, I was pretty surprised that it was still unplayable. At this point I should clarify that the game was not something that requires a CRT and wired controller for super-low response times. So when I say unplayable, I mean legitimately unplayable. I'm talking periodic lag spikes that caused the video to drop a few seconds or more behind the inputs. Even turn-based games would have issues with the amount of lag.

Narrowing the scope

To make sure it wasn't an issue with the streaming software, the gaming PC, or the network, I used ping to test the latency to the gateway from a few hosts. All the computers except my laptop had normal ping times. Running it on my laptop looked like this:

$ ping pfsense.lan | tee >(grep -oP 'time=\K\S*' | spark)
PING pfsense.lan (192.168.1.1): 56 data bytes
bytes from 192.168.1.1: icmp_seq=0 ttl=64 time=1.574 ms
bytes from 192.168.1.1: icmp_seq=1 ttl=64 time=0.996 ms
bytes from 192.168.1.1: icmp_seq=2 ttl=64 time=1.568 ms
bytes from 192.168.1.1: icmp_seq=3 ttl=64 time=92.153 ms
bytes from 192.168.1.1: icmp_seq=4 ttl=64 time=39.079 ms
bytes from 192.168.1.1: icmp_seq=5 ttl=64 time=14.307 ms
bytes from 192.168.1.1: icmp_seq=6 ttl=64 time=1.570 ms
bytes from 192.168.1.1: icmp_seq=7 ttl=64 time=1.636 ms
bytes from 192.168.1.1: icmp_seq=8 ttl=64 time=1.698 ms
bytes from 192.168.1.1: icmp_seq=9 ttl=64 time=1.568 ms
bytes from 192.168.1.1: icmp_seq=10 ttl=64 time=1.550 ms
bytes from 192.168.1.1: icmp_seq=11 ttl=64 time=1.106 ms
bytes from 192.168.1.1: icmp_seq=12 ttl=64 time=1.832 ms
bytes from 192.168.1.1: icmp_seq=13 ttl=64 time=1.816 ms
bytes from 192.168.1.1: icmp_seq=14 ttl=64 time=36.598 ms
bytes from 192.168.1.1: icmp_seq=15 ttl=64 time=42.863 ms
bytes from 192.168.1.1: icmp_seq=16 ttl=64 time=20.747 ms
bytes from 192.168.1.1: icmp_seq=17 ttl=64 time=3.112 ms
bytes from 192.168.1.1: icmp_seq=18 ttl=64 time=1.695 ms
bytes from 192.168.1.1: icmp_seq=19 ttl=64 time=1.795 ms
bytes from 192.168.1.1: icmp_seq=20 ttl=64 time=1.504 ms
bytes from 192.168.1.1: icmp_seq=21 ttl=64 time=1.644 ms
bytes from 192.168.1.1: icmp_seq=22 ttl=64 time=1.732 ms
bytes from 192.168.1.1: icmp_seq=23 ttl=64 time=1.581 ms
bytes from 192.168.1.1: icmp_seq=24 ttl=64 time=1.794 ms
bytes from 192.168.1.1: icmp_seq=25 ttl=64 time=33.367 ms
bytes from 192.168.1.1: icmp_seq=26 ttl=64 time=124.282 ms
bytes from 192.168.1.1: icmp_seq=27 ttl=64 time=97.719 ms
bytes from 192.168.1.1: icmp_seq=28 ttl=64 time=76.501 ms
bytes from 192.168.1.1: icmp_seq=29 ttl=64 time=1.734 ms
bytes from 192.168.1.1: icmp_seq=30 ttl=64 time=1.622 ms
bytes from 192.168.1.1: icmp_seq=31 ttl=64 time=1.758 ms
bytes from 192.168.1.1: icmp_seq=32 ttl=64 time=1.578 ms
bytes from 192.168.1.1: icmp_seq=33 ttl=64 time=1.527 ms
bytes from 192.168.1.1: icmp_seq=34 ttl=64 time=1.017 ms
^C
 ▁▁▁▆▃▁▁▁▁▁▁▁▁▁▃▃▂▁▁▁▁▁▁▁▁▂█▆▅▁▁▁▁▁▁

(sweet graph at the end courtesy of spark)

Notice the periodic spikes. Clearly there's something going on with the network connection on my laptop. Connecting via a wired connection and running it again confirmed that it was only happening on WiFi.

WiFi debugging

Now that I had narrowed the issue down to an issue with the WiFi, I could start trying to debug it.

After coming across this very helpful post explaining how, I enabled WiFi debug logging by running:

sudo /System/Library/PrivateFrameworks/Apple80211.framework/Versions/A/Resources/airport en0 debug +AllUserland

(remember to turn this off after, airport is pretty chatty with debug logs enabled)

I then watched the wifi log (sudo tail -f /var/log/wifi.log) as I ran the same ping command. Whenever I saw ping times spike, I also saw the following entries in the WiFi log:

Tue Nov 27 02:01:33.764 IPC: <airportd[68]> ADDED XPC CLIENT CONNECTION [QSyncthingTray (pid=14397, euid=501, egid=20)]
Tue Nov 27 02:01:33.765 Info: <airportd[68]> SCAN request received from pid 14397 (QSyncthingTray) with priority 0
Tue Nov 27 02:01:33.766 Scan: <airportd[68]> Cache-assisted scan request for QSyncthingTray on channel 1 does not require a live scan
Tue Nov 27 02:01:33.766 Scan: <airportd[68]> Cache-assisted scan request for QSyncthingTray on channel 2 does not require a live scan
Tue Nov 27 02:01:33.766 Scan: <airportd[68]> Cache-assisted scan request for QSyncthingTray on channel 3 does not require a live scan
Tue Nov 27 02:01:33.766 Scan: <airportd[68]> Cache-assisted scan request for QSyncthingTray on channel 4 does not require a live scan
Tue Nov 27 02:01:33.766 Scan: <airportd[68]> Cache-assisted scan request for QSyncthingTray on channel 5 does not require a live scan
Tue Nov 27 02:01:33.766 Scan: <airportd[68]> Cache-assisted scan request for QSyncthingTray on channel 6 does not require a live scan
Tue Nov 27 02:01:33.766 Scan: <airportd[68]> Cache-assisted scan request for QSyncthingTray does not require a live scan
Tue Nov 27 02:01:33.766 AutoJoin: <airportd[68]> Successful cache-assisted scan request for QSyncthingTray with channels {(
Tue Nov 27 02:01:33.766     <CWChannel: 0x7f9073514e70> [channelNumber=1(2GHz), channelWidth={20MHz}, active],
Tue Nov 27 02:01:33.766     <CWChannel: 0x7f90735507b0> [channelNumber=2(2GHz), channelWidth={20MHz}, active],
Tue Nov 27 02:01:33.766     <CWChannel: 0x7f9073526370> [channelNumber=3(2GHz), channelWidth={20MHz}, active],
Tue Nov 27 02:01:33.766     <CWChannel: 0x7f9073526420> [channelNumber=4(2GHz), channelWidth={20MHz}, active],
Tue Nov 27 02:01:33.766     <CWChannel: 0x7f907354ddb0> [channelNumber=5(2GHz), channelWidth={20MHz}, active],
Tue Nov 27 02:01:33.766     <CWChannel: 0x7f907354c6c0> [channelNumber=6(2GHz), channelWidth={20MHz}, active]
Tue Nov 27 02:01:33.766 )} took 0.0007 seconds, returned 5 results
< insert logs like above for all channels all the way up to 144 in batches of 6 >
Tue Nov 27 02:01:33.779 IPC: <airportd[68]> INVALIDATED XPC CLIENT CONNECTION [QSyncthingTray (pid=14397, euid=501, egid=20)]
< 10 seconds pass>
Tue Nov 27 02:01:43.847 IPC: <airportd[68]> ADDED XPC CLIENT CONNECTION [QSyncthingTray (pid=14397, euid=501, egid=20)]
< same scanning logs as above >
Tue Nov 27 02:01:46.741 IPC: <airportd[68]> INVALIDATED XPC CLIENT CONNECTION [QSyncthingTray (pid=14397, euid=501, egid=20)]
< ... and repeat >

This pretty clearly indicates that the QSyncthingTray application (a tray icon to monitor Syncthing) was periodically requesting WiFi scans which, in turn, caused the bursts of latency. After quitting the application and running ping again, the times were all <2ms, even after running for a few minutes. Also, when I started up the game again to test it, the lag was completely gone!

Digging deeper

Since QSyncthingTray is open source, I filed a bug report on the project and dug into the code to see if I could pinpoint the issue. After finding nothing out of the ordinary and doing a bunch of searching the internet, it turns out that it's not an issue with QSyncthingTray at all, but with Qt, a cross-platform SDK that QSyncthingTray and many other programs use. Some classmates and I actually used it to make a game called ParticleStorm back in university.

There are open bug reports against this issue from 2013, 2014, and 2015. Additionally, there are loads of blog posts and per-program workarounds. It's a little disappointing that just running a network-enabled Qt application can seriously disrupt real-time applications on the same system. Granted, I have no knowledge of the complexities of this issue, but at the very least it seems the documentation could be clearer on the issue so developers stop getting surprised by it.

Anyway, the "fix" is to set QT_BEARER_POLL_TIMEOUT=-1 as an environment variable on your system. Honestly, I've just stopped using QSyncthingTray instead. Not that it's not useful, I just realized that I didn't really need it. Syncthing works well enough that I don't find myself ever worrying about it. If a Qt application ever becomes an essential part of my setup I guess I'll have to revisit this.

Takeaways

When macOS scans for networks (as it does when requested by an application, when you click the WiFi icon, or when you request your location) it causes high ping times and dropped packets. According to bug reports, scanning for WiFi networks on other OS's causes the same issues. This is something I didn't really expect, but good to know going forward.
Qt has a bug where instantiating a QNetworkConfigurationManager causes it to scan for WiFi networks every 10 seconds by default. Some applications set the QT_BEARER_POLL_TIMEOUT environment variable themselves to disable this behaviour, but the majority probably have no idea that it's even an issue until a user reports it.

So if you're seeing regular latency spikes, audit what programs you have running. Specifically look for programs built with Qt. Setting the QT_BEARER_POLL_TIMEOUT environment variable to -1 can help with these. Applications that use your location could also be the culprit. Who knew keeping your ping down to an acceptable level was such a minefield...

Proxied access to the Namecheap DNS API

Background

Both this site and my home server use HTTPS certificates provided by Let's Encrypt. Previously I was using the http-01 challenge method, but once wildcard certificates became available, I decided to switch to the dns-01 method for simplicity.

My domains are currently registered through Namecheap which provides an API for modifying DNS records. The only catch is that it can only be accessed via manually-whitelisted IP addresses. Because the server I'm trying to access the API from is assigned a dynamic IP, this restriction was a bit of an issue.

Back in November when I initially made the switch to the DNS-based challenge, I set it up the easy way: I manually added my current IP to the whitelist, added a TODO entry to fix it somehow, and set a reminder scheduled for the week before the cert would expire telling me to update the IP whitelist. Fast-forward to today when I was reminded to update my IP whitelist. Instead of continuing to kick the can down the road, I decided to actually fix the issue.

Setting up a temporary proxy

My home server is connected to the internet though a normal residential connection which is assigned a dynamic IP address. However, the server that I run this website on is configured with a static IP since it's a hosted VPS. By proxying all traffic to the Namecheap API through my VPS, I could add my VPS's static IP to the whitelist and not have to worry about my home IP changing all the time.

SSH is perfect tool for this. The OpenSSH client supports port forwarding using the -D flag. By then setting the HTTP[S]_PROXY environment variables to point to the forwarded port, programs that support those environment variables will transparently forward all their requests through the proxy.

After a bunch of research and testing, I came up with the following script to easily set up the temporary proxy:

#!/bin/sh
# Source this file to automatically setup and teardown an HTTP* proxy through cmetcalfe.ca
# Use the PROXY_PORT and PROXY_DEST environment variables to customize the proxy

PROXY_PORT="${PROXY_PORT:-11111}"
PROXY_DEST="${PROXY_DEST:-<username>@cmetcalfe.ca}"
PID="$$"

# Teardown the SSH connection when the script exits
trap 'ssh -q -S ".ctrl-socket-$PID" -O exit "$PROXY_DEST"' EXIT

# Set up an SSH tunnel and wait for the port to be forwarded before continuing
if ! ssh -o ExitOnForwardFailure=yes -M -S ".ctrl-socket-$PID" -f -N -D "$PROXY_PORT" "$PROXY_DEST"; then
    echo "Failed to open SSH tunnel, exiting"
    exit 1
fi

# Set environment variables to redirect HTTP* traffic through the proxy
export HTTP_PROXY="socks5://127.0.0.1:$PROXY_PORT"
export HTTPS_PROXY="$HTTP_PROXY"

This script is saved as ~/.config/tempproxy.rc and can be sourced to automatically set up a proxy session and have it be torn down when the script exits.

You'll want to use key-based authentication with an unencrypted private key so that you don't need to type anything to initiate the SSH session. For this reason you'll probably want to create a limited user on the target system that can only really do port forwarding. There's a good post on this here.

Talking to the API through the proxy

To allow programs that use the tempproxy.rc script to talk to the Namecheap API, the IP address of the VPS was added to the whitelist. Now that the proxying issue was taken care of, I just needed to wire up the actual certificate renewal process to use it.

The tool I'm using to talk to the Namecheap DNS API is lexicon. It can handle manipulating the DNS records of a ton of providers and integrates really nicely with my ACME client of choice, dehydrated. Also, because it's using the PyNamecheap Python library, which in turn uses requests under the hood, it will automatically use the HTTP*_PROXY environment variables when making requests.

The only tricky bit is that the base install of lexicon won't automatically pull in the packages required for accessing the Namecheap API. Likewise, requests won't install packages to support SOCKS proxies. To install all the required packages you'll need to run a command like:

pip install 'requests[socks]' 'dns-lexicon[namecheap]'

Since lexicon can use environment variables as configuration, I created another small source-able file at ~/.config/lexicon.rc:

#!/bin/sh
# Sets environment variables for accessing the Namecheap API

# Turn on API access and get an API token here:
# https://ap.www.namecheap.com/settings/tools/apiaccess/
export PROVIDER=namecheap
export LEXICON_NAMECHEAP_USERNAME=<username>
export LEXICON_NAMECHEAP_TOKEN=<api token>

With that in place, the existing automation script that I had previously set up when I switched to using the dns-01 challenge just needed some minor tweaks to source the two new files. I've provided the script below, along with some other useful ones that use the DNS API.

Scripts

`do-letsencrypt.sh`

This is the main script that handles all the domain renewals. It's called via cron on the first of every month.

Fun fact: The previous http-01 version of this script was a mess - it involved juggling nginx configurations with symlinks, manually opening up the .well-known/acme-challenge endpoint, and some other terrible hacks. This version is much nicer.

#!/bin/sh
# Generate/renew Let's Encrypt certificates using dehydrated and lexicon

set -e

# Set configuration variables and setup the proxy
source ~/.config/lexicon.rc
source ~/.config/tempproxy.rc

cd /etc/nginx/ssl

# Update the certs.
# Hook script copied verbatim from
# https://github.com/AnalogJ/lexicon/blob/master/examples/dehydrated.default.sh
if ! dehydrated --accept-terms --cron --challenge dns-01 --hook dehydrated.default.sh; then
    echo "Failed to renew certificates"
    exit 1
fi

# Reload the nginx config if it's currently running
if ! systemctl is-active nginx.service > /dev/null; then
    echo "nginx isn't running, not reloading it"
    exit 0
fi

if ! systemctl reload nginx.service; then
    systemctl status nginx.service
    echo "Failed to reload nginx config, check the error log"
    exit 1
fi

`dns.sh`

This one is really simple - it just augments the normal lexicon CLI interface with the proxy and configuration variables.

#!/bin/sh
# A simple wrapper around lexicon.
# Sets up the proxy and configuration, then passes all the arguments to the configured provider

source ~/.config/lexicon.rc
source ~/.config/tempproxy.rc
lexicon "$PROVIDER" "$@"

$ ./dns.sh list cmetcalfe.ca A
ID       TYPE NAME             CONTENT         TTL
-------- ---- ---------------- --------------- ----
xxxxxxxx A    @.cmetcalfe.ca   xxx.xxx.xxx.xxx 1800
xxxxxxxx A    www.cmetcalfe.ca xxx.xxx.xxx.xxx 1800

`dns-update.sh`

An updated version of my previous script that uses the API instead of the DynamicDNS service Namecheap provides. Called via cron every 30 minutes.

Update

A previous version of this script didn't try to delete the DNS entry before setting the new one, resulting in multiple IPs being set as the same A record.

This is because the update function of the Namecheap plugin for lexicon is implemented as a 2-step delete and add. When it tries to delete the old record, it looks for one with the same content as the old one. This means that if you update a record with a new IP, it doesn't find the old record to delete first, leaving you with all of your old IPs set as individual A records.

#!/bin/sh
# Check the server's current IP against the IP listed in its DNS entry.
# Set the current IP if they differ.

set -e

resolve() {
    line=$(drill "$1" @resolver1.opendns.com 2> /dev/null | sed '/;;.*$/d;/^\s*$/d' | grep "$1")
    echo "$line" | head -1 | cut -f5
}

dns=$(resolve <subdomain>.cmetcalfe.ca)
curr=$(resolve myip.opendns.com)
if [ "$dns" != "$curr" ]; then
    source ~/.config/lexicon.rc
    source ~/.config/tempproxy.rc
    if ! lexicon "$PROVIDER" delete cmetcalfe.ca A --name "<subdomain>.cmetcalfe.ca"; then
        echo "Failed to delete old DNS record, not setting the new one."
        exit 1
    fi
    if lexicon "$PROVIDER" update cmetcalfe.ca A --name "<subdomain>.cmetcalfe.ca" --content "$curr" --ttl=900; then
        echo "Server DNS record updated ($dns -> $curr)"
    else
        echo "Server DNS record update FAILED (tried $dns -> $curr)"
    fi
fi

Reducing Chromecast Idle Bandwidth

Google's Chromecast is a pretty useful device. It plugs into the HDMI port of a TV and allows users to "cast" content to it via a smartphone, Chrome browser, etc. This content is mostly assumed to be streaming services like Youtube, Spotify, or Netflix, but for someone who doesn't really buy into the whole cloud revolution like myself, it's still perfectly capable of playing local content as well.

The one downside of the Chromecast is that when it's not being used, instead of going into a low-power state and/or turning off the TV, it instead enters Ambient Mode. This shows some useful information overlaid on a changing backdrop of featured photos downloaded from Google Photos. The issue with this is that the images are high-resolution, not cached, and are continually being downloaded 24/7 even when the TV is turned off. Although I haven't personally measured it, the general consensus seems to be that it uses around 15GB of data per month from just being plugged in.

What I'll be going over here is reducing the data the Chromecast uses by configuring it to only download some tiny black 1x1 px images to use as a backdrop.

The short version

Make a tiny black PNG (pixel.png)
Make a slightly different tiny PNG to avoid deduplication (pixel_alt1.png)
Upload both to an album in Google Photos
Mark both images as favorites
Configure the Chromecast to only pull images from your specific album every 10 minutes

If you like details, keep reading...

Making a small image

I ended up creating a 1x1 solid black image in GIMP, saving it as a PNG, hacking out as much data as possible using a hex editor, then using pngcrush to attempt to optimize the data that was left.

As I was working on the PNG file, I used a small script based on the Python construct library to visualize the chunks and structure of it so I could figure out what to cut.

To install construct and download the example PNG specification:

python3 -m venv .venv
source .venv/bin/activate
pip install construct
wget "https://raw.githubusercontent.com/construct/construct/abd48c4892ceddc60c11d25f4a955573e2c61111/deprecated_gallery/png.py"

Create pngview.py:

import png  # the png spec downloaded above
import sys

with open(sys.argv[1], 'rb') as f:
    data = f.read()
    print("Size:", len(data), "bytes")
    print(png.png_file.parse(data))

Running it on the PNG exported from GIMP using the default settings gives:

$ ./pngview.py pixel.png
Size: 146 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) truecolor 2
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 2423739358
    chunks = ListContainer: 
        Container: 
            length = 9
            type = b'pHYs' (total 4)
            data = Container: 
                pixels_per_unit_x = 11811
                pixels_per_unit_y = 11811
                unit = (enum) meter 1
            crc = 2024095606
        Container: 
            length = 7
            type = b'tIME' (total 4)
            data = Container: 
                year = 2018
                month = 12
                day = 11
                hour = 5
                minute = 2
                second = 19
            crc = 904567710
        Container: 
            length = 25
            type = b'tEXt' (total 4)
            data = Container: 
                keyword = u'Comment' (total 7)
                text = b'Created with GIM'... (truncated, total 17)
            crc = 1468075543
        Container: 
            length = 12
            type = b'IDAT' (total 4)
            data = b'\x08\xd7c```\x00\x00\x00\x04\x00\x01' (total 12)
            crc = 657729290
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

After removing the unneeded pHYs, tIME, and tEXt chunks using a hex editor, it looks like this:

$ ./pngview.py pixel.png 
Size: 69 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) truecolor 2
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 2423739358
    chunks = ListContainer: 
        Container: 
            length = 12
            type = b'IDAT' (total 4)
            data = b'\x08\xd7c```\x00\x00\x00\x04\x00\x01' (total 12)
            crc = 657729290
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

At this point, all that's left to optimize is the image data itself. This is where pngcrush shines. After running pngcrush -brute -ow pixel.png we see:

$ ./pngview.py pixel.png
Size: 67 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) greyscale 0
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 981375829
    chunks = ListContainer: 
        Container: 
            length = 10
            type = b'IDAT' (total 4)
            data = b'\x08\x1dc`\x00\x00\x00\x02\x00\x01' (total 10)
            crc = 3486004709
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

The 2 byte savings in the data seem to have come from setting the color_type to greyscale instead of truecolor. To view what actually changed in the data, we can un-deflate it using Python:

>>> import zlib
>>> # before pngcrush (12 bytes)
>>> zlib.decompress(b'\x08\xd7c```\x00\x00\x00\x04\x00\x01')
b'\x00\x00\x00\x00'
>>> # after pngcrush (10 bytes)
>>> zlib.decompress(b'\x08\x1dc`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>>

This makes sense since, according to the PNG spec, when using truecolor, each pixel is an RGB triple, requiring 3 bytes. For greyscale, only a single byte representing luminance is needed. The first byte is the filtering method, which isn't relevant here since we only have a single pixel.

Interestingly enough, in both cases we would actually be much better off if we could opt to not use compression, but alas, the spec does not allow for anything except deflate.

But I digress, we now have a black 1x1 px PNG image that's just 67 bytes (pixel.png).

Generating multiple unique small images

So now we have the small PNG we want to display. Since the Chromecast's ambient mode requires at least 2 different images in an album to cycle through, all we need to to do is upload 2 copies of this PNG and we're done right? Almost.

Since Google Photos will automatically deduplicate uploaded images, we need to find a way to make the second image slightly different. Normally this would involve tweaking a comment or something, but in this case, the image has already been stripped down to its bare essentials.

My strategy was attempt to abuse the data compression to see if I could generate an image that compressed the same data into the same number of bytes, but differently. Fortunately, pngcrush can be told to use specific compression strategies (there are currently 177 of them). My hope is that at least one of these will achieve the same results, but in a different way. For starters we'll try the first 9:

for x in $(seq 9); do
    pngcrush -m "$x" pixel.png pixel_alt${x}.png;
done

A quick ls -l reveals that all of the pixel_alt*.png files are still 67 bytes. Now we just need to find one that has different data. sha1sum is the perfect utility for this:

$ sha1sum pixel.png pixel_alt*.png
d99a9d63b1cd9e4b3f823d4d03144ccd95328f48  pixel.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt1.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt2.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt3.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt4.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt5.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt6.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt7.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt8.png
d99a9d63b1cd9e4b3f823d4d03144ccd95328f48  pixel_alt9.png

pixel_alt1.png looks like a good candidate, let's see what changed:

$ diff <(./pngview.py pixel.png) <(./pngview.py pixel_alt1.png)
20,21c20,21
<             data = b'\x08\x1dc`\x00\x00\x00\x02\x00\x01' (total 10)
<             crc = 3486004709
---
>             data = b'\x08[c`\x00\x00\x00\x02\x00\x01' (total 10)
>             crc = 1648381800

There's the same amount of data, but it's different. Let's check if it decompresses to the same image data:

>>> import zlib
>>> zlib.decompress(b'\x08\x1dc`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>> zlib.decompress(b'\x08[c`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>>

Yep, this means that pixel_alt1.png is the exact same image with the exact same size, but won't be deduplicated when uploading it to Google Photos since the compressed data is different.

Final touches

Now that we have 2 different tiny images, we can upload them to Google Photos and put them in an album so they can be pulled down by the Chromecast. Something to note is that you may have to mark the images as favorites to get them to be displayed. There seems to be some sort of AI-fueled "we know better than you" algorithm that initially refused to display my images until I starred them.

Now just set your ambient mode to display the album with your images in it (or your favorites album), set the slideshow speed to its maximum value (change image every 10 mins), and you're done.