Posts | Tags | Archive

Proxied access to the Namecheap DNS API

Background

Both this site and my home server use HTTPS certificates provided by Let's Encrypt. Previously I was using the http-01 challenge method, but once wildcard certificates became available, I decided to switch to the dns-01 method for simplicity.

My domains are currently registered through Namecheap which provides an API for modifying DNS records. The only catch is that it can only be accessed via manually-whitelisted IP addresses. Because the server I'm trying to access the API from is assigned a dynamic IP, this restriction was a bit of an issue.

Back in November when I initially made the switch to the DNS-based challenge, I set it up the easy way: I manually added my current IP to the whitelist, added a TODO entry to fix it somehow, and set a reminder scheduled for the week before the cert would expire telling me to update the IP whitelist. Fast-forward to today when I was reminded to update my IP whitelist. Instead of continuing to kick the can down the road, I decided to actually fix the issue.

Setting up a temporary proxy

My home server is connected to the internet though a normal residential connection which is assigned a dynamic IP address. However, the server that I run this website on is configured with a static IP since it's a hosted VPS. By proxying all traffic to the Namecheap API through my VPS, I could add my VPS's static IP to the whitelist and not have to worry about my home IP changing all the time.

SSH is perfect tool for this. The OpenSSH client supports port forwarding using the -D flag. By then setting the HTTP[S]_PROXY environment variables to point to the forwarded port, programs that support those environment variables will transparently forward all their requests through the proxy.

After a bunch of research and testing, I came up with the following script to easily set up the temporary proxy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
#!/bin/sh
# Source this file to automatically setup and teardown an HTTP* proxy through cmetcalfe.ca
# Use the PROXY_PORT and PROXY_DEST environment variables to customize the proxy

PROXY_PORT="${PROXY_PORT:-11111}"
PROXY_DEST="${PROXY_DEST:-<username>@cmetcalfe.ca}"
PID="$$"

# Teardown the SSH connection when the script exits
trap 'ssh -q -S ".ctrl-socket-$PID" -O exit "$PROXY_DEST"' EXIT

# Set up an SSH tunnel and wait for the port to be forwarded before continuing
if ! ssh -o ExitOnForwardFailure=yes -M -S ".ctrl-socket-$PID" -f -N -D "$PROXY_PORT" "$PROXY_DEST"; then
    echo "Failed to open SSH tunnel, exiting"
    exit 1
fi

# Set environment variables to redirect HTTP* traffic through the proxy
export HTTP_PROXY="socks5://127.0.0.1:$PROXY_PORT"
export HTTPS_PROXY="$HTTP_PROXY"

This script is saved as ~/.config/tempproxy.rc and can be sourced to automatically set up a proxy session and have it be torn down when the script exits.

You'll want to use key-based authentication with an unencrypted private key so that you don't need to type anything to initiate the SSH session. For this reason you'll probably want to create a limited user on the target system that can only really do port forwarding. There's a good post on this here.

Talking to the API through the proxy

To allow programs that use the tempproxy.rc script to talk to the Namecheap API, the IP address of the VPS was added to the whitelist. Now that the proxying issue was taken care of, I just needed to wire up the actual certificate renewal process to use it.

The tool I'm using to talk to the Namecheap DNS API is lexicon. It can handle manipulating the DNS records of a ton of providers and integrates really nicely with my ACME client of choice, dehydrated. Also, because it's using the PyNamecheap Python library, which in turn uses requests under the hood, it will automatically use the HTTP*_PROXY environment variables when making requests.

The only tricky bit is that the base install of lexicon won't automatically pull in the packages required for accessing the Namecheap API. Likewise, requests won't install packages to support SOCKS proxies. To install all the required packages you'll need to run a command like:

1
pip install 'requests[socks]' 'dns-lexicon[namecheap]'

Since lexicon can use environment variables as configuration, I created another small source-able file at ~/.config/lexicon.rc:

1
2
3
4
5
6
7
8
#!/bin/sh
# Sets environment variables for accessing the Namecheap API

# Turn on API access and get an API token here:
# https://ap.www.namecheap.com/settings/tools/apiaccess/
export PROVIDER=namecheap
export LEXICON_NAMECHEAP_USERNAME=<username>
export LEXICON_NAMECHEAP_TOKEN=<api token>

With that in place, the existing automation script that I had previously set up when I switched to using the dns-01 challenge just needed some minor tweaks to source the two new files. I've provided the script below, along with some other useful ones that use the DNS API.

Scripts

do-letsencrypt.sh

This is the main script that handles all the domain renewals. It's called via cron on the first of every month.

Fun fact: The previous http-01 version of this script was a mess - it involved juggling nginx configurations with symlinks, manually opening up the .well-known/acme-challenge endpoint, and some other terrible hacks. This version is much nicer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
#!/bin/sh
# Generate/renew Let's Encrypt certificates using dehydrated and lexicon

set -e

# Set configuration variables and setup the proxy
source ~/.config/lexicon.rc
source ~/.config/tempproxy.rc

cd /etc/nginx/ssl

# Update the certs.
# Hook script copied verbatim from
# https://github.com/AnalogJ/lexicon/blob/master/examples/dehydrated.default.sh
if ! dehydrated --accept-terms --cron --challenge dns-01 --hook dehydrated.default.sh; then
    echo "Failed to renew certificates"
    exit 1
fi

# Reload the nginx config if it's currently running
if ! systemctl is-active nginx.service > /dev/null; then
    echo "nginx isn't running, not reloading it"
    exit 0
fi

if ! systemctl reload nginx.service; then
    systemctl status nginx.service
    echo "Failed to reload nginx config, check the error log"
    exit 1
fi

dns.sh

This one is really simple - it just augments the normal lexicon CLI interface with the proxy and configuration variables.

1
2
3
4
5
6
7
#!/bin/sh
# A simple wrapper around lexicon.
# Sets up the proxy and configuration, then passes all the arguments to the configured provider

source ~/.config/lexicon.rc
source ~/.config/tempproxy.rc
lexicon "$PROVIDER" "$@"
1
2
3
4
5
$ ./dns.sh list cmetcalfe.ca A
ID       TYPE NAME             CONTENT         TTL
-------- ---- ---------------- --------------- ----
xxxxxxxx A    @.cmetcalfe.ca   xxx.xxx.xxx.xxx 1800
xxxxxxxx A    www.cmetcalfe.ca xxx.xxx.xxx.xxx 1800

dns-update.sh

An updated version of my previous script that uses the API instead of the DynamicDNS service Namecheap provides. Called via cron every 30 minutes.

Update

A previous version of this script didn't try to delete the DNS entry before setting the new one, resulting in multiple IPs being set as the same A record.

This is because the update function of the Namecheap plugin for lexicon is implemented as a 2-step delete and add. When it tries to delete the old record, it looks for one with the same content as the old one. This means that if you update a record with a new IP, it doesn't find the old record to delete first, leaving you with all of your old IPs set as individual A records.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
#!/bin/sh
# Check the server's current IP against the IP listed in its DNS entry.
# Set the current IP if they differ.

set -e

resolve() {
    line=$(drill "$1" @resolver1.opendns.com 2> /dev/null | sed '/;;.*$/d;/^\s*$/d' | grep "$1")
    echo "$line" | head -1 | cut -f5
}

dns=$(resolve <subdomain>.cmetcalfe.ca)
curr=$(resolve myip.opendns.com)
if [ "$dns" != "$curr" ]; then
    source ~/.config/lexicon.rc
    source ~/.config/tempproxy.rc
    if ! lexicon "$PROVIDER" delete cmetcalfe.ca A --name "<subdomain>.cmetcalfe.ca"; then
        echo "Failed to delete old DNS record, not setting the new one."
        exit 1
    fi
    if lexicon "$PROVIDER" update cmetcalfe.ca A --name "<subdomain>.cmetcalfe.ca" --content "$curr" --ttl=900; then
        echo "Server DNS record updated ($dns -> $curr)"
    else
        echo "Server DNS record update FAILED (tried $dns -> $curr)"
    fi
fi

Reducing Chromecast Idle Bandwidth

Google's Chromecast is a pretty useful device. It plugs into the HDMI port of a TV and allows users to "cast" content to it via a smartphone, Chrome browser, etc. This content is mostly assumed to be streaming services like Youtube, Spotify, or Netflix, but for someone who doesn't really buy into the whole cloud revolution like myself, it's still perfectly capable of playing local content as well.

The one downside of the Chromecast is that when it's not being used, instead of going into a low-power state and/or turning off the TV, it instead enters Ambient Mode. This shows some useful information overlaid on a changing backdrop of featured photos downloaded from Google Photos. The issue with this is that the images are high-resolution, not cached, and are continually being downloaded 24/7 even when the TV is turned off. Although I haven't personally measured it, the general consensus seems to be that it uses around 15GB of data per month from just being plugged in.

What I'll be going over here is reducing the data the Chromecast uses by configuring it to only download some tiny black 1x1 px images to use as a backdrop.

The short version

  • Make a tiny black PNG (pixel.png)
  • Make a slightly different tiny PNG to avoid deduplication (pixel_alt1.png)
  • Upload both to an album in Google Photos
  • Mark both images as favorites
  • Configure the Chromecast to only pull images from your specific album every 10 minutes

If you like details, keep reading...

Making a small image

I ended up creating a 1x1 solid black image in GIMP, saving it as a PNG, hacking out as much data as possible using a hex editor, then using pngcrush to attempt to optimize the data that was left.

As I was working on the PNG file, I used a small script based on the Python construct library to visualize the chunks and structure of it so I could figure out what to cut.

To install construct and download the example PNG specification:

1
2
3
4
python3 -m venv .venv
source .venv/bin/activate
pip install construct
wget "https://raw.githubusercontent.com/construct/construct/abd48c4892ceddc60c11d25f4a955573e2c61111/deprecated_gallery/png.py"

Create pngview.py:

1
2
3
4
5
6
7
import png  # the png spec downloaded above
import sys

with open(sys.argv[1], 'rb') as f:
    data = f.read()
    print("Size:", len(data), "bytes")
    print(png.png_file.parse(data))

Running it on the PNG exported from GIMP using the default settings gives:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
$ ./pngview.py pixel.png
Size: 146 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) truecolor 2
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 2423739358
    chunks = ListContainer: 
        Container: 
            length = 9
            type = b'pHYs' (total 4)
            data = Container: 
                pixels_per_unit_x = 11811
                pixels_per_unit_y = 11811
                unit = (enum) meter 1
            crc = 2024095606
        Container: 
            length = 7
            type = b'tIME' (total 4)
            data = Container: 
                year = 2018
                month = 12
                day = 11
                hour = 5
                minute = 2
                second = 19
            crc = 904567710
        Container: 
            length = 25
            type = b'tEXt' (total 4)
            data = Container: 
                keyword = u'Comment' (total 7)
                text = b'Created with GIM'... (truncated, total 17)
            crc = 1468075543
        Container: 
            length = 12
            type = b'IDAT' (total 4)
            data = b'\x08\xd7c```\x00\x00\x00\x04\x00\x01' (total 12)
            crc = 657729290
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

After removing the unneeded pHYs, tIME, and tEXt chunks using a hex editor, it looks like this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ ./pngview.py pixel.png 
Size: 69 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) truecolor 2
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 2423739358
    chunks = ListContainer: 
        Container: 
            length = 12
            type = b'IDAT' (total 4)
            data = b'\x08\xd7c```\x00\x00\x00\x04\x00\x01' (total 12)
            crc = 657729290
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

At this point, all that's left to optimize is the image data itself. This is where pngcrush shines. After running pngcrush -brute -ow pixel.png we see:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
$ ./pngview.py pixel.png
Size: 67 bytes
Container: 
    signature = b'\x89PNG\r\n\x1a\n' (total 8)
    image_header = Container: 
        length = 13
        signature = b'IHDR' (total 4)
        width = 1
        height = 1
        bit_depth = 8
        color_type = (enum) greyscale 0
        compression_method = (enum) deflate 0
        filter_method = (enum) adaptive5 0
        interlace_method = (enum) none 0
        crc = 981375829
    chunks = ListContainer: 
        Container: 
            length = 10
            type = b'IDAT' (total 4)
            data = b'\x08\x1dc`\x00\x00\x00\x02\x00\x01' (total 10)
            crc = 3486004709
        Container: 
            length = 0
            type = b'IEND' (total 4)
            data = None
            crc = 2923585666

The 2 byte savings in the data seem to have come from setting the color_type to greyscale instead of truecolor. To view what actually changed in the data, we can un-deflate it using Python:

1
2
3
4
5
6
7
8
>>> import zlib
>>> # before pngcrush (12 bytes)
>>> zlib.decompress(b'\x08\xd7c```\x00\x00\x00\x04\x00\x01')
b'\x00\x00\x00\x00'
>>> # after pngcrush (10 bytes)
>>> zlib.decompress(b'\x08\x1dc`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>>

This makes sense since, according to the PNG spec, when using truecolor, each pixel is an RGB triple, requiring 3 bytes. For greyscale, only a single byte representing luminance is needed. The first byte is the filtering method, which isn't relevant here since we only have a single pixel.

Interestingly enough, in both cases we would actually be much better off if we could opt to not use compression, but alas, the spec does not allow for anything except deflate.

But I digress, we now have a black 1x1 px PNG image that's just 67 bytes (pixel.png).

Generating multiple unique small images

So now we have the small PNG we want to display. Since the Chromecast's ambient mode requires at least 2 different images in an album to cycle through, all we need to to do is upload 2 copies of this PNG and we're done right? Almost.

Since Google Photos will automatically deduplicate uploaded images, we need to find a way to make the second image slightly different. Normally this would involve tweaking a comment or something, but in this case, the image has already been stripped down to its bare essentials.

My strategy was attempt to abuse the data compression to see if I could generate an image that compressed the same data into the same number of bytes, but differently. Fortunately, pngcrush can be told to use specific compression strategies (there are currently 177 of them). My hope is that at least one of these will achieve the same results, but in a different way. For starters we'll try the first 9:

1
2
3
for x in $(seq 9); do
    pngcrush -m "$x" pixel.png pixel_alt${x}.png;
done

A quick ls -l reveals that all of the pixel_alt*.png files are still 67 bytes. Now we just need to find one that has different data. sha1sum is the perfect utility for this:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
$ sha1sum pixel.png pixel_alt*.png
d99a9d63b1cd9e4b3f823d4d03144ccd95328f48  pixel.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt1.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt2.png
7487dcce2b2bb81a442faf139b0a547bf070d5e2  pixel_alt3.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt4.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt5.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt6.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt7.png
c3ea09bfcfcb36ce22d3f19eacada359f3984ed1  pixel_alt8.png
d99a9d63b1cd9e4b3f823d4d03144ccd95328f48  pixel_alt9.png

pixel_alt1.png looks like a good candidate, let's see what changed:

1
2
3
4
5
6
7
$ diff <(./pngview.py pixel.png) <(./pngview.py pixel_alt1.png)
20,21c20,21
<             data = b'\x08\x1dc`\x00\x00\x00\x02\x00\x01' (total 10)
<             crc = 3486004709
---
>             data = b'\x08[c`\x00\x00\x00\x02\x00\x01' (total 10)
>             crc = 1648381800

There's the same amount of data, but it's different. Let's check if it decompresses to the same image data:

1
2
3
4
5
6
>>> import zlib
>>> zlib.decompress(b'\x08\x1dc`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>> zlib.decompress(b'\x08[c`\x00\x00\x00\x02\x00\x01')
b'\x00\x00'
>>>

Yep, this means that pixel_alt1.png is the exact same image with the exact same size, but won't be deduplicated when uploading it to Google Photos since the compressed data is different.

Final touches

Now that we have 2 different tiny images, we can upload them to Google Photos and put them in an album so they can be pulled down by the Chromecast. Something to note is that you may have to mark the images as favorites to get them to be displayed. There seems to be some sort of AI-fueled "we know better than you" algorithm that initially refused to display my images until I starred them.

Now just set your ambient mode to display the album with your images in it (or your favorites album), set the slideshow speed to its maximum value (change image every 10 mins), and you're done.


Recovering data off a bootlooping Nexus 5

The situation

The Nexus 5 that I've had for almost 5 years started bootlooping repeatedly.

I've replaced/repaired multiple parts of it over the years but lately it had been more flaky than usual. For example:

  • Sometimes when booting it would show a "Firmware update in progress, don't disconnect your computer" message, even though there was nothing connected to the USB port.
  • It would randomly freeze and soft reboot when using it about once a week.
  • When propping it up on a book or something to angle the screen towards me, the screen would sometimes turn off and on (as it turns out, this was a warning sign)

I've been ready to replace it for a while, but couldn't justify it while this one was perfectly (ok, mostly) functional. That changed today when I tried to turn it on and it started bootlooping.

So buy a new phone and move on, right? Right, except I committed the cardinal sin of living in the digital age - not having complete backups. Yeah, yeah, I know.

I got most of it. I'd been using Syncthing to automatically back up photos, videos, and other files to my home server every night so they were all good. The problem was that I had been using FreeOTP-export to back up my 2FA secrets manually and the latest backup was missing a bunch of logins.

So I needed to somehow get into my non-booting phone and pull the data off it. At least because I was replacing it, all destructive options were on the table.

Diagnosis

The first step was seeing if other people had this problem. Some searching revealed that it was most likely the power button gone bad and being stuck in the pressed position.

To confirm this, I rapidly hammered the power button on a hard surface while booting, hoping that the jarring would un-stick it and let it boot. This sort of worked in that it would boot, but stopping at any point would result it it powering off again. It was also extremely difficult to tap it consistently enough to keep the phone on for long enough to pull any data off it.

The issue I mentioned earlier where propping the phone up made the screen turn on and off now made sense. I'm guessing this was causing a slight flexing of the power button switch, causing it to close and trigger. Over time this must've permanently bent something so it stayed closed, causing the bootloop.

Now that I had pretty much confirmed that it was something physically wrong with the power button circuitry, I messaged my friend Mike. When it comes to electronics, he's definitely more experienced than me. Plus, he lets me borrow his tools :)

The "fix"

The next day Mike came over with his soldering iron and we started brainstorming.

The plan of attack was to just take the power button off the board entirely. It can't always be pressed down if it's not on the board right?

In preparation for this, I wanted the phone to automatically boot when it was plugged into a charger so I wouldn't have to manually short the pins on the board. Fortunately this can be done with a fastboot command: fastboot oem off-mode-charge 0 (basically: don't allow charging while off, therefore turn on when charging).

After desoldering the power button from the main board, it was still bootlooping. Not good. I went back and cleaned up the solder to make absolutely sure that the power button signal pins weren't connected in any way. Still bootlooping when powered on. It would get to the bootloader and stay there, meaning the power button wasn't being pressed anymore, but when launching to either recovery or the main OS, it would immediately restart.

Looked into dumping data from the bootloader - impossible.

Reflashed the boot and recovery partitions - same thing.

Tried booting into recovery using fastboot boot <recovery-image> - bootloop.

sounds like it's time to cry

-- Someone on Freenode's #lineageos-dev channel after I explained the situation

Thankfully, it was not. Right after that, someone else asked if I had reconnected the battery. I had not.

Turns out that a working battery is required to boot the phone, even when it's plugged in. I had taken the battery out to desolder the power button and never put it back in since the phone seemed to boot fine without it. Whoops.

After plugging the battery into the board and booting the phone, it launched the main OS without any issues. Huge relief.

Recovering the data

With the phone booted up normally I used adb (running as root) to pull the 2FA codes and other important things that I knew I needed over to my laptop.

I really don't trust myself to remember everything so I booted the phone into recovery mode and pulled a complete backup of /data/data, /data/app, and /storage/sdcard (not actually an sdcard) to my laptop as well.

To make absolutely sure I didn't miss anything, I also pulled a raw image of the entire flash memory (disk-based encryption was not enabled) using adb pull /dev/block/mmcblk0 mmcblk0.img. Everything I need should be in the normal backups, but if not, I can always go spelunking through that image.

Lessons learned

  • If it's important, back it up automatically. If it's not automatic, at some point it will be missed.
  • Make sure you have the ability to get root access on every device you own before you need it - in this case I wouldn't've been able to pull the 2FA codes or do full backups without it.

Future plans

For the Nexus 5, I'm planning on either buying a replacement power button or maybe just soldering some wires to the exposed pads and snaking them out of the phone to an external button if I can't find one. The phone can then continue to be used as an app development testbed, a Chromecast remote, or as a basic emulation console until it dies for real. In theory I could keep using it as a phone too, but at this point I just don't trust it enough.

For my next phone, I've decided on the Xperia XZ1 Compact. It's a smaller phone that's mostly waterproof (IP68), has a fast SoC (Snapdragon 835), a headphone jack, SD card support, and great battery life. I have high hopes for it. Also, as evidenced by this post, having root access to the OS is pretty critical at times so I'll be flashing LineageOS on it ASAP.

© Carey Metcalfe. Built using Pelican. Theme is subtle by Carey Metcalfe. Based on svbhack by Giulio Fidente.