Proxmox VE is an open-source (AGPL v3) virtualization platform for running containers and VMs. It's comparable to a proprietary solution like VMware ESXi.

It can be downloaded and installed completely free of charge, lacking only access to support and other things that more enterprise-focused users care about. For a homelab install, the free version is perfect.

...except for one thing. Each time you log in, you have to click through this dialog:

The goal of this post is to walk through developing a solution to automatically remove that notification in any installed version of Proxmox VE, as well as have that removal survive future updates.

To skip to the solution, click here.

Developing a patch

Searching the internet reveals that a few other people (1, 2, 3) have already attacked this problem. All articles point to some code in /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js being responsible for the notification so this is where we'll start.

Scanning though that file reveals a function called checked_command(orig_cmd). This function just executes orig_cmd after showing the subscription notification if it determines that you don't have a valid subscription.

Based on this, the goal of this patch will be to make checked_command always execute orig_cmd without bothering to check if there is a subscription at all. As a bonus, this will also speed up the login process since it removes a blocking call to the server¹.

Since we want this patch to work for as many versions as possible, it's a good idea to take a look into how this function has changed over time. By looking at the proxmox-widget-toolkit history, we can see that checked_command was originally added in commit 5f93e010. However, it was actually moved from the pve-manager project where it was written for the initial implementation of the subscription notice. By looking at the entire history of this function we can see that while the function implementation has changed slightly over time, the name, arguments, and goal of it has stayed exactly the same.

Given this, as long as we only use the function definition to do the patch, it should work for every version of Proxmox VE to date (and hopefully into the future). Using sed to prepend orig_cmd(); return; to the function should work nicely:

1

sed --in-place 's/checked_command: function(orig_cmd) {$/& orig_cmd(); return;/' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js

This results in the following change:

1
2

-checked_command: function(orig_cmd) {
+checked_command: function(orig_cmd) { orig_cmd(); return;

Notice that the sed script is looking for the function definition followed by a line ending ($). Because the patch is added to the same line, if the command is run again it won't match and won't make any changes, making it safe to blindly run multiple times. This becomes important for automating it later.

Now that we know it works, we should harden it up against whitespace changes by replacing the spaces in the regex with \s* (0 or more whitespace characters):

1

sed --in-place 's/checked_command:\s*function(orig_cmd)\s*{\s*$/& orig_cmd(); return;/' /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js

The only thing left to solve is which file checked_command is defined in. Since it used to be a part of the pve-manager project, it's safe to say that it wasn't always stored in /usr/share/javascript/proxmox-widget-toolkit/proxmoxlib.js. It could also move around in the future.

To deal with this, we use grep to find the file that it's defined in, then run the previous command over that file:

1
2

grep --files-with-matches --include '*.js' --recursive --null 'checked_command:\s*function(orig_cmd)' /usr/share/ \
  | xargs --null --no-run-if-empty sed --in-place 's/checked_command:\s*function(orig_cmd)\s*{\s*$/& orig_cmd(); return;/'

Simply run this command on your Proxmox VE server², refresh the web UI, and the notice should be gone!

Automating it

At this point we have a working command that will remove the subscription notice from any version of Proxmox VE to date. However, after updating the system, there's a chance that a newer version of the file will have overwritten our patched version.

While we could just save the command in a script and manually run it after every update, that's annoying and will definitely be forgotten. Instead, we want to automatically run the command every time the web server starts. This will ensure that it will always be serving a patched version of the file.

Proxmox VE uses systemd to manage the startup of its services. This means that once we find the service that manages the web server, we should be able to make systemd launch a service that applies our patch just before the web server.

The first step is to find the service that manages the web server. Since we know it's running on port 8006, we can use netstat to enumerate all listening TCP ports and grep to filter it down to just the port we're interested in:

1
2

$ netstat --listening --tcp --numeric-ports --program | grep 8006
tcp  0  0  0.0.0.0:8006  0.0.0.0:*  LISTEN  15771/pveproxy

Once we know the PID of the program, running systemctl status <PID> will show the status (including the service name) of the service that's managing it³. In this case it shows that the service is pveproxy.service.

We can now define a service that runs our code, is wanted by pveproxy.service, and must be run before it. Then, whenever systemd starts (or restarts) pveproxy.service, it will make sure to also run our service that applies the patch.

The solution

Putting it all together:

1. Create /etc/systemd/system/no-subscription-notice.service:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

[Unit]
Description=Remove Proxmox VE subscription notice
Before=pveproxy.service

[Service]
Type=oneshot
ExecStart=/bin/sh -c "grep --files-with-matches --include '*.js' --recursive --null 'checked_command:\s*function(orig_cmd)' /usr/share/ | xargs --null --no-run-if-empty sed --in-place 's/checked_command:\s*function(orig_cmd)\s*{\s*$/& orig_cmd(); return;/'"

[Install]
WantedBy=pveproxy.service

2. systemctl enable --now no-subscription-notice.service

And that's it! The subscription notice should now be gone for good. I'll update this post if it breaks, but that hopefully won't be for a while.

Background

Both this site and my home server use HTTPS certificates provided by Let's Encrypt. Previously I was using the http-01 challenge method, but once wildcard certificates became available, I decided to switch to the dns-01 method for simplicity.

My domains are currently registered through Namecheap which provides an API for modifying DNS records. The only catch is that it can only be accessed via manually-whitelisted IP addresses. Because the server I'm trying to access the API from is assigned a dynamic IP, this restriction was a bit of an issue.

Back in November when I initially made the switch to the DNS-based challenge, I set it up the easy way: I manually added my current IP to the whitelist, added a TODO entry to fix it somehow, and set a reminder scheduled for the week before the cert would expire telling me to update the IP whitelist. Fast-forward to today when I was reminded to update my IP whitelist. Instead of continuing to kick the can down the road, I decided to actually fix the issue.

Setting up a temporary proxy

My home server is connected to the internet though a normal residential connection which is assigned a dynamic IP address. However, the server that I run this website on is configured with a static IP since it's a hosted VPS. By proxying all traffic to the Namecheap API through my VPS, I could add my VPS's static IP to the whitelist and not have to worry about my home IP changing all the time.

SSH is perfect tool for this. The OpenSSH client supports port forwarding using the -D flag. By then setting the HTTP[S]_PROXY environment variables to point to the forwarded port, programs that support those environment variables will transparently forward all their requests through the proxy.

After a bunch of research and testing, I came up with the following script to easily set up the temporary proxy:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20

#!/bin/sh
# Source this file to automatically setup and teardown an HTTP* proxy through cmetcalfe.ca
# Use the PROXY_PORT and PROXY_DEST environment variables to customize the proxy

PROXY_PORT="${PROXY_PORT:-11111}"
PROXY_DEST="${PROXY_DEST:-<username>@cmetcalfe.ca}"
PID="$$"

# Teardown the SSH connection when the script exits
trap 'ssh -q -S ".ctrl-socket-$PID" -O exit "$PROXY_DEST"' EXIT

# Set up an SSH tunnel and wait for the port to be forwarded before continuing
if ! ssh -o ExitOnForwardFailure=yes -M -S ".ctrl-socket-$PID" -f -N -D "$PROXY_PORT" "$PROXY_DEST"; then
    echo "Failed to open SSH tunnel, exiting"
    exit 1
fi

# Set environment variables to redirect HTTP* traffic through the proxy
export HTTP_PROXY="socks5://127.0.0.1:$PROXY_PORT"
export HTTPS_PROXY="$HTTP_PROXY"

This script is saved as ~/.config/tempproxy.rc and can be sourced to automatically set up a proxy session and have it be torn down when the script exits.

You'll want to use key-based authentication with an unencrypted private key so that you don't need to type anything to initiate the SSH session. For this reason you'll probably want to create a limited user on the target system that can only really do port forwarding. There's a good post on this here.

Talking to the API through the proxy

To allow programs that use the tempproxy.rc script to talk to the Namecheap API, the IP address of the VPS was added to the whitelist. Now that the proxying issue was taken care of, I just needed to wire up the actual certificate renewal process to use it.

The tool I'm using to talk to the Namecheap DNS API is lexicon. It can handle manipulating the DNS records of a ton of providers and integrates really nicely with my ACME client of choice, dehydrated. Also, because it's using the PyNamecheap Python library, which in turn uses requests under the hood, it will automatically use the HTTP*_PROXY environment variables when making requests.

The only tricky bit is that the base install of lexicon won't automatically pull in the packages required for accessing the Namecheap API. Likewise, requests won't install packages to support SOCKS proxies. To install all the required packages you'll need to run a command like:

1

pip install 'requests[socks]' 'dns-lexicon[namecheap]'

Since lexicon can use environment variables as configuration, I created another small source-able file at ~/.config/lexicon.rc:

1
2
3
4
5
6
7
8

#!/bin/sh
# Sets environment variables for accessing the Namecheap API

# Turn on API access and get an API token here:
# https://ap.www.namecheap.com/settings/tools/apiaccess/
export PROVIDER=namecheap
export LEXICON_NAMECHEAP_USERNAME=<username>
export LEXICON_NAMECHEAP_TOKEN=<api token>

With that in place, the existing automation script that I had previously set up when I switched to using the dns-01 challenge just needed some minor tweaks to source the two new files. I've provided the script below, along with some other useful ones that use the DNS API.

Scripts

do-letsencrypt.sh

This is the main script that handles all the domain renewals. It's called via cron on the first of every month.

Fun fact: The previous http-01 version of this script was a mess - it involved juggling nginx configurations with symlinks, manually opening up the .well-known/acme-challenge endpoint, and some other terrible hacks. This version is much nicer.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30

#!/bin/sh
# Generate/renew Let's Encrypt certificates using dehydrated and lexicon

set -e

# Set configuration variables and setup the proxy
source ~/.config/lexicon.rc
source ~/.config/tempproxy.rc

cd /etc/nginx/ssl

# Update the certs.
# Hook script copied verbatim from
# https://github.com/AnalogJ/lexicon/blob/master/examples/dehydrated.default.sh
if ! dehydrated --accept-terms --cron --challenge dns-01 --hook dehydrated.default.sh; then
    echo "Failed to renew certificates"
    exit 1
fi

# Reload the nginx config if it's currently running
if ! systemctl is-active nginx.service > /dev/null; then
    echo "nginx isn't running, not reloading it"
    exit 0
fi

if ! systemctl reload nginx.service; then
    systemctl status nginx.service
    echo "Failed to reload nginx config, check the error log"
    exit 1
fi

dns.sh

This one is really simple - it just augments the normal lexicon CLI interface with the proxy and configuration variables.

1
2
3
4
5
6
7

#!/bin/sh
# A simple wrapper around lexicon.
# Sets up the proxy and configuration, then passes all the arguments to the configured provider

source ~/.config/lexicon.rc
source ~/.config/tempproxy.rc
lexicon "$PROVIDER" "$@"

1
2
3
4
5

$ ./dns.sh list cmetcalfe.ca A
ID       TYPE NAME             CONTENT         TTL
-------- ---- ---------------- --------------- ----
xxxxxxxx A    @.cmetcalfe.ca   xxx.xxx.xxx.xxx 1800
xxxxxxxx A    www.cmetcalfe.ca xxx.xxx.xxx.xxx 1800

dns-update.sh

An updated version of my previous script that uses the API instead of the DynamicDNS service Namecheap provides. Called via cron every 30 minutes.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

#!/bin/sh
# Check the server's current IP against the IP listed in its DNS entry.
# Set the current IP if they differ.

set -e

resolve() {
    line=$(drill "$1" @resolver1.opendns.com 2> /dev/null | sed '/;;.*$/d;/^\s*$/d' | grep "$1")
    echo "$line" | head -1 | cut -f5
}

dns=$(resolve <subdomain>.cmetcalfe.ca)
curr=$(resolve myip.opendns.com)
if [ "$dns" != "$curr" ]; then
    source ~/.config/lexicon.rc
    source ~/.config/tempproxy.rc
    if ! lexicon "$PROVIDER" delete cmetcalfe.ca A --name "<subdomain>.cmetcalfe.ca"; then
        echo "Failed to delete old DNS record, not setting the new one."
        exit 1
    fi
    if lexicon "$PROVIDER" update cmetcalfe.ca A --name "<subdomain>.cmetcalfe.ca" --content "$curr" --ttl=900; then
        echo "Server DNS record updated ($dns -> $curr)"
    else
        echo "Server DNS record update FAILED (tried $dns -> $curr)"
    fi
fi

In addition to running this website, I also run a home server. For convenience, I point a subdomain of cmetcalfe.ca at it so even though it's connected using a dynamic IP (and actually seems to change fairly frequently), I can get access to it from anywhere.

As a bit of background, the domain for this website is registered and managed through Namecheap. While they do provide a recommended DDNS client for keeping a domain's DNS updated, it only runs on Windows.

Instead, after enabling DDNS for the domain and reading Namecheap's article on using the browser to update DDNS I came up with the following dns-update script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/bin/sh

# Abort if anything goes wrong (negates the need for error-checking)
set -e

# Uses drill instead of dig
resolve() {
    #dig "$1" @resolver1.opendns.com +short 2> /dev/null
    line=$(drill "$1" @resolver1.opendns.com 2> /dev/null | sed '/;;.*$/d;/^\s*$/d' | grep "$1")
    echo "$line" | head -1 | cut -f5
}

dns=$(resolve <subdomain>.cmetcalfe.ca)
curr=$(resolve myip.opendns.com)
if [ "$dns" != "$curr" ]; then
    if curl -s "https://dynamicdns.park-your-domain.com/update?host=<subdomain>&domain=cmetcalfe.ca&password=<my passkey>" | grep -q "<ErrCount>0</ErrCount>"; then
        echo "Server DNS record updated ($dns -> $curr)"
    else
        echo "Server DNS record update FAILED (tried $dns -> $curr)"
    fi
fi

It basically checks if the IP returned by a DNS query for the subdomain matches the current IP of the server (as reported by an OpenDNS resolver) and if it doesn't, sends a request to update the DNS. The echo commands are there just to output some record of the IP changing. Maybe I'll do some analysis of it at some point.

To run the script every 30 minutes and redirect any output from it to the syslog, the following crontab entry can be used:

1

*/30 * * * * /path/to/dns-update | /usr/bin/logger -t dns-update

With the script automatically running every 30 minutes I can now be confident that my subdomain will always be pointing at my home server whenever I need access to it.