Preface

In this entry I'll use two command line tools (curl and jq) to request currency conversion rates from the Bank of Canada. Now obviously getting the same data through the normal web interface is possible as well. However, doing it this way returns the data in a machine-readable form, allowing it to be customized or used as input into other programs.

The basic flow will be request the data using curl, then pipe it to jq for processing.

We'll go from seeing the raw data the server returns to something that we can more easily use. To skip to the final result, click here.

Lets go!

Hitting the server without any processing will give the following result:

1

curl -s "https://www.bankofcanada.ca/valet/observations/FXUSDCAD/json?start_date=2017-01-01&end_date=2017-01-10"

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
"terms":{
    "url": "http://www.bankofcanada.ca/terms/"
},
"seriesDetail":{
"FXUSDCAD":{"label":"USD/CAD","description":"US dollar to Canadian dollar daily exchange rate"}
},
"observations":[
{"d":"2017-01-02","FXUSDCAD":{"e":-64}},
{"d":"2017-01-03","FXUSDCAD":{"v":1.3435}},
{"d":"2017-01-04","FXUSDCAD":{"v":1.3315}},
{"d":"2017-01-05","FXUSDCAD":{"v":1.3244}},
{"d":"2017-01-06","FXUSDCAD":{"v":1.3214}},
{"d":"2017-01-09","FXUSDCAD":{"v":1.3240}},
{"d":"2017-01-10","FXUSDCAD":{"v":1.3213}}
]
}

To get the information we want, we need to iterate over the elements in the observations list and map the date (d) to the conversion rate (FXUSDCAD.v). Since this will create a list of single-element dictionaries, we'll also need to "add" (merge) them together.

To do this with jq, it looks like this:

1
2
3
4

curl -s "https://www.bankofcanada.ca/valet/observations/FXUSDCAD/json?start_date=2017-01-01&end_date=2017-01-10" |\
jq '
    [.observations[] | {(.d) : .FXUSDCAD.v}] | add
'

1
2
3
4
5
6
7
8
9

{
  "2017-01-02": null,
  "2017-01-03": 1.3435,
  "2017-01-04": 1.3315,
  "2017-01-05": 1.3244,
  "2017-01-06": 1.3214,
  "2017-01-09": 1.324,
  "2017-01-10": 1.3213
}

That first null entry where there was no data is going to cause problems later, so let's just delete it using the del function:

1
2
3
4

curl -s "https://www.bankofcanada.ca/valet/observations/FXUSDCAD/json?start_date=2017-01-01&end_date=2017-01-10" |\
jq '
    [.observations[] | {(.d) : .FXUSDCAD.v}] | add | del(.[] | nulls)
'

1
2
3
4
5
6
7
8

{
  "2017-01-03": 1.3435,
  "2017-01-04": 1.3315,
  "2017-01-05": 1.3244,
  "2017-01-06": 1.3214,
  "2017-01-09": 1.324,
  "2017-01-10": 1.3213
}

Now this may be good enough for most purposes, but it would be useful to compute some other statistics from this data. For example, we want to know the average exchange rate for the date range. Now that we have the data in an easy to use format, computing an average with jq is just a matter of passing the values to an add / length filter. For example:

1
2
3
4

curl -s "https://www.bankofcanada.ca/valet/observations/FXUSDCAD/json?start_date=2017-01-01&end_date=2017-01-10" |\
jq '
    [.observations[] | {(.d) : .FXUSDCAD.v}] | add | del(.[] | nulls) | add / length
'

1

1.327683333333333

We can now construct an object that has both the average, as well as all the data. Putting it all together we get:

1
2
3
4
5
6
7
8

curl -s "https://www.bankofcanada.ca/valet/observations/FXUSDCAD/json?start_date=2017-01-01&end_date=2017-01-10" |\
jq '
    [.observations[] | {(.d) : .FXUSDCAD.v}] | add | del(.[] | nulls) |
    {
        "average": (add / length),
        "values": .
    }
'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

{
  "average": 1.327683333333333,
  "values": {
    "2017-01-03": 1.3435,
    "2017-01-04": 1.3315,
    "2017-01-05": 1.3244,
    "2017-01-06": 1.3214,
    "2017-01-09": 1.324,
    "2017-01-10": 1.3213
  }
}

And that's it! Hopefully you found this useful, if not for it's stated purpose of retrieving exchange rates, then at least for getting more familiar with jq. It's pretty much the best tool out there when it comes to manipulating JSON on the command line.

Appendix

Date ranges

• The server will only ever return data from 2017-01-03 and onwards. If the start range is before that, it doesn't return an error, it just doesn't return any data for those dates.
• If the end date isn't specified, the server assumes the current date.
• There doesn't seem to be any limit on the amount of data retrieved

Currency codes

A list of codes can be easily scraped from the lookup page. The command and its result are below for reference.

1

curl -s "https://www.bankofcanada.ca/rates/exchange/daily-exchange-rates-lookup/" | grep -Eo '"FX......".*<'

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

"FXAUDCAD">Australian dollar<
"FXBRLCAD">Brazilian real<
"FXCNYCAD">Chinese renminbi<
"FXEURCAD">European euro<
"FXHKDCAD">Hong Kong dollar<
"FXINRCAD">Indian rupee<
"FXIDRCAD">Indonesian rupiah<
"FXJPYCAD">Japanese yen<
"FXMYRCAD">Malaysian ringgit<
"FXMXNCAD">Mexican peso<
"FXNZDCAD">New Zealand dollar<
"FXNOKCAD">Norwegian krone<
"FXPENCAD">Peruvian new sol<
"FXRUBCAD">Russian ruble<
"FXSARCAD">Saudi riyal<
"FXSGDCAD">Singapore dollar<
"FXZARCAD">South African rand<
"FXKRWCAD">South Korean won<
"FXSEKCAD">Swedish krona<
"FXCHFCAD">Swiss franc<
"FXTWDCAD">Taiwanese dollar<
"FXTHBCAD">Thai baht<
"FXTRYCAD">Turkish lira<
"FXGBPCAD">UK pound sterling<
"FXUSDCAD">US dollar<
"FXVNDCAD">Vietnamese dong<

In addition to running this website, I also run a home server. For convenience, I point a subdomain of cmetcalfe.ca at it so even though it's connected using a dynamic IP (and actually seems to change fairly frequently), I can get access to it from anywhere.

As a bit of background, the domain for this website is registered and managed through Namecheap. While they do provide a recommended DDNS client for keeping a domain's DNS updated, it only runs on Windows.

Instead, after enabling DDNS for the domain and reading Namecheap's article on using the browser to update DDNS I came up with the following dns-update script.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21

#!/bin/sh

# Abort if anything goes wrong (negates the need for error-checking)
set -e

# Uses drill instead of dig
resolve() {
    #dig "$1" @resolver1.opendns.com +short 2> /dev/null
    line=$(drill "$1" @resolver1.opendns.com 2> /dev/null | sed '/;;.*$/d;/^\s*$/d' | grep "$1")
    echo "$line" | head -1 | cut -f5
}

dns=$(resolve <subdomain>.cmetcalfe.ca)
curr=$(resolve myip.opendns.com)
if [ "$dns" != "$curr" ]; then
    if curl -s "https://dynamicdns.park-your-domain.com/update?host=<subdomain>&domain=cmetcalfe.ca&password=<my passkey>" | grep -q "<ErrCount>0</ErrCount>"; then
        echo "Server DNS record updated ($dns -> $curr)"
    else
        echo "Server DNS record update FAILED (tried $dns -> $curr)"
    fi
fi

It basically checks if the IP returned by a DNS query for the subdomain matches the current IP of the server (as reported by an OpenDNS resolver) and if it doesn't, sends a request to update the DNS. The echo commands are there just to output some record of the IP changing. Maybe I'll do some analysis of it at some point.

To run the script every 30 minutes and redirect any output from it to the syslog, the following crontab entry can be used:

1

*/30 * * * * /path/to/dns-update | /usr/bin/logger -t dns-update

With the script automatically running every 30 minutes I can now be confident that my subdomain will always be pointing at my home server whenever I need access to it.

Having Git autocomplete branch names when doing a checkout is super useful. Having the autocomplete hang for 30 seconds because it has to look up all 3000 of the remote branches in a massive repo is not so useful. It's actually pretty frustrating.

My solution: changing the git checkout autocomplete to only look at local branches, while using a git checkoutr alias to preserve the original behaviour (because sometimes it's actually needed).

How it's done:

Define an alias for checkout

We're going to use the same checkout command for the remote checkout, but need to have a different command so the script can differentiate between them. Making an alias does exactly this.

Create the alias by running git config --global alias.checkoutr checkout

Change the autocompletion function

The checkout autocomplete behaviour is defined in a function called _git_checkout in the git autocompletion file. We're going to override the function with our own version that has different autocomplete logic in it.

The location of the file varies over different operating systems and configurations, but here are a few spots to look:

• /etc/bash_completion.d/git
• /usr/share/bash-completion/completions/git

For a brew-installed git autocomplete on macOS, the file will probably be $(brew --prefix)/etc/bash_completion.d/git-completion.bash

Once you've found the file, copy the entire _git_checkout function into your .bashrc (or equivalent non-login shell startup script). Now look for the line

__git_complete_refs $track_opt

We're going to replace that line with:

1
2
3
4
5

if [ "$command" = "checkoutr" ]; then
    __git_complete_refs $track_opt
else
    __gitcomp_direct "$(__git_heads "" "$cur" " ")"
fi

After saving and re-sourcing your .bashrc file, git will autocomplete local branches and tags when using git checkout, but will go back to the default behaviour of autocompleting all references when using git checkoutr.

Credits to a combination of answers on this StackOverflow post.

EDIT 2017-10-02: Updated to work with git v2.13.0 thanks to Alexander Ko's comment.
EDIT 2018-03-24: Updated to speed up branch and tag completion.