Setting up rdiff-backup on FreeBSD 12.1

My main PC workstation (as opposed to my Mac Pro) is a dual-boot Windows and Linux machine. While backing up the Windows portion is relatively easy via some cheap-ish commercial backup software, I ended up backing up my Linux home directories only very occasionally. Clearly, Something Had To Be Done ™.

I had a look around for Linux backup software. I was familiar with was Timeshift, but at least the Manjaro port can’t back up to a remote machine and was useless as a result. I eventually settled on rdiff-backup as it seemed to be simple, has been around for a while and also looks very cron-friendly. So far, so good.

Read More

MongoDB tips – How to find documents containing a non-empty array in MongoDB

Why do we need another post cluttering up the Interpipes on how to find a set of documents in a MongoDB collection that contain a non-empty array field? It’s not like we suddenly have a shortage of posts and articles on this topic after all. Well, it maybe a shocking revelation but not all of these posts and Stack Overflow answers are, well, as correct as I’d like them to be.

Go on, what’s wrong with the typical approach then?

When you search for ‘how to find MongoDB documents with non-empty arrays’, you usually end up finding suggestions like db.test.find({ my_huge_array: { $exists: true, $ne: [] } }) or variations thereof. And this approach works most of the time, especially if you have a fairly rigid schema and a little helping of luck. The only problem with this approach is that, well, it’s almost correct but not correct enough to be robust. But, I hear you say you’ve used this find clause in production for years and it just works, what I am talking about?

Read More

Building an OpenBSD WireGuard VPN server part 3 – Unbound DNS filtering

In part 2, I reconfigured my WireGuard VPN to use an Unbound DNS server on the VPN server rather than rely on a third party server I had used for the original quick and dirty configuration. It was important for me to set up a validating DNS server, which I did in that part.

In this part, I’m extending the existing configuration to include some basic block lists for known ad and tracking servers. As I’m mainly trying to use the VPN while on the road, I want to ensure that anything I end up doing using the VPN is as secure as I can make it with reasonable effort. That makes tracking and preventing malicious ads the next step. That said, I’m not planning to go for a full Pi-Hole like setup. Initially, I am trying to do is integrate one known good blocklists into the Unbound configuration and automate the process. I can get fancy with a more Pi-Hole like setup later if I want to.

Picking a DNS filter list and using it from Unbound

I started by using a single DNS block list from StevenBlack’s github repo. deadc0de.re’s blog has a good how-to post including the necessary awk incantations for converting the file into the format Unbound needs. I used that blog post as my as my starting point.

The process overall is relatively simple. First, pick the flavour of blocklist you want. Download the hosts file for the blocklist. I used wget to download it as it’s already installed on my OpenBSD system, but curl would also work. My assumption is that you’re running these commands in /var/unbound/etc, so all the absolute paths refer to that location.

wget -O /tmp/blocklist-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

This list is in the traditional host file format, so we need to convert it into the format that Unbound needs. Fortunately, deadc0de.re’s blog post contains the necessary code to accomplish that transformation:

cat /tmp/blocklist-hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > adblocker.conf

We also need to update our unbound.conf file to include the freshly generated host blocklist. To do that, we have to add an include statement to the server section of the configuration file. The change looks like this:


server:
  ...
  include: /var/unbound/etc/block-ads.conf

I also had to set up unbound-control to ensure that I didn’t have to restart the server every time I updated the block list. Thus, I had to make a slight detour that involved running unbound-control-setup to generate the necessary certificates. I also needed to add the following statement to my unbound.conf:

remote-control:
  control-enable: yes

Note that remote-control is its own section and is not part of the server: section. The default setup only accepts remote control connections on the loopback interface. This is perfect for my setup, so I didn’t need to make any other changes.
After restarting unbound to pick up the configuration changes I was able to test the VPN with the new DNS blocking and it proved to be mostly effective on the websites I visited.

Automating the process

Of course I’m a software engineer by background, so clearly I wanted to automate the process. Fortunately this is about as complicated as pasting the commands I listed above into a shell script and periodically run it. My script looks like this:

#!/bin/sh

cd /var/unbound/etc
wget -O /tmp/black-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
cat /tmp/black-hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > block-ads.conf
rm /tmp/black-hosts
unbound-control reload

I’m running this script via cron once a week, as that seems to roughly correspond to the update frequency of the list. I may change that in the future, especially if I end up combining the multiple block lists instead. In the meantime I don’t think it’s necessary to run the cron script more often.

What’s next?

Right now it looks like WireGuard VPN server is working as intended and does what I need it to. TuM’Fatig has a great blog post on expanding the approach I’m using to include all of the public block lists that Pi-Hole is using, so I’m probably going to look into that sooner or later. Other than that and keeping the machine up to date, the next project will be to create orchestration scripts to recreate the server from scratch. I found setting up WireGuard on OpenBSD much easier than doing it on a Linux host. Nevertheless having a Terraform or Ansible script that allowed me to quickly stand one up from scratch if the current instance was damage would likely be a good thing.

Would I go the OpenBSD route again?

I had mentioned at the start of this series that I hadn’t had done much with OpenBSD for years. Was it worth choosing OpenBSD over FreeBSD, which I use on a very regular basis? Maybe, maybe not. OpenBSD required less additional fiddling to secure it out of the box. Some of the time saved was taken up by reacquainting myself with the system, so I’d call that a wash. I suspect that setting up the actual WireGuard VPN instance would take the same amount of time on FreeBSD as I’d run the same commands, so no clear advantage there. I do like the newer version of pf that comes with OpenBSD – FreeBSD’s is a few versions behind and is missing some features, and I doubt it’ll ever catch up.

Put that down as a ‘maybe’.

How well does WireGuard work for me?

So far it is proving performant enough to run on one of Vultr’s $5 instances that aren’t exactly swimming in computing power. Of course this is only a personal VPN and I’m not running a whole company’s worth of traffic across it. That would very likely require more oomph (technical term) on the server, but overall WireGuard VPN appears to be both very lightweight and more importantly, easy to configure both at the server and client end.

Throughput is good enough to use it to watch YouTube videos in good quality on crappy hotel Wifi without visibly dropping frames. That’s good enough for me as a performance metric because it shows that the VPN server is not the bottleneck. The real use cases for the VPN really are for accessing more sensitive information like financial information, and that is usually all interactive anyway.

Building an OpenBSD WireGuard VPN server part 2 – Unbound DNS setup

In the first part, I described how I set up the basic OpenBSD WireGuard VPN server. I also hinted that I wanted to set up my own validating, filtering DNS server. With a little bit of spare time during the holidays I decided now was a good time as any.

Making sure the VPN server uses the local Unbound DNS resolver first

Before I set up Unbound itself, I need to make sure that the VPN server’s resolv.conf points at its local DNS server first.

My Vultr VPS has a static IP, which it receives via DHCP. It also receives information about its upstream DNS servers via DHCP. So as the first step, we need to update /etc/dhclient.conf to ensure that resolv.conf lists our local server first. We do this by adding the following line to /etc/dhclient.conf:

prepend domain-name-servers 127.0.0.1;

In fact, that is the complete dhclient.conf file on my VPN server.

Setting up Unbound as a validating DNS server

OpenBSD already includes the Unbound authoritative DNS server, so it’s a matter of configuring it to act as the local DNS server and also to enable validation. Everything you need to run the server is already on the system.

Root server list and trust anchor files

Once we told the local system to use the local Unbound as its DNS server, we need to configure it. Note that dhclient will still add the upstream provider DNS server to resolv.conf, so it is still there as a fallback.

Time to set up Unbound. A validating DNS server needs a few more pieces of data compared to a standard recursive DNS server. The two important items are the trust anchor and the root hints so Unbound knows where to find the root name servers. Note that OpenBSD runs Unbound in a sandbox and thus all the configuration is under /var/unbound.

We start with retrieving the root.hints file that contains the list of root DNS servers:

wget https://www.internic.net/domain/named.root -O /var/unbound/etc/root.hints

A quick check with more /var/unbound/etc/root.hints shows that we received the expected list of root servers.

The second part we need is the trust anchor, which is the cryptographic key our DNS server needs so it can validate DNSSEC. After a couple of false starts I simply ended up using unbound-anchor to download the trust anchor and configured Unbound via the auto-trust-anchor setting to automatically check and update the trust anchor when necessary.

Pretty much all of the Internet tutorials suggest that the trust anchor should be stored in /var/unbound/etc. In my OpenBSD setup, unbound can’t write to that directory so I ended up sotring it in /var/unbound/db instead.

Unbound.conf minimal setup for validating server

Now we’re finally ready to configure unbound itself. My basic, stripped down version of the configuration looks like this:


server:
  verbosity: 1
  #
  # Interfaces and access control. Make sure we're not suddenly
  # running an open public resolver
  #
  outgoing-interface: 'real' server ip'
  interface: 127.0.0.1
  interface: 192.168.1.1         # Internal VPN interface
  access-control: 127.0.0.0/8    allow
  access-control: 192.168.1.0/24 allow
  #
  # Basic configuration
  #
  port: 53
  do-ip4: yes
  do-ip6: no                     # VPN server currently doesn't support IPV6
  do-udp: yes
  do-tcp: yes
  #
  # Root servers, trust anchor and validation logging
  #
  root-hints: "/var/unbound/etc/root.hints"
  auto-trust-anchor-file: "/var/unbound/db/root.key"
  val-log-level: 2
  val-clean-additional: yes
  #
  # Some basic hardening
  #
  hide-identity: yes
  hide-version: yes
  harden-glue: yes
  harden-dnssec-stripped: yes
  use-caps-for-id: yes
  unwanted-reply-threshold: 10000
  #
  # Ensure we don't return private addresses to prevent DNS
  # rebinding attacks
  #
  private-address: 10.0.0.0/8
  private-address: 172.16.0.0/12
  private-address: 192.168.0.0/16
  #
  # Other settings. I don't expect to see a lot of DNS traffic so
  # I turned off prefetching
  #
  prefetch: no
  #
  num-threads: 1

Now that we have a working configuration file, we need to enable and start Unbound:

vpn# rcctl enable unbound
vpn# rcctl start unbound
unbound(ok)

Testing DNSSEC validation

At this point we have a working server with supposedly working DNSSEC validation. Obviously we work on ‘trust, but verify’. To check that we have indeed a working validating server, we can run the following command:

dig www.nic.cz. +dnssec

The header section of the result should look like this:

; <<>> DiG 9.4.2-P2 <<>> www.nic.cz. +dnssec
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18417
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

See the bolded ‘ad’ in the flags line? Now compare this to the output of the same command, but run on my MacBook using the ISP’s resolver:

; <<>> DiG 9.10.6 <<>> www.nic.cz. +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12527
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

The ISP’s resolver doesn’t support DNSSEC in this case, so you can see the ‘ad’ flag missing. That flag indicates that the result from the upstream server validated.

Update the WireGuard config on the client

After I verified that the validation is working, it is time to update the WireGuard configuration on all clients so they point at our shiny new validating server rather than the third party DNS servers I used in the first installment.

Summary

At this point I have a validating DNS server, however it still passes through requests for every domain including the typical ad slinging domains. In the next installment of this series I’ll add filtering rules to try to get rid of most, if not all of them.

Building an OpenBSD Wireguard server

In my previous post, I mentioned that I somehow ended up with a corrupted filesystem on the WireGuard server I had set up earlier this year. That iteration of my VPN server was built on Linux as I expected I would get better performance using the kernel-based WireGuard implementation. It had taken me a while to set it up right, and I didn’t get the impression that the performance was so much better anyway. Keep in mind that I mostly use my VPN server from hotel WiFi and we all know how “good” that tends to be performance wise.

While I’ve done a fair bit of Linux admin work, I didn’t fancy re-doing the whole setup again. I also hadn’t scripted it up using Ansible or similar. I tend to prefer BSD anyway, and most of my personal servers run some flavour of BSD Unix. As I didn’t want to spend too much time securing this server, I used OpenBSD as it is a little more secure out of the box compared to FreeBSD. I also hadn’t experimented with OpenBSD for a while so I was curious to see the more recent improvements.

OpenBSD WireGuard Server setup at Vultr

I re-used the VPS I already had set up for the old Linux WireGuard VPN server at Vultr. Heck, it was corrupted already so formatting it was the only choice. In the interest of getting the VPN up and running quickly again, I used Vultr’s preconfigured OpenBSD image. With hindsight I probably wouldn’t do that again. More about this later. I followed a combination of instructions, mainly from the Cryptsus blog post on setting up WireGuard on an OpenBSD 6.6 server, but also referencing Jasper.La’s blog and Ankur Kothari’s blog. Setting up WireGuard on OpenBSD took me a lot less time than configuring the Linux version. I think it took me an hour or two to get the basic VPN tunnel up and working, including configuring the macOS client.

Overall I found this approach simpler than setting up WireGuard on Linux. I think there are two reasons for this – using the precompiled binaries for the user mode processes means you don’t have to futz around with kernel modules, third party repos and all that fun stuff, plus I personally find FreeBSD and to a certain extent OpenBSD easier to set up. This is obviously a personal preference. Plus, no systemd :).

Notes on the default Vultr OpenBSD Image

The Cryptsus instruction include setting up OpenBSD using full disk encryption. This is more secure than using Vultr’s default, preconfigured VPS image as that one doesn’t support full disk encryption out of the box. Using the preconfigured image makes the installation process easier and quicker, but at the expense of not getting the full official OpenBSD configuration out of the box. With hindsight, I probably what should have used a custom (well, the official) OpenBSD 6.6 image instead of the preconfigured Vultr one, but as mentioned I was in a bit of a rush and it was a case of “good enough and working” trumping “more secure”. When setting up a VPN server, there are basically multiple issues you want to protect yourself against:

  1. Someone remotely logging into your VPN server and getting hold of the encryption keys, and/or being able to capture traffic on your VPN server. That’s obviously the worst-case scenario from a security and privacy perspective, and the main one that I wanted to protect myself against. For that, I rely on OpenBSD’s security with strong passwords and key-based SSH authentication to log into the machine.
  2. Someone getting hold of the disk image – that’s what the full disk encryption part protects against. Something I will want to add to the server sooner or later – it does require a reinstall first, though. The risk of this is considerably lower than someone breaking into the VPN server itself and requires either an infrastructure fluke or an infrastructure compromise.
  3. Someone getting a level of access to the VPS host such that they can observe the running VPS and dump out its memory. Again, that’s possible but it requires a lot of work to get there. I like to think that there are a lot juicier targets than myself out there that would warrant this level of effort. So for right now, I’m discounting this threat model.

I’ve got the first point covered right now, will probably address the second one at some point and do my best to ignore the third one for now.

What’s still left to be done

The basic setup for my OpenBSD WireGuard server is up and running, and I’ve successfully used it while traveling. It’s definitely fast enough even on one of Vultr’s $5 instances. Most of the time, when I use the VPN I don’t need lots of bandwidth, but even when “testing” by watching YouTube videos, the performance was more than good enough.

I’m currently using the AdGuard DNS servers that were mentioned in the Cryptsus blog post I linked to above, but I really want to move to an Unbound DNS server on the VPN host itself that a) validates domains were possible and b) uses something like the Pi-hole blocklists to block ads and trackers. The latter is somewhat optional as I don’t really use the VPN for normal browsing, but it’s a definite nice to have.

Looks like I get to redo my WireGuard VPN server

I’ve blogged about setting up a WireGuard VPN server earlier this year. It’s been running well since, but I needed to take care of some overdue maintenance tasks. Trying to log into the server this morning and I am greeted with “no route to host”. Eh? A quick check on my Vultr UI showed that the VPS had trouble booting. The error suggests a corrupted boot drive. Oops.

Guess what the maintenance task I was looking at was? Creating an Ansible script so I’d be able to stand up the server from scratch in case something like this happened. And yes, the irony of being the guy who regularly preaches to his clients about the need for backups doesn’t quite escape me.

Anyway, at least this gives me an excuse to set up my WireGuard server on OpenBSD. This is something I’ve been thinking about for a while so now I have the perfect excuse for it. I realise that OpenBSD can only use the user space daemon for WireGuard rather than the in-kernel version Linux uses. This is generally good enough for my use case as I’m only looking for added security when I’m on public WiFi and don’t need really high performance.

And yes, this time I’m going to create the Ansible script either as part of the build or directly after :).

How to rename a database in MongoDB

MongoDB has a handy command to rename a collection, db.collectionName.renameCollection(). There is currently no equivalent to rename a database. Now if we accept that from time to time, one positively, absolutely just has to rename a database in MongoDB, well, there are a couple of options. Unfortunately they aren’t quite as straight forward as single MongoDB command. All methods for renaming a database in MongoDB also take a fair amount of time and/or disk space to complete. Keep this in mind when you try to use any of them.

Read More

[HOWTO] Installing Emacs 26.3 on Ubuntu or XUbuntu 19.04

My previous instructions for installing a newer Emacs version on Ubuntu still work. Ubuntu (and in my case, XUbuntu) 19.04 ships with Emacs 26.1 out of the box. As usual I want to run the latest version – Emacs 26.3 – as I run that on my other Linux, FreeBSD and macOS machines.

I only had to make one small change compared to the older instructions. Instead of running¬†the versioned sudo apt-get build-dep emacs25 I rand sudo apt-get build-dep emacs. Once the dependencies are installed, you’re a configure/make/make install away from having a working Emacs 26.3:


timo@timos-thinkpad:~/Downloads/emacs-26.3$ lsb_release -a
No LSB modules are available.
Distributor ID:	Ubuntu
Description:	Ubuntu 19.04
Release:	19.04
Codename:	disco
timo@timos-thinkpad:~/Downloads/emacs-26.3$ $HOME/local/bin/emacs --version
GNU Emacs 26.3
Copyright (C) 2019 Free Software Foundation, Inc.
GNU Emacs comes with ABSOLUTELY NO WARRANTY.
You may redistribute copies of GNU Emacs
under the terms of the GNU General Public License.
For more information about these matters, see the file named COPYING.

Some of the other instructions on the web mention that there is now a PPA for the latest stable Emacs versions. I’ve not personally used it as I’m comfortable with building Emacs from the command line. The other advantage with building Emacs from scratch is that it coexists with any other version that you installed from the Ubuntu repositories or other PPAs. This way you can avoid problems like the one described in this askubuntu Stackexchange discussion.

Installing leiningen on Manjaro Linux

I like Lispy languages. One I’ve been playing with – and occasionally been using for smaller projects – is Clojure. Clojure projects usually use Leiningen for their build system. There are generally two ways to install leiningen – just download the script as per the Leiningen web site, or use the OS package manager. I usually prefer using the OS package manager, but Manjaro doesn’t include leiningen as a package in its repositories. Installing leiningen is pretty easy via the package manager and I’ll show you how.

Read More

How to speed up macOS Time Machine backups

macOS Time Machine is usually set up to work in the background and not overly affect anything that’s going on in the foreground while the user is working. Under normal circumstances, this is desirable behaviour. It is not desirable when you try to take one last backup of a failing SSD before it keels over completely. Which was the unfortunate situation I found myself in.

Turns out there is a sysctl that can be used to disable or enable this behaviour. If you turn it off, the backup in macOS Time Machine runs much faster, at the expense of additional network bandwidth and disk IOPS. The backup daemon will increase disk IOPS usage both for reading and writing.

The sysctl to turn off the low priority backup in the background is:

sudo sysctl debug.lowpri_throttle_enabled=0

Obviously, set the value back to its default of 1 if you want to restore the original behaviour. Based on the atop stats on my home server, network bandwidth usage went up from 5-10% to about 20%, and disk IOPS usage from 7-8% to about 65-70%. The backup is not maxing out the server or client. On my old 6 core Mac Pro, I have no problem running the backup at the higher speed without a big impact to my main work. I suspect that it would be different if I were to run disk intensive applications, though.