Building an OpenBSD WireGuard VPN server part 3 – Unbound DNS filtering

In part 2, I reconfigured my WireGuard VPN to use an Unbound DNS server on the VPN server rather than rely on a third party server I had used for the original quick and dirty configuration. It was important for me to set up a validating DNS server, which I did in that part.

In this part, I’m extending the existing configuration to include some basic block lists for known ad and tracking servers. As I’m mainly trying to use the VPN while on the road, I want to ensure that anything I end up doing using the VPN is as secure as I can make it with reasonable effort. That makes tracking and preventing malicious ads the next step. That said, I’m not planning to go for a full Pi-Hole like setup. Initially, I am trying to do is integrate one known good blocklists into the Unbound configuration and automate the process. I can get fancy with a more Pi-Hole like setup later if I want to.

Picking a DNS filter list and using it from Unbound

I started by using a single DNS block list from StevenBlack’s github repo. deadc0de.re’s blog has a good how-to post including the necessary awk incantations for converting the file into the format Unbound needs. I used that blog post as my as my starting point.

The process overall is relatively simple. First, pick the flavour of blocklist you want. Download the hosts file for the blocklist. I used wget to download it as it’s already installed on my OpenBSD system, but curl would also work. My assumption is that you’re running these commands in /var/unbound/etc, so all the absolute paths refer to that location.

wget -O /tmp/blocklist-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts

This list is in the traditional host file format, so we need to convert it into the format that Unbound needs. Fortunately, deadc0de.re’s blog post contains the necessary code to accomplish that transformation:

cat /tmp/blocklist-hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > adblocker.conf

We also need to update our unbound.conf file to include the freshly generated host blocklist. To do that, we have to add an include statement to the server section of the configuration file. The change looks like this:


server:
  ...
  include: /var/unbound/etc/block-ads.conf

I also had to set up unbound-control to ensure that I didn’t have to restart the server every time I updated the block list. Thus, I had to make a slight detour that involved running unbound-control-setup to generate the necessary certificates. I also needed to add the following statement to my unbound.conf:

remote-control:
  control-enable: yes

Note that remote-control is its own section and is not part of the server: section. The default setup only accepts remote control connections on the loopback interface. This is perfect for my setup, so I didn’t need to make any other changes.
After restarting unbound to pick up the configuration changes I was able to test the VPN with the new DNS blocking and it proved to be mostly effective on the websites I visited.

Automating the process

Of course I’m a software engineer by background, so clearly I wanted to automate the process. Fortunately this is about as complicated as pasting the commands I listed above into a shell script and periodically run it. My script looks like this:

#!/bin/sh

cd /var/unbound/etc
wget -O /tmp/black-hosts https://raw.githubusercontent.com/StevenBlack/hosts/master/hosts
cat /tmp/black-hosts | grep '^0\.0\.0\.0' | awk '{print "local-zone: \""$2"\" redirect\nlocal-data: \""$2" A 0.0.0.0\""}' > block-ads.conf
rm /tmp/black-hosts
unbound-control reload

I’m running this script via cron once a week, as that seems to roughly correspond to the update frequency of the list. I may change that in the future, especially if I end up combining the multiple block lists instead. In the meantime I don’t think it’s necessary to run the cron script more often.

What’s next?

Right now it looks like WireGuard VPN server is working as intended and does what I need it to. TuM’Fatig has a great blog post on expanding the approach I’m using to include all of the public block lists that Pi-Hole is using, so I’m probably going to look into that sooner or later. Other than that and keeping the machine up to date, the next project will be to create orchestration scripts to recreate the server from scratch. I found setting up WireGuard on OpenBSD much easier than doing it on a Linux host. Nevertheless having a Terraform or Ansible script that allowed me to quickly stand one up from scratch if the current instance was damage would likely be a good thing.

Would I go the OpenBSD route again?

I had mentioned at the start of this series that I hadn’t had done much with OpenBSD for years. Was it worth choosing OpenBSD over FreeBSD, which I use on a very regular basis? Maybe, maybe not. OpenBSD required less additional fiddling to secure it out of the box. Some of the time saved was taken up by reacquainting myself with the system, so I’d call that a wash. I suspect that setting up the actual WireGuard VPN instance would take the same amount of time on FreeBSD as I’d run the same commands, so no clear advantage there. I do like the newer version of pf that comes with OpenBSD – FreeBSD’s is a few versions behind and is missing some features, and I doubt it’ll ever catch up.

Put that down as a ‘maybe’.

How well does WireGuard work for me?

So far it is proving performant enough to run on one of Vultr’s $5 instances that aren’t exactly swimming in computing power. Of course this is only a personal VPN and I’m not running a whole company’s worth of traffic across it. That would very likely require more oomph (technical term) on the server, but overall WireGuard VPN appears to be both very lightweight and more importantly, easy to configure both at the server and client end.

Throughput is good enough to use it to watch YouTube videos in good quality on crappy hotel Wifi without visibly dropping frames. That’s good enough for me as a performance metric because it shows that the VPN server is not the bottleneck. The real use cases for the VPN really are for accessing more sensitive information like financial information, and that is usually all interactive anyway.

Building an OpenBSD WireGuard VPN server part 2 – Unbound DNS setup

In the first part, I described how I set up the basic OpenBSD WireGuard VPN server. I also hinted that I wanted to set up my own validating, filtering DNS server. With a little bit of spare time during the holidays I decided now was a good time as any.

Making sure the VPN server uses the local Unbound DNS resolver first

Before I set up Unbound itself, I need to make sure that the VPN server’s resolv.conf points at its local DNS server first.

My Vultr VPS has a static IP, which it receives via DHCP. It also receives information about its upstream DNS servers via DHCP. So as the first step, we need to update /etc/dhclient.conf to ensure that resolv.conf lists our local server first. We do this by adding the following line to /etc/dhclient.conf:

prepend domain-name-servers 127.0.0.1;

In fact, that is the complete dhclient.conf file on my VPN server.

Setting up Unbound as a validating DNS server

OpenBSD already includes the Unbound authoritative DNS server, so it’s a matter of configuring it to act as the local DNS server and also to enable validation. Everything you need to run the server is already on the system.

Root server list and trust anchor files

Once we told the local system to use the local Unbound as its DNS server, we need to configure it. Note that dhclient will still add the upstream provider DNS server to resolv.conf, so it is still there as a fallback.

Time to set up Unbound. A validating DNS server needs a few more pieces of data compared to a standard recursive DNS server. The two important items are the trust anchor and the root hints so Unbound knows where to find the root name servers. Note that OpenBSD runs Unbound in a sandbox and thus all the configuration is under /var/unbound.

We start with retrieving the root.hints file that contains the list of root DNS servers:

wget https://www.internic.net/domain/named.root -O /var/unbound/etc/root.hints

A quick check with more /var/unbound/etc/root.hints shows that we received the expected list of root servers.

The second part we need is the trust anchor, which is the cryptographic key our DNS server needs so it can validate DNSSEC. After a couple of false starts I simply ended up using unbound-anchor to download the trust anchor and configured Unbound via the auto-trust-anchor setting to automatically check and update the trust anchor when necessary.

Pretty much all of the Internet tutorials suggest that the trust anchor should be stored in /var/unbound/etc. In my OpenBSD setup, unbound can’t write to that directory so I ended up sotring it in /var/unbound/db instead.

Unbound.conf minimal setup for validating server

Now we’re finally ready to configure unbound itself. My basic, stripped down version of the configuration looks like this:


server:
  verbosity: 1
  #
  # Interfaces and access control. Make sure we're not suddenly
  # running an open public resolver
  #
  outgoing-interface: 'real' server ip'
  interface: 127.0.0.1
  interface: 192.168.1.1         # Internal VPN interface
  access-control: 127.0.0.0/8    allow
  access-control: 192.168.1.0/24 allow
  #
  # Basic configuration
  #
  port: 53
  do-ip4: yes
  do-ip6: no                     # VPN server currently doesn't support IPV6
  do-udp: yes
  do-tcp: yes
  #
  # Root servers, trust anchor and validation logging
  #
  root-hints: "/var/unbound/etc/root.hints"
  auto-trust-anchor-file: "/var/unbound/db/root.key"
  val-log-level: 2
  val-clean-additional: yes
  #
  # Some basic hardening
  #
  hide-identity: yes
  hide-version: yes
  harden-glue: yes
  harden-dnssec-stripped: yes
  use-caps-for-id: yes
  unwanted-reply-threshold: 10000
  #
  # Ensure we don't return private addresses to prevent DNS
  # rebinding attacks
  #
  private-address: 10.0.0.0/8
  private-address: 172.16.0.0/12
  private-address: 192.168.0.0/16
  #
  # Other settings. I don't expect to see a lot of DNS traffic so
  # I turned off prefetching
  #
  prefetch: no
  #
  num-threads: 1

Now that we have a working configuration file, we need to enable and start Unbound:

vpn# rcctl enable unbound
vpn# rcctl start unbound
unbound(ok)

Testing DNSSEC validation

At this point we have a working server with supposedly working DNSSEC validation. Obviously we work on ‘trust, but verify’. To check that we have indeed a working validating server, we can run the following command:

dig www.nic.cz. +dnssec

The header section of the result should look like this:

; <<>> DiG 9.4.2-P2 <<>> www.nic.cz. +dnssec
;; global options:  printcmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 18417
;; flags: qr rd ra ad; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

See the bolded ‘ad’ in the flags line? Now compare this to the output of the same command, but run on my MacBook using the ISP’s resolver:

; <<>> DiG 9.10.6 <<>> www.nic.cz. +dnssec
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 12527
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

The ISP’s resolver doesn’t support DNSSEC in this case, so you can see the ‘ad’ flag missing. That flag indicates that the result from the upstream server validated.

Update the WireGuard config on the client

After I verified that the validation is working, it is time to update the WireGuard configuration on all clients so they point at our shiny new validating server rather than the third party DNS servers I used in the first installment.

Summary

At this point I have a validating DNS server, however it still passes through requests for every domain including the typical ad slinging domains. In the next installment of this series I’ll add filtering rules to try to get rid of most, if not all of them.

Building an OpenBSD Wireguard server

In my previous post, I mentioned that I somehow ended up with a corrupted filesystem on the WireGuard server I had set up earlier this year. That iteration of my VPN server was built on Linux as I expected I would get better performance using the kernel-based WireGuard implementation. It had taken me a while to set it up right, and I didn’t get the impression that the performance was so much better anyway. Keep in mind that I mostly use my VPN server from hotel WiFi and we all know how “good” that tends to be performance wise.

While I’ve done a fair bit of Linux admin work, I didn’t fancy re-doing the whole setup again. I also hadn’t scripted it up using Ansible or similar. I tend to prefer BSD anyway, and most of my personal servers run some flavour of BSD Unix. As I didn’t want to spend too much time securing this server, I used OpenBSD as it is a little more secure out of the box compared to FreeBSD. I also hadn’t experimented with OpenBSD for a while so I was curious to see the more recent improvements.

OpenBSD WireGuard Server setup at Vultr

I re-used the VPS I already had set up for the old Linux WireGuard VPN server at Vultr. Heck, it was corrupted already so formatting it was the only choice. In the interest of getting the VPN up and running quickly again, I used Vultr’s preconfigured OpenBSD image. With hindsight I probably wouldn’t do that again. More about this later. I followed a combination of instructions, mainly from the Cryptsus blog post on setting up WireGuard on an OpenBSD 6.6 server, but also referencing Jasper.La’s blog and Ankur Kothari’s blog. Setting up WireGuard on OpenBSD took me a lot less time than configuring the Linux version. I think it took me an hour or two to get the basic VPN tunnel up and working, including configuring the macOS client.

Overall I found this approach simpler than setting up WireGuard on Linux. I think there are two reasons for this – using the precompiled binaries for the user mode processes means you don’t have to futz around with kernel modules, third party repos and all that fun stuff, plus I personally find FreeBSD and to a certain extent OpenBSD easier to set up. This is obviously a personal preference. Plus, no systemd :).

Notes on the default Vultr OpenBSD Image

The Cryptsus instruction include setting up OpenBSD using full disk encryption. This is more secure than using Vultr’s default, preconfigured VPS image as that one doesn’t support full disk encryption out of the box. Using the preconfigured image makes the installation process easier and quicker, but at the expense of not getting the full official OpenBSD configuration out of the box. With hindsight, I probably what should have used a custom (well, the official) OpenBSD 6.6 image instead of the preconfigured Vultr one, but as mentioned I was in a bit of a rush and it was a case of “good enough and working” trumping “more secure”. When setting up a VPN server, there are basically multiple issues you want to protect yourself against:

  1. Someone remotely logging into your VPN server and getting hold of the encryption keys, and/or being able to capture traffic on your VPN server. That’s obviously the worst-case scenario from a security and privacy perspective, and the main one that I wanted to protect myself against. For that, I rely on OpenBSD’s security with strong passwords and key-based SSH authentication to log into the machine.
  2. Someone getting hold of the disk image – that’s what the full disk encryption part protects against. Something I will want to add to the server sooner or later – it does require a reinstall first, though. The risk of this is considerably lower than someone breaking into the VPN server itself and requires either an infrastructure fluke or an infrastructure compromise.
  3. Someone getting a level of access to the VPS host such that they can observe the running VPS and dump out its memory. Again, that’s possible but it requires a lot of work to get there. I like to think that there are a lot juicier targets than myself out there that would warrant this level of effort. So for right now, I’m discounting this threat model.

I’ve got the first point covered right now, will probably address the second one at some point and do my best to ignore the third one for now.

What’s still left to be done

The basic setup for my OpenBSD WireGuard server is up and running, and I’ve successfully used it while traveling. It’s definitely fast enough even on one of Vultr’s $5 instances. Most of the time, when I use the VPN I don’t need lots of bandwidth, but even when “testing” by watching YouTube videos, the performance was more than good enough.

I’m currently using the AdGuard DNS servers that were mentioned in the Cryptsus blog post I linked to above, but I really want to move to an Unbound DNS server on the VPN host itself that a) validates domains were possible and b) uses something like the Pi-hole blocklists to block ads and trackers. The latter is somewhat optional as I don’t really use the VPN for normal browsing, but it’s a definite nice to have.

Looks like I get to redo my WireGuard VPN server

I’ve blogged about setting up a WireGuard VPN server earlier this year. It’s been running well since, but I needed to take care of some overdue maintenance tasks. Trying to log into the server this morning and I am greeted with “no route to host”. Eh? A quick check on my Vultr UI showed that the VPS had trouble booting. The error suggests a corrupted boot drive. Oops.

Guess what the maintenance task I was looking at was? Creating an Ansible script so I’d be able to stand up the server from scratch in case something like this happened. And yes, the irony of being the guy who regularly preaches to his clients about the need for backups doesn’t quite escape me.

Anyway, at least this gives me an excuse to set up my WireGuard server on OpenBSD. This is something I’ve been thinking about for a while so now I have the perfect excuse for it. I realise that OpenBSD can only use the user space daemon for WireGuard rather than the in-kernel version Linux uses. This is generally good enough for my use case as I’m only looking for added security when I’m on public WiFi and don’t need really high performance.

And yes, this time I’m going to create the Ansible script either as part of the build or directly after :).

Installing leiningen on Manjaro Linux

I like Lispy languages. One I’ve been playing with – and occasionally been using for smaller projects – is Clojure. Clojure projects usually use Leiningen for their build system. There are generally two ways to install leiningen – just download the script as per the Leiningen web site, or use the OS package manager. I usually prefer using the OS package manager, but Manjaro doesn’t include leiningen as a package in its repositories. Installing leiningen is pretty easy via the package manager and I’ll show you how.

Read More

How to speed up macOS Time Machine backups

macOS Time Machine is usually set up to work in the background and not overly affect anything that’s going on in the foreground while the user is working. Under normal circumstances, this is desirable behaviour. It is not desirable when you try to take one last backup of a failing SSD before it keels over completely. Which was the unfortunate situation I found myself in.

Turns out there is a sysctl that can be used to disable or enable this behaviour. If you turn it off, the backup in macOS Time Machine runs much faster, at the expense of additional network bandwidth and disk IOPS. The backup daemon will increase disk IOPS usage both for reading and writing.

The sysctl to turn off the low priority backup in the background is:

sudo sysctl debug.lowpri_throttle_enabled=0

Obviously, set the value back to its default of 1 if you want to restore the original behaviour. Based on the atop stats on my home server, network bandwidth usage went up from 5-10% to about 20%, and disk IOPS usage from 7-8% to about 65-70%. The backup is not maxing out the server or client. On my old 6 core Mac Pro, I have no problem running the backup at the higher speed without a big impact to my main work. I suspect that it would be different if I were to run disk intensive applications, though.

Setting up my own VPN server on Vultr with Centos 7 and WireGuard

As an IT consultant, I travel a lot. I mean, a lot. Part of the pleasure is having to deal with day-to-day online life on open, potentially free-for-all hotel and conference WiFi. In other words, the type of networks you really want to do your online banking, ecommerce and other potentially sensitive operations on. After seeing one too many ads for VPN services on bad late night TV I finally decided I needed to do something about it. Ideally I intended to this on the cheap and learn something in the process. I also didn’t want to spend the whole weekend trying to set it up, which is how WireGuard entered the picture. I only really needed to protect my most sensitive device – my personal travel laptop.

As I’m already a customer at Vultr (affiliate link) I decided to just spin up another of their tiny instances and set it up as my WireGuard VPN server. Note that I’m not setting up a VPN service for the whole family, all my friends and some additional people, all I’m trying to do is secure some of my online communications a little bit more.

I also decided to document this experiment, both for my own reference and in the hope that it will be useful for someone else. Readers will need to have some experience setting up and administering Linux server. Come on in and follow along!

Read More

Installing specific major Java JDK versions on macOS via Homebrew

Update 2019-11-16: I updated the syntax for the homebrew cask search as “brew cask search” is deprecated. Also, while the post below still mentions the java8 cask, only the java6 and java 11 casks are currently available.

Update 2019-05-07: The java8 cask is affected by recent licensing changes by Oracle. There’s a discussion over on github about this. I’m leaving the post up partially for historic context. The java8 cask is no longer available, at least at the time of writing.

In an earlier post, I described how to install the latest version of the Oracle Java JDK using homebrew. What hadn’t been completely obvious to me when I wrote the original blog post is that the ‘java’ cask will install the latest major version of the JDK. As a result, when I upgraded my JDK install today, I ended up with an upgrade from Java 8 to Java 9. On my personal machine that’s not a problem, but what if I wanted to stick with a specific major version¬† of Java?

Read More

Tracking down why the iOS mail app couldn’t retrieve my email over an LTE connection

We all love the odd debugging story, so I finally sat down and wrote up how I debugged a configuration issue that got in the way of the iOS mail app’s ability to retrieve email while I was on the go.

tl;dr – iOS Mail uses IPV6 to access you email server when the server supports IPV6 and doesn’t fall back to IPV4 if the IPV6 connection attempt fails. If if fails, you don’t get an error, but you don’t get any email either.

The long story of why I sporadically couldn’t access my email from the iOS 10 Mail app

Somewhere around the time of upgrading my iPhone 6 to iOS 10 or even iOS 10.2, I lost the ability to check my email using the built-in iOS Mail app over an LTE connection. I am not really able to nail down the exact point in time was because I used Spark for a little while on my phone. Spark is a very good email app and I like it a lot, but it turned out that I’m not that much of an email power user on the go. I didn’t really need Spark as Apple had added the main reason for my Spark usage to the built-in Mail app. In case you’re wondering, it’s the heuristics determining which folder you want to move an email to that allow both Spark and now Mail to suggest a usually correct destination folder when you want to move the message.

Read More

Manjaro Linux and AMD RX 470, revisited

I’ve blogged about getting Manjaro Linux to work with my AMD RX 470 before. The method described in that post got my AMD RX 470 graphics card working with the default 4.4 kernel. This worked fine – with the usual caveats regarding VESA software rendering – until I tried to upgrade to newer versions of the kernel.

My understanding is that the 4.4 kernel series doesn’t include drivers for the relatively recent AMD RX 470 GPUs, whereas later kernel series (4.8 and 4.9 specifically) do. Unfortunately trying to boot into a 4.9 kernel resulted in the X server locking up so well that even the usual Alt-Fx didn’t get me to a console to fix the problem.

Read More