The challenges of preserving digital content

A problem archivists have been bringing up for a while now is that with the majority of content going digital and the pace of change in storage mechanisms and formats, it’s becoming harder to preserve content even when it is not what would be considered old by the standards of other historic documents created by humanity.

Case in point – the efforts required to preserve even recent movies as described in this article on IEEE Spectrum. As the article mentions, we’ve already lost access to 90% of US movies made during the silent area and about 50% of movies made before 1950. I suspect that the numbers for the European film industry might be even worse thanks to World War 2. However, keep in mind that those are numbers for movies stored on a more durable medium (and yes, I know that the early nitrate film is about as flammable as they come).

One of the moist poignant quotes, regarding Pixar’s challenge when trying to re-render Finding Nemo in 3D nine years after it’s initial release:

The fact that the studio had lost access to its own film after less than a decade is a sobering commentary on the challenges of archiving computer-generated work.

Even consumer households will face the same issue sooner or later when it comes to preserving family photos, home movies and other digital content. Just looking aroud my office, there are terrabytes of data floating around, a fair number of photographs and video. And of course this blog doesn’t have a dead tree version either, nor have I ever had an article published in a software development magazine. Not to mention that the ones I would have like to publish in (C/C++ User’s Journal and Dr Dobbs) are now dead and gone as well. Their paper products are hopefully still being preserved, but how long are we able to read their digital archives?

If we had infinite space to store physical objects we could try to preserve the computers the content is stored on, but that’s not realistically possible either.

For me personally, I am trying to make sure that the relatively recent content migrates with me when I update my computers and remains accessible. So far I’ve been lucky that my approximately 10 years of digital photos are all still usable. Even with that precaution in place I will try to shoot more film again, but only after checking that my decade-old Minolta film scanner is still working. Fortunately I also own a flatbed scanner that can scan film up to 5″x4″ and as long as it remains functional and software like Hamrick’s Vuescan supports it, I should be OK with a dual analog/digital strategy. Plus places like Freestyle photo seem to be able to supply more film that’s been reissued these days compared to a few years ago.

Installing a Java 8 JDK on OS X using Homebrew

I’ve had a ‘manual’ install of JDK 8 on my Mac for quite a while, mainly to run Clojure. It was the typical “download from the Oracle website, then manually run the installer” deployment. As I move the management of more development tools from manual management over to homebrew, I decided to use homebrew to manage my Java installation also. It’s just so much easier to get updates and update information all in one place. Oh, and installs the same JDK anyway, just without all the additional pointy clicky work.

Removing the existing installation

Fortunately Oracle has uninstall operations on their website. It’s a rather manual approach but at least it is documented and the whole procedure consists of three commands. Unfortunately in my case this didn’t end up uninstalling an older version of the JDK. For some reason, I had ended up with both 1.8.0_60 and 1.8.0_131 installed on my machine, and Oracle’s uninstall instructions didn’t touch the 1.8.0_60 install in /System/Library/Frameworks/JavaVM.framework. I suspect this is an older JDK brought over from the Yosemite install and the consensus on the Internet I could find suggest to leave that alone as the system needs those.

Apparently in older versions of OS X is was possible to run /usr/libexec/java_home -uninstall to get rid of a Java install, but that option does not appear to work in OS X Sierra anymore

Installing Java using Homebrew

The installation via homebrew is about as simple as expected. I have cask installed already, so for me it’s a simple matter of running

brew cask install java

and it will install the latest Oracle JDK. You can use

brew cask info java

to verify which version it will install.

If you haven’t got homebrew installed, follow the installation instructions on and also make sure that you install cask:

brew tap caskroom/cask

After re-installing the JDK using homebrew, java_home also finally reports the correct version:

odie-pro:~ timo$ /usr/libexec/java_home
odie-pro:~ timo$

Manjaro Linux and AMD RX 470, revisited

I’ve blogged about getting Manjaro Linux to work with my AMD RX 470 before. The method described in that post got my AMD RX 470 graphics card working with the default 4.4 kernel. This worked fine – with the usual caveats regarding VESA software rendering – until I tried to upgrade to newer versions of the kernel.

My understanding is that the 4.4 kernel series doesn’t include drivers for the relatively recent AMD RX 470 GPUs, whereas later kernel series (4.8 and 4.9 specifically) do. Unfortunately trying to boot into a 4.9 kernel resulted in the X server locking up so well that even the usual Alt-Fx didn’t get me to a console to fix the problem.

Read More

Fixing package download performance problems in Manjaro Linux

My adventures with Manjaro Linux continue and I’ve even moved my “craptop” – a somewhat ancient Lenovo X240 that I use as a semi-disposable travel laptop – from XUbuntu to Manjaro Linux. But that’s a subject for another blog post. Today, I wanted to write about package download performance issues I started encountering on my desktop recently and how I managed to fix them.

I was trying to install terminator this morning and kept getting errors from Pamac that the downloads timed out. Looking at the detailed output, I noticed it was trying to download the packages from a server in South Africa, which isn’t exactly in my neighbourhood. Pamac doesn’t appear to have an obvious way to update the mirror list like the Ubuntu flavours do, but a quick dive into the command line helped me fix the issue.

First, update and rank the mirror list:

sudo pacman-mirrors -g

Inspecting /etc/pacman.d/mirrorlist after running the above commands showed that the top ranked mirrors now are listed in the US with very short response times.

After updating the mirrors, it’s time to re-sync the package databases:

sudo pacman -Syy

Now with the new, faster mirrors ranked accordingly, I suddenly get my Manjaro packages and package updates at the full speed of my Internet connection rather than having hit and miss updates that take forever to install.

Uncle Bob Martin discovered Clojure

Uncle Bob Martin discovered Clojure fairly recently and really, really likes it. Having had the privilege to see him speak at various SD West conferences back when they still were a thing, I wasn’t surprised by this. Anyway, do yourself a favour and spend a few minutes reading the article. It’s worth your time.

I also strongly agree with him that reading Structure and Interpretation of Computer Programs is very much worth the effort. It is a very well written book, especially as computer science textbooks go. It’s not going to land you a new job with the latest hotness in technology, but it will help you improve your knowledge of computer science fundamentals. Those fundamentals are allow you to build a long term career and going from an “XX programmer” to a well rounded software engineer.


How to enable telnet in Windows 10

Turns out it’s not only Windows 8 that has its telnet client disabled, Windows 10 is in the same boat. I’ve been using Windows 10 for quite a while now and just discovered this issue. Anyway, the way to enable is as follows:

  • Right click on the start button
  • Select “Programs and Features”
  • “Turn Windows features On or Off”
  • Select ‘Telnet client’ in the dialog box that appears, like here:Job done.

Lisp like filtered container views in C++

Lisp dialects like Clojure have a very rich set of algorithms that can present altered views on containers without modifying data in the underlying container. This is very important in functional languages as data is immutable and returning copies of containers is costly despite the containers being optimised for copy-on-write. Having these algorithms available prevents unnecessary data copies. While I am not going into mutating algorithms in this post, the tradition of non-modifying alghorithms that work on containers leads to an expressiveness that I often miss in multi-paradigm languages like C++. As an example I will show you how to use a filtered container view in C++ like you would in Clojure.

Read More

Cherry pick and merge revisions in Mercurial

I’ve mentioned before that I prefer Mercurial to Git, at least for my own work. That said, git has a nice feature that allows you to cherry pick revisions to merge between branches. That’s extremely useful if you want to move a single change between branches and not do a full branch merge. Turns out mercurial has that ability, too, but it goes by a slightly different name.

There are actually two options in Mercurial – the older transplant extension and from Mercurial 2.0 onwards, the built-in graft command. I prefer to use the graft command, mainly because it is built into base mercurial and thus is available everywhere as long as one is running Mercurial 2.0 or up. Given that the current release is 4.0.1 as of the time of writing, you should really run something newer than 2.0. Also, graft uses mercurial’s merge abilities to cherry pick the change, you have a somewhat better chance of the change applying cleanly. Transplant uses the patch mechanism with works reasonably well, but in my opinion the merge system works better especially if you’re dealing with something that turns into a three way merge.

Usage is pretty simple – switch to the branch that you want to graft a change onto (the destination branch) and then graft away using the revision number of the change you want to use. Say, you want to graft a change 9534 to the release_356 branch:

hg co release_356
hg graft 9534

Note that hg graft does a merge and commit of the specific revision in one step as long as you don’t encounter a merge conflict. If you do encounter a merge conflict you’ll have to resolve it like you would resolve any other merge conflict, followed by a manual commit.

hg graft has additional functionality over and above simple cherry picking of one revision. For example, you can graft a range of revisions onto another branch. Have look at the documentation for hg graft for more information.

Mutt regex pattern modifiers

I still use the mutt email client when I’m remoted into some of my FreeBSD servers. It might not be the most eye pleasing email client ever, but it’s powerful, lightweight and fast.

Mutt has a very powerful feature that allows you to tag messages via regular expressions. It has a couple of special pattern modifiers that allow you to apply the regex to certain mail headers only. I can never remember so I’m starting a list of the ones I tend to use most in the hope that I’ll either remember them eventually or can refer back to this post. The full documentation can be found here, so this is only a cheat sheet that reflects my personal usage of the mutt regex pattern modifiers.

~f – match from
~t – match to
~c – match cc
~C – match to or cc
~e – match sender