TLS issues with Emacs and Git for Windows

I’ve recently blogged about adding TLS support to Emacs 24.5 on Windows and improving git performance on Windows by installing an alternative git command line client. The reason I ended up investigating how to add SSL and TLS support to Emacs is that when I originally upgraded from the official git Windows client to the Git for Windows build, I ended up with non-working TLS support in Emacs.

The TLS issues only occur if you tell the git installer to add git and all supporting Unix utilities to the path, which is not the default setting for a git installation on Windows.

However if you do install git with this setting, Emacs ends up finding the openssl libraries on the path. This works fine with the older versions packaged with the installer from git-scm.com but results in TLS issues with SSL/TLS connections failing with the newer binaries shipped with Git for Windows. I didn’t dig deeper into why I was seeing the issue as I was in a hurry to get Emacs working again, so I looked for the quickest way to resolve it.

tl;dr – if you’re running into TLS issues with Emacs on Windows and just installed Git for Windows, follow the instructions in Adding TLS Support to Emacs 24.5 on Windows to get SSL/TLS connection from inside Emacs working again.

Improve git performance on Windows without patching your git install

I’ve blogged about improving the performance of Git on Windows in the past and rightly labelled the suggested solution as a bad hack because it requires you to manually replace binaries that are part of the installation. For people who tend to use DVCSs from the command line, manually replacing binaries is unlikely to be a big deal but it’s clunky and should really be a wakeup call for some people to include a newer base system.

By now there is a much easier way to get the same performance improvement and this is to use Git for Windows instead of the default Windows git client from git-scm.com. Not only does the Git for Windows installer include the newer openssl and openssh binaries that I suggested dropping into the git installation directory in my original post, it is also a much newer version of git.

For me, installing the Git for Windows client kills a couple of birds with one stone.

First, it addresses a large part of my complaint that Windows is a second class citizen to the Git developers. Using git on Windows is still a tad clunkier than using it in its native environment (ie, the Unix world) but a dedicated project to improve the official command line client goes a long way to address this issue. Plus, the client is much more up to date compared to the official client from git-scm.com.

Second, addressing the performance issues that the official client has is a big deal, at least to those of us who need to work with git repositories in the multi-gigabyte size class. With repositories of that size, it does make a difference if your clone performance suddenly is an order of magnitude faster. In my case it also finally allows me to use these large git repositories with Mercurial’s hg-git plugin, which simply was not possible before.

I’ve not tried to verify if the newer openssh and openssl binaries address the issue I described in Making git work better on Windows. My assumption is that it’s not the case as I saw the same behaviour with the manually updated binaries. For use with a CI system like Jenkins I still recommend to use http access to the repository.

Git Logo

Making git work better on Windows

In a previous blog post I explained how you can substantially improve the performance of git on Windows updating the underlying SSH implementation. This performance improvement is very worthwhile in a standard Unix-style git setup where access to the git repository is done using ssh as the transport layer. For a regular development workstation, this update works fine as long as you keep remembering that you need to check and possibly update the ssh binaries after every git update.

I’ve since run into a couple of other issues that are connected to using OpenSSH on Windows, especially in the context of a Jenkins CI system.

Accessing multiple git repositories via OpenSSH can cause problems on Windows

I’ve seen this a lot on a Jenkins system I administer.

When Jenkins is executing a longer-running git operation like a clone or large update, it can also check for updates on another project. During the check, you’ll suddenly see an “unrecognised host” message pop up on the console you’re running Jenkins from and it’s asking you to confirm the host fingerprint/key for the git server it uses all the time. What’s happening behind the scenes is that the first ssh process is locking .ssh/known_hosts and the second ssh process suddenly can’t check the host key due to the lock.

This problem occurs if you’re using OpenSSH on Windows to access your git server. PuTTY/Pageant is the recommended setup but I personally prefer using OpenSSH because if it is working, it’s seamless the same way it works on a Unix machine. OK, the real reason is that I tend to forget to start pageant and load its keys but we don’t need to talk about that here.

One workaround that is being suggested for this issue is to turn off the key check and make /dev/null “storage” for known_hosts. I don’t personally like that approach much as it feels wrong to me – why add security by insisting on using ssh as a transport and then turn off said security, which results in a somewhat performance challenged git on Windows with not much in the way of security?

Another workaround improves performance, gets rid of the parallel access issue and isn’t much less safe.

Use http/https transport for git on Windows

Yes, I know that git is “supposed” to use ssh, but using http/https access on Windows just works better. I’m using the two interchangeably even though my general preference would be to just use https. If you have to access the server over the public Internet and it contains confidential information, I’d probably still use ssh, but I’d also question why you’re not accessing it over a VPN tunnel. But I digress.

The big advantages of using http for git on Windows is that it works better than ssh simply by virtue of not being a “foreign object” in the world of Windows. There is also the bonus that clones and large updates tend to be faster even compared to a git installation with updated OpenSSH binaries. As an aside, when I tested the OpenSSH version that is shipped with git for Windows against PuTTY/Pageant, the speeds are roughly the same so you’ll be seeing the performance improvements no matter which ssh transport you use.

As a bonus, it also gets rid of the problematic race condition that is triggered by the locking of known_hosts.

It’s not all roses though as it’ll require some additional setup on behalf of your git admin. Especially if you use a tool like gitolite for access control, the fact that you end up with two paths in and out of your repository (ssh and http) means that you essentially have to manage two types of access control as the http transport needs its own set of access control. Even with the additional setup cost, in my experience offering both access methods is worth it if you’re dealing with repositories that are a few hundred megabytes in size or even gigabytes in size. It still takes a fair amount of time to shovel an large unbundled git repo across the wire this way, but you’ll be drinking less coffee while waiting for it to finish.

Improving the performance of Git for Windows

Admittedly I’m  not the biggest fan of git – I prefer Mercurial – but we’re using it at work and it does a good job as a DVCS. However, we’re mostly a Windows shop and the out of the box performance of Git for Windows is anything but stellar when you are using ssh as the transport for git. That’s not too much bother with most of our repos but we have a couple of fairly big ones and clone performance with those matters.

I finally got fed up with the performance after noticing that cloning a large repository from the Linux git server to a FreeBSD box was over an order of magnitude faster than cloning the same repository onto a Windows machine. So I decided to start digging around for a solution.

The clone performance left a lot to be desired using either PuTTY or the bundled OpenSSH as the SSH client. I finally settled on using OpenSSH as I find it easier to deal with multiple keys. Well, it might just be easier if you’re a Unixhead.

My search led me to this discussion, which implies that the problem lies with the version of OpenSSH and OpenSSL that comes prepackaged with Git for Windows. The version is rather out of date. Now, I had come across this discussion before and as a result attempt to build my own SSH binary that included the high performance ssh patches, but even after I got those to build using cygwin, I never managed to actually get it to work with Git for Windows. Turns out I was missing a crucial detail. It looks like the Git for Windows binaries ignore the PATH variable when they look for their OpenSSH binaries and just look in their local directory. After re-reading the above discussion, it turned out that the easiest way to get Git for Windows to recognise the new ssh binaries is to simply overwrite the ones that are bundled with the Git for Windows installer.

*** Bad hack alert ***

The simple recipe to improve the Git performance on Windows when using a git+ssh server is thus:

  • Install Git for Windows and configure it to use OpenSSH
  • Install the latest MinGW system. You only need the base system and their OpenSSH binaries. The OpenSSH and OpenSSL binaries that come with this installation are much newer than the ones you get with the Git for Windows installer.
  • Copy the new SSH binaries into your git installation directory. You will need to have local administrator rights for this as the binaries reside under Program Files (or “Program Files (x86)” if you’re on a 64 bit OS). The binaries you need to copy are msys-crypto-1.0.0.dll, msys-ssl-1.0.0.dll, ssh-add.exe, ssh-agent.exe, ssh-keygen.exe, ssh-keyscan.exe and ssh.exe

After the above modifications, the clone performance on my Windows machine went from 1.5MB/s – 2.5MB/s to 10MB/s-28MB/s depending on which part of the repository it was processing. That’s obviously a major speedup for cloning, but another nice side effect is that this change will result in noticeable performance improvements for pretty much all operations that involve a remote git repository. They all feel much snappier.

How to enable (hack) git-p4 in msysgit for Windows

The default installation of msysgit (aka the official git client for Windows) is unfortunately built without python support. There are understandable reasons as to why this is, starting with “where the heck do I find the various python versions on Windows”. For me the problem was that I needed git-p4 to extract some code history out of a Perforce repository and guess what, git-p4 is written in Python. Only solution for me was that I had to find a way to make this work short of throwing Linux in a VM just to get a git import going.

It actually turned out to be fairly simple. The git-p4 that comes with the msysgit installation is a very basic placeholder that the main git executable runs via its shell. Getting the git-p4 plugin to work was a simple case of dropping the “real” git-p4.py from the Linux distribution into an appropriate location and then modifying git-p4 to run my local python with the appropriate command line. Just keep in mind that the shell used by msysgit is a unix shell so you need to make sure that the paths and parameters are in /bin/sh syntax and not in DOS batch syntax. Here’s my current hacked version of git-p4 that seems to do the job:


#!/bin/sh

c:/python27/python “c:/program files (x86)/Git/libexec/git-core/git-p4.py” $1 $2 $3 $4 $5 $6 $7 $8 $9