Git Logo

Making git work better on Windows

In a previous blog post I explained how you can substantially improve the performance of git on Windows updating the underlying SSH implementation. This performance improvement is very worthwhile in a standard Unix-style git setup where access to the git repository is done using ssh as the transport layer. For a regular development workstation, this update works fine as long as you keep remembering that you need to check and possibly update the ssh binaries after every git update.

I’ve since run into a couple of other issues that are connected to using OpenSSH on Windows, especially in the context of a Jenkins CI system.

Accessing multiple git repositories via OpenSSH can cause problems on Windows

I’ve seen this a lot on a Jenkins system I administer.

When Jenkins is executing a longer-running git operation like a clone or large update, it can also check for updates on another project. During the check, you’ll suddenly see an “unrecognised host” message pop up on the console you’re running Jenkins from and it’s asking you to confirm the host fingerprint/key for the git server it uses all the time. What’s happening behind the scenes is that the first ssh process is locking .ssh/known_hosts and the second ssh process suddenly can’t check the host key due to the lock.

This problem occurs if you’re using OpenSSH on Windows to access your git server. PuTTY/Pageant is the recommended setup but I personally prefer using OpenSSH because if it is working, it’s seamless the same way it works on a Unix machine. OK, the real reason is that I tend to forget to start pageant and load its keys but we don’t need to talk about that here.

One workaround that is being suggested for this issue is to turn off the key check and make /dev/null “storage” for known_hosts. I don’t personally like that approach much as it feels wrong to me – why add security by insisting on using ssh as a transport and then turn off said security, which results in a somewhat performance challenged git on Windows with not much in the way of security?

Another workaround improves performance, gets rid of the parallel access issue and isn’t much less safe.

Use http/https transport for git on Windows

Yes, I know that git is “supposed” to use ssh, but using http/https access on Windows just works better. I’m using the two interchangeably even though my general preference would be to just use https. If you have to access the server over the public Internet and it contains confidential information, I’d probably still use ssh, but I’d also question why you’re not accessing it over a VPN tunnel. But I digress.

The big advantages of using http for git on Windows is that it works better than ssh simply by virtue of not being a “foreign object” in the world of Windows. There is also the bonus that clones and large updates tend to be faster even compared to a git installation with updated OpenSSH binaries. As an aside, when I tested the OpenSSH version that is shipped with git for Windows against PuTTY/Pageant, the speeds are roughly the same so you’ll be seeing the performance improvements no matter which ssh transport you use.

As a bonus, it also gets rid of the problematic race condition that is triggered by the locking of known_hosts.

It’s not all roses though as it’ll require some additional setup on behalf of your git admin. Especially if you use a tool like gitolite for access control, the fact that you end up with two paths in and out of your repository (ssh and http) means that you essentially have to manage two types of access control as the http transport needs its own set of access control. Even with the additional setup cost, in my experience offering both access methods is worth it if you’re dealing with repositories that are a few hundred megabytes in size or even gigabytes in size. It still takes a fair amount of time to shovel an large unbundled git repo across the wire this way, but you’ll be drinking less coffee while waiting for it to finish.

Using tortoisehg and mercurial on Windows with openssh

The default setup for the Mercurial DVCS on Windows with tortoisehg uses plink and Pageant to manage SSH keys when you are using ssh as the transport protocol for mercurial. That’s most likely the right choice for a normal Windows setup, but if you already have openssh installed and configured to talk to various servers, it’s easy to switch mercurial and tortoisehg to use openssh. It’s also very helpful if you’re forgetful like me and forget to start pageant, add new keys to it etc.

Just add or modify the ssh setting in the [ui] section of your .hgrc. In my case, I’m using very basic settings and already have the ssh executable in my path so my settings look like this:


[ui]
... settings ...
ssh = ssh -2 -C -x

In order for this to work, $HOME must be set such that openssh can find the existing private keys and configuration in $HOME/.ssh. Set the ssh parameters to your personal preference, in my case I’m requesting it to use ssh protocol v2 only (-2), disable X forwarding because I don’t need it (-x) and enable compression (-C).

Make sure you’re using the right command processor when running Incredibuild

Quick hack/warning for those using an alternative command line processor like TCC and also use Xoreax’ Incredibuild for distributed builds. Incredibuild is awesome, by the way, and if you have a larger C++ project that takes a long time to build, you should use it. And no, I’m not getting paid or receive free stuff for writing that.

However, if you have to start your Visual Studio instance from the command line because you need to set some environment variables first, or because of your general awesomeness, make sure you’re starting it from a stock Windows shell. Either the standard Windows command line (cmd.exe) or PowerShell will do nicely, thank you. If you start VS from TCC and have a couple of build tasks that spawn out to the shell, Incredibuild wants to shell out into TCC to run these tasks and the shelled out task don’t seem to return control to Incredibuild again. Yes, I was too lazy to investigate further as the method described above works.

Timo’s occasional link roundup, late July edition

A couple of interesting articles about debugging. Debugging doesn’t seem to get a lot of attention when people are taught about programming, I assume you’re supposed to acquire this skill by osmosis, but it is actually one of those skills that should receive much greater attention because it’s one of those that separates highly productive developers from, well, not so productive ones.

Why I’m Productive in Clojure. I’ve long been a fan of Lisp and Lisp-like languages, even though I wasn’t originally that happy with having Lisp inflicted on me when I was at university. Because it was weird and back then I didn’t much appreciate non-mainstream languages. These days I do because that’s where you usually find better expressiveness and ideas supposedly too strange for mainstream languages. I guess that makes me a language hipster.

And while we’re on the subject of lisp-like languages – I’ve never heard of Julia, but this blog post made me wonder if it should be on my list of languages to look at.

We have a Nest thermostat and I wasn’t too keen when I heard that Google bought them. Probably have to look into securing it (aka stopping the data leakage). While I understand the trade “your data for our free service” model from an economics perspective, I do take some issue with the “we’ll sell you a very expensive device and your data still leaks out to us” model. Nests aren’t exactly cheap to begin with.

Debugging on a live system that’s shouldn’t be live. Been there done that, on a trading system…

Netflix and network neutrality, as seen from the other side. I’m an advocate of regulating ISPs (especially the larger ones) as public utilities and essentially enforcing network neutrality that way. Netflix obviously has been going on about network neutrality for a while now but the linked article does make me wonder if those supposed “pay to play” payments were actually more like payments for the server hosting. You know, like the charges that us mere mortals also have to pay if we want to stick a server into someone’s data centre.

 

I prefer ConEmu over Console2, and so should you…

OK, I admit it – I’m a dinosaur. I still use the command line a lot as I’m subscribing to the belief that I can often type faster than I can move my hand off the keyboard to the mouse, click, and move my hand back. Plus, I grew up in an era when the command line was what you got when you turned on the computer, and Windows 2.0 or GEM was a big improvement.

One of the neat features of the console emulators on both on Linux and Mac OS X was and is that you could run a set of shells in a tabbed single console window. A post on Scott Hanselman’s blog put me onto Console2. That was more like it and I pretty much immediately housed my Windows shells – either cmd.exe or PowerShell – in there. Much better, but over time the pace of development slowed and the last beta release dates from 2011. It’s not like the Beta is buggy or anything – in fact, in my experience it works very nicely indeed – but of course as a software engineer I like shiny new things.

Enter, via another post on Scott Hanselman’s blog, ConEmu – or ConEmu-Maximus5, to give it its full name. If Console2 is the VW Golf to the stock Windows’ console emulator’s 1200cc VW Bug, then ConEmu is the VW Phaeton to Console2’s VW Golf. It’s got a lot more features, it’s actively developed, it works well with Far Manager if you miss the Norton Commander days and it’s highly configurable. Of course, it also can handle transparent backgrounds, but so can Console2.

For me, it has one killer feature – recent versions detect which shells you have installed on your machine and offer you a selection via the green “new tab” button (the one that looks a bit like a French Pharmacy sign), with a choice of running them either as a regular user or admin user:

ConEmu with visible command line processor menu
ConEmu with visible command line processor menu

Why is this such a big deal? Well, it’s neat if you’re using both PowerShell and cmd.exe, but for me it’s a killer feature because I like using TCC/LE, at least at home. TCC/LE is the familiar Windows command prompt at first glance but in the same way that ConEmu is a much expanded console emulator compared to the regular Windows one, TCC/LE is a much expanded command prompt that is a lot more feature rich and has a lot of sensible extensions. And because I’m such a dinosaur, I’ve actually been using its predecessors (4DOS and 4NT) way back when they were distributed as shareware on a floppy disk and you had to buy the manuals for them to get the registration code. And yes, I still have at least the 4DOS manual.

Back to console emulators, though. If I wanted to go nitpicking, both ConEmu and Console2 work less well over an RDP connection than the stock console, which is noticeable if you tend to remote into machines quite frequently. It’s not that they work badly, but Microsoft clearly spent a lot of time optimising the stock console to work well over RDP (or to have RDP work well with the stock console), so there is a bit of lag when scrolling. It doesn’t make either tool unusable but you notice it’s there.

Anyway, if you check out one new tool this week, make it ConEmu.

The coder/programmer/software engineer debate seems to be rising from the undead again

First, a confession – I actually occasionally call myself a coder, but in a tongue in cheek, post-modern and ironic way. Heck, it does make for a good blog title and license plate.

Nevertheless, with all the recent “coding schools” cropping up all over the place – at least if you are in the Bay Area – it does seem that being able to code in the context of a reasonably sought after web technology without much further formal training is the path to new, fulfilling careers and of course untold riches in an economy where recent graduates in all fields have problems finding work. Well, at least a career that allows you to rent a room instead of crashing on somebody’s couch.

There are some problems with the whole “mere coder” thing. Dave Winer has some interesting thoughts on his blog, and I agree with a lot of what he says. Scott Radcliff has some additional thoughts, which I also find myself agreeing with a lot.

The process of building a software system is a lot more than just coding unless you subscribe to the view that all the coders do is to take the gospel handed down by A Visionary Leader and convert it into software. Anybody who’s ever built a moderately complex system knows that software doesn’t happen that way, at least not the type of software that doesn’t collapse under its own weight shortly after its release. Of course that doesn’t sit well with the notion of The Visionary Genius that is all that is required to build the next and the narrative doesn’t work at all once you recognise that building software is, in most cases, a team sport.

Expecting someone who has learned to write code to o create a great piece of software is like expecting someone who’s just gone through a foreign language course of similar length to go and write the next great novel in that particular language. Sometimes you get lucky, but most of the time the flow isn’t there, the language isn’t yet used in an idiomatic way and all that together makes for less than an enjoyable experience. Some of the people who manage to enter the profession of software development will learn on the job and grow into people who can build robust, maintainable systems and those are to be congratulated.

The way most people use the term coder, ie in a non-postmodern ironic way, reminds me very much of the time when a programmer’s job was to unthinkingly program as part of a little cog in the giant waterfall development machine, which led to the WIMP (Why Isn’t Mary Programming) acronym. Basically, if the keyboard wasn’t constantly clattering, there was no programming going on.

That said, maybe we are at a point in time where an army of coders as the modern equivalent of the typing pool is able to create good enough software. Given that most users are conditioned to believe that software is infuriating and buggy, we as an industry might well get away with. Is that the world we want to live in, though?

As creators of software, we do have the ability to choose the environment that we work and live in. If you care about quality, work with like-minded people.

Me? I prefer to call myself a software craftsman. I’m not an engineer, I write code but that’s almost an afterthought once the design is figured out but at the end of the day what I do is build something that wasn’t there before, using vague guidelines that are wrong as often as they are right while trying to tease out what the customer needs rather than what they tell me they want. In most cases there is no detailed blueprint, no perfect specification that only needs to be translated 1:1 into lines of code. Instead, I get to use my experience and judgement to fill in the blanks that most people – myself included – probably didn’t even think were there.

Sounds like a job for a craftsman to me.

Initial thoughts on Swift

Like pretty much every other programmer with a Mac, I’m currently looking at Swift. Will I write anything but toy programs in it? I don’t know yet – I don’t really write any Mac-ish software on my Mac,  just unix-ish programs. If Swift doesn’t escape the OS X and iOS ecosystems it’ll be a nice exercise in a neat language that’s not really that relevant to the world at large, or at least to my part of the world at large. Not that this sort of vendor lock-in can’t work well – Visual Basic 6, anybody?

I hope Apple will open the language to be used outside its software ecosystem but from what I have read so far, it’s tightly integrated into the iOS and Mac OS X runtimes so I’m not sure how feasible that even would be. Still, I’m pretty sure it’ll remain on my list of languages to play with.

Someone is building a BBC Micro emulator in Javascript

For those of us who remember when the BBC Micro was the home computer with the fastest Basic implementation available, a long time ago, and was pretty legendary in home computing circles in Europe. It didn’t sell that much outside of the UK, mostly because of its price. It was also the target system for the original implementation of Elite. Matt Godbolt is building an emulator in JavaScript. First post of his series can be found here.

It’s amazing how far we have come since I started playing with computers, yet they’re still not fast enough.