Using ELPA with pinned packages in GNU Emacs 24.4

Yes, I promise I’ll shut up about Emacs package management via ELPA any minute now.

Based on the feedback I had on my last post about using a combination of melpa and melpa-stable, I looked into using pinned packages via the package-pinned-packages variable that’s new in Emacs 24.4’s package.el. I couldn’t find any simple examples on how to use it, but a quick look at the source code and some playing around in ielm got me there. Well, after I finally upgraded to Emacs 24.4 on my main machine.

Using pinned packages via package-pinned-packages is actually pretty simple. First, you need a list of ELPA package archives to pull packages from. If you’re already using ELPA, you most likely have package-archives set up already. Second, you need to create the list of pinned packages. For me that was the majority of the work and it required a few round trips through the *Packages* buffer. It would have been smarter to uninstall all the base packages, run list-packages once and note which package should be installed from which archive instead of relying on my famously selective memory. Don’t do what I did, just do it once. Either way, I got there. Eventually.

Once you have both variables set up, you’re left with a massively simplified loop that really just takes a list of packages and calls package-install on them if they’re not already present in the system.

For those who like me learn better by looking at code but like me couldn’t find an example on how to use package-pinned-packages, here’s the updated code. It’s Emacs 24.4 only, partially for clarity and partially because I don’t need to support anything older than 24.4 in my .emacs anymore.

 
(require 'package)

(when (>= emacs-major-version 24)
  (setq package-archives '(("ELPA" . "http://tromey.com/elpa/")
                           ("gnu" . "http://elpa.gnu.org/packages/")
                           ("melpa" . "http://melpa.org/packages/"<span">)
                           ("melpa-stable" . "http://stable.melpa.org/packages/")
                           ("marmalade" . "http://marmalade-repo.org/packages/")
                           )))

;; Check if we're on Emacs 24.4 or newer, if so, use the pinned package feature
(when (boundp 'package-pinned-packages)
  (setq package-pinned-packages
                '((bm                 . "marmalade")
                  (smex               . "melpa-stable")
                  (zenburn-theme      . "melpa-stable")
                  (anti-zenburn-theme . "melpa-stable")
                  (zen-and-art-theme  . "marmalade")
                  (cider              . "melpa-stable")
                  (clojure-mode       . "melpa-stable")
                  (htmlize            . "marmalade")
                  (rainbow-delimiters . "melpa-stable")
                  ;; "unstable" package
                  (icicles            . "melpa"))))

(package-initialize t)

(defun install-required-packages (package-list)
  (when (>= emacs-major-version 24)
    (package-refresh-contents)
    (mapc (lambda (package)
            (unless (require package nil t)
              (package-install package)))
          package-list)))

(setq required-package-list '(bm icicles smex zenburn-theme zen-and-art-theme htmlize cider clojure-mode rainbow-delimiters))
 

You can see the whole code is a lot more compact even with the formatting I use and most importantly, it’s a lot more readable. Keep in mind that I stripped out all the code that made the melpa-stable/melpa combination work in Emacs 24.3; the code gets more complicated again when you’re trying to accommodate both versions. For my use, I was happy to just get everything working for Emacs 24.4 after I upgraded the installs on my various machines.

So what’s the take away here?

Package management is hard. Well, all of us who have worked on medium and large software systems know that from first hand experience. Emacs is no different. I think having package-pinned-packages available is a nice feature in my use case. I only use the above code to bootstrap my various Emacsen and individual Emacs instances usually have a few more packages installed. If I wanted to pin all the packages I use I’d probably be grumpy by the end of the exercise but for the basic set of packages I use, this works better than my previous attempts.

Set up Emacs to use both melpa and melpa-stable

I’ve blogged about a little elisp snippet I use to install my preferred base set of Emacs packages before. Thanks for all the feedback, it definitely helped improve the code.

One issue that kept annoying me is that there is no simple way to tell ELPA to mainly pull packages from melpa-stable and only fall back to melpa for those packages I can’t get on melpa-stable yet. I decided to extend my code to handle that situation with some manual inputs as I know which packages can’t be found on melpa-stable. It proved surprisingly easy to do so after mulling over the problem a little.

First, I updated my function install-required-packages so that it accepts an optional parameter containing a list of packages repositories. When the parameter is non-nil, I make a temporary copy of the existing packages-archives list to preserve my default settings and replace it with the list that’s been passed in. Then the function checks and install the packages as before and then restores the original package-archives variable. The code now looks like this:

 
(defun install-required-packages (package-list &optional package-archive-list)
  (when (>= emacs-major-version 24)
    (if package-archive-list
        (setq temp-package-archives package-archives
              package-archives package-archive-list))
    (package-refresh-contents)

    (mapc (lambda (package)
            (unless (require package nil t)
              (package-install package)))
          package-list)
    (if package-archive-list
        (setq package-archives temp-package-archives))))
 

As you can see, the function is now a just little more complicated thanks to the additional state preservation code. The big bonus  is that it now lets me specify which packages I don’t want to pull from my list of default repositories. To make things easier I also pre-populated the lists of my preferred ELPA repos:

 (setq stable-package-archives '(("ELPA" . "http://tromey.com/elpa/")
                                ("gnu" . "http://elpa.gnu.org/packages/")
                                ("melpa-stable" . "http://stable.melpa.org/packages/")
                                ("marmalade" . "http://marmalade-repo.org/packages/"))
      unstable-package-archives '(("melpa" . "http://melpa.org/packages/")))
 

Now I can simply tell ELPA which packages should be pulled from the default repositories and which ones come from the special repositories:

 
(install-required-packages '(smex zenburn-theme zen-and-art-theme htmlize cider clojure-mode rainbow-delimiters))
(install-required-packages '(bm icicles) unstable-package-archives)
 

Quite simple, isn’t it?

Obviously this is still work in progress. The whole approach feels clunky to me and that suggests there is room for improvement. Yes, it works and there is a lot to be said for that, but ideally I would like to build it out in such a way that I specify with package to pull from which repository and add functionality to semi-automatically update the packages as well. I don’t like the idea of fully automated upgrades – especially not from melpa – as I’ve ended up with broken packages before when I took that approach, but a manually triggered auto-update.

Isn’t it great that we spend hours customizing our tools to save five minutes?

Polymorphism and boost::shared_ptr

Reposted from my old blog. Here’s the news from 2009…

I’m currently in the final stages of converting a library from raw pointers to boost::shared_ptr. I’m mainly doing this to ensure the correct pointer/pointee ownership rather than the outright prevention of memory leaks, but the latter is a nice side effect of the former.

One problem I ran into was that the library I’m updating and its clients make rather heavy use of polymorphism. Of course in 99% of the code that was fine as the objects were accessed through pointers to base classes, but the last 1% was causing problems because that part of the code contained lots of dynamic_cast statements. These classes unfortunately need to know the exact type they were dealing with so there was no easy way around the use of these idioms. It probably isn’t news to most of the people reading this blog that dynamic_cast and boost::shared_ptr don’t play that nicely.

The main issue is that taking a pointer that is held inside a boost::shared_ptr, dynamic_casting it down the hierarchy and then stuffing it into another boost::shared_ptr is a good way to ensure double deletion. Oops. So, if you see the following code you better get the debugger out…

 

   boost::shared_ptr<A> something(new A(blah));
   ...
   boost::shared_ptr<B> something_else(dynamic_cast<B>(something.get()));

 

So far so bad, but I couldn’t believe that something with a flaw that obvious would be in the boost libraries. And of course, there is a way around this – boost provides alternatives to the four C++ casts with similar names that work on boost::shared_ptrs. You can find these alternatives – which are really just wrappers around the C++ casts, but designed to work with the boost smart pointers – in the include file boost/pointer_cast.hpp.. If you’re using smart pointers because you need polymorphic behaviour of, say, items stored in standard C++ containers, have a look at this page right now. If you don’t have the time or inclination to check the linked document right now, the management summary is: “The next time someone tells you that you can’t use boost:.shared_ptr with dynamic_cast, point them in the direction of boost::dynamic_pointer_cast”. Using boost::dynamic_pointer_cast would change the above example to:

 

  

   boost::shared_ptr<A> something(new A(blah));
   ...
   boost::shared_ptr<B> something_else =
     boost::dynamic_pointer_cast<B>(something));

 

Problem solved.

The official GNU Emacs 24.4 build is available for Windows

Looks like the Windows build of Emacs 24.4 has been released to http://ftp.gnu.org/gnu/emacs/windows/ on November 15th. As usual, I appear to be a few days behind the times.

Time to upgrade and see how it compares to the unofficial 64 bit builds I’ve been using recently.

Top new feature in 24.4 for me so far is the new rectangle-mark-mode. I’m doing a bunch of code conversions right now – basically transferring a bunch of manual assignments into a big lookup table – and it’s seeing a lot of use because of it.

Using pantheios to log from a C++ JNI DLL

I originally published this post on my old blog in 2009. I’ve edited it a little for readability but left the contents unchanged, so it may be out of date and not reflect the current state of the pantheios library. I also haven’t been using pantheios for logging since about 2010, and have been using Boost.Log instead.

I recently had to come up with a logging solution for C++ code a JNI DLL/shared library that is providing the data translation layer between Java and underlying native C and C++ libraries. As usual, some logging was required to aid fault-finding in a production environment, if necessary. A quick survey of the state of C++ logging showed that not a lot had changed since I last looked at logging libraries. In fact, a lot of them seem to have survived unchanged for several years. I’m not sure if that is a good thing and a sign of maturity or a sign of “making do” and the low priority most projects assign to a performant logging library. Eventually I settled on pantheios as it offered several features that were crucial for this application. The major one was that pantheios it is extremely modular and will only link in the parts you really need. I consider this a major advantage over the more monolithic libraries that pull in all their functionality all the time, especially when you link them in as a static library (yes, log4cxx, I’m looking at you). Linking in the logging library as a static library was necessary to avoid conflicts with other libraries that are being used in the same process.

Initial tests in a simple command line program suggested that worked well and matched the requirements. Unfortunately I couldn’t get it to log at all inside the JNI DLL, so I ended up trawling Google’s search results for quite a while and experimented quite a lot of different settings until I ended up with a working combination.

First, pantheios initialises itself automatically if you use it inside a regular executable. For various reasons, it can’t do that inside a Windows DLL, so you have to do that explicitly. Fortunately, Matthew Wilson, the author of pantheios, had explained on a mailing list how to do this. Typically, I can’t find the post anymore so here’s the code that I’m using to initialise the library, which is more or less a verbatim copy of Matthew’s code minus a couple of lines that weren’t required:

#include <iostream>

#include <pantheios/pantheios.hpp>
#include <pantheios/inserters.hpp> 
#include <pantheios/frontends/stock.h>

const char PANTHEIOS_FE_PROCESS_IDENTITY[] = "JNITestDll.1";

BOOL APIENTRY DllMain( HMODULE hModule,
                       DWORD  dwReason,
                       LPVOID lpReserved
                                         )
{
  if (dwReason == DLL_PROCESS_ATTACH)
  {
    if (pantheios::pantheios_init() < 0)
    {
      std::cout << "Failed to initialise the Pantheios logging libraries!n" << std::endl;
      return FALSE;
    }
    else
    {
      pantheios::log_INFORMATIONAL("Logger enabled!");
    }
  }
  else if (dwReason == DLL_PROCESS_DETACH)
  {
    pantheios::log_INFORMATIONAL("Logger disabled!");
    pantheios::pantheios_uninit();
  }
  return TRUE;  // ok
}

This seemed to initialise the library correctly. I wasn’t getting any error messages to prove otherwise but unfortunately I still wasn’t getting any output either. Yes, I could see that pantheios_init() returned a value that indicated successful initialisation, the logging functions were called and the output went straight into the bit bucket somewhere.

It took me a little while to work out what happened but in the end I tracked it down to something that I filed under “JNI oddity”. Pantheios supports implicit linking for both its frontends (the part that you interact with) and its backends which are responsible for sending the output somewhere. Being the usual lazy so-and-so programmer, I had borrowed one of the implicit link files from the samples. Which should have worked OK as it was for a command line executable, but didn’t.

After some poking and prodding I realised that the issue was that by default, in this particular implicit link file pantheios would use the Windows Console logger when the code was built Windows. This didn’t work (probably because this was in a DLL and there wasn’t a console associated with it. Switching to the fprintf backend fixed this issue and I was finally seeing logging output from the JNI DLL. Here is the code for the implicit linking:

/* Pantheios Header Files */
#include <pantheios/implicit_link/core.h>
#include <pantheios/implicit_link/fe.simple.h>
#include <platformstl/platformstl.h>
#include <pantheios/implicit_link/be.fprintf.h>

#if (   defined(UNIX) || 
        defined(unix))&& 
    (   defined(_WIN32) || 
        defined(_WIN64))
# include <unixem/implicit_link.h>
#endif /* _WIN32 || _WIN64 */

All in all I’m happy with Pantheios as a logging solution. If you’re looking for a versatile C++ logger, I’d recommend you look at it.

Note from 2014: In a project that is not using or cannot use Boost, I would still look at pantheios first before looking at other libraries.

Using tortoisehg and mercurial on Windows with openssh

The default setup for the Mercurial DVCS on Windows with tortoisehg uses plink and Pageant to manage SSH keys when you are using ssh as the transport protocol for mercurial. That’s most likely the right choice for a normal Windows setup, but if you already have openssh installed and configured to talk to various servers, it’s easy to switch mercurial and tortoisehg to use openssh. It’s also very helpful if you’re forgetful like me and forget to start pageant, add new keys to it etc.

Just add or modify the ssh setting in the [ui] section of your .hgrc. In my case, I’m using very basic settings and already have the ssh executable in my path so my settings look like this:


[ui]
... settings ...
ssh = ssh -2 -C -x

In order for this to work, $HOME must be set such that openssh can find the existing private keys and configuration in $HOME/.ssh. Set the ssh parameters to your personal preference, in my case I’m requesting it to use ssh protocol v2 only (-2), disable X forwarding because I don’t need it (-x) and enable compression (-C).

A neat way of handling common C++ exceptions

Another slightly edited post from my old C++ blog. Again, you may have seen this one before.

This post is a quasi follow-up to the  “little exception shop of horrors”. As I mentioned in that post, I believe the reason for dumping all the exception handling code into a single header file was a misguided attempt at avoiding code duplication. No, I didn’t write that code, so I can only speculate as to why it was done. I inherited the project that contained this code and the reasons were lost in the mists of time. I did file it under “sound idea but bad execution”. It doesn’t fix the problem and you still have code duplication as the preprocessor will do the duplication work for you. Ah well, at least to don’t have to type the code in yourself multiple times. I couldn’t help but think that there must be a better way.

I more or less forgot about the whole thing as the code was about to be retired anyway. At some point I was talking to a colleague about it and he showed me a much nicer way of addressing this problem without the code duplication and in a much cleaner fashion. Start with a try/catch block like this:

try {
  // ... Lots of code here
}
catch (...) {
  handle_common_exceptions();
}

The handler function looks like this:

void handle_common_exceptions() {
  try {
    throw;
  }
  catch (specialised_exception const &ref) {
    // handling code
  }
  catch (another_sub_exception const &ex) {
    // ... more exception handling code ..
  }
  catch (std::bad_alloc const &ref) {
    // ... even more
  }
}

The elegant part is that you rethrow the exception that has been caught in the original try block and then handle those exceptions that your code actually can handle at this point. Normally, catch (…) is only useful if you are in a piece of code which requires you to stop any exceptions from escaping. You can’t determine what the actual exception is, so you can’t handle it appropriately. The only thing you can do is to say “oops, sorry”.

Rethrowing the exception inside the catch-all handler does restore access to the exception information so you can handle the exceptions that are appropriate at the point of the catch handler. As long as you don’t add another catch-all handler inside the handler function (note that I didn’t), those exceptions that you cannot handle
at this point propagate further up the call chain as they escape the handler function due to the rethrow.

Quite neat, don’t you think? Thanks to Alan Stokes for showing this technique to me.

The little exception shop of horrors

This post first appeared on my old C++ blog. You might have seen it before.
I think by now we can all agree that exceptions are generally a good thing in C++. They allow us to separate the error handling from the general flow of control inside a program and also enable us to handle errors at the most appropriate point. More often than not, the best point to handle errors is quite far removed from the source of the exception.

Imagine my surprise when I came across the following gem that was almost worthy of the Daily WTF:

catch (std::string ex) {
  // Do something horrible with the exception
}
 catch (const char *ex) {
   // Eh?
 }
 catch (int i) {
   // Uh oh
 }

I guess in some circles, being able to throw any sort of POD or object as an exception is considered a good thing. It’s not necessarily something I consider a good idea. For the moment we’ll also gloss over the advisability of catching exceptions by value. That doesn’t mean I’m condoning that sort of coding, especially not when you’re catching types that might trigger memory allocations when copied.

But wait, it gets better. You thought that the code snippet above was part of a larger chunk of code that I omitted, didn’t you? Well, it actually wasn’t – it was all parked in a header file. Obviously in the interests of brevity I removed the code that actually handled the exceptions while preserving the full structure of the code in the header file.

So what the heck is going on here then? It looks like in order to avoid duplication of code – always a worthy goal – the above header file got include wherever “needed”, so at least one of the project’s source files was full of code like this:

void doSomething() {
  try {
    // Lots of processing that might throw
  }
#include "catch_clauses.H"
}

Pass the Stroustrup, I need to refactor a developer or three…

Finally trying out a 64 bit Emacs on Windows

I’ve been using the official GNU distribution of Emacs for Windows for the last few years and am very happy with it. Well, usually I am very happy with it until someone sends me a 25GB log file I need to analyse and the 32 bit Emacs refuses to play when faced with the enormity of the file in question.

This happened again recently so I finally decided to go look for a 64 bit Emacs for Windows, which led me to emacs-w64:

emacs-24-64-bit

I haven’t spent a lot of time with it (yet) but first impressions are very favourable. It starts quickly, seems responsive and, well, edits texts. In other words, I like it so far and I’ll keep using it for a while to see if first impressions hold up.

Oh, and there’s also the side benefit of being emacs-w64 already having a 24.4 build available. At the time of writing, the official GNU Windows build still appears to be 24.3.