Large file handling in Emacs revisited, with a quick look at VLF

I recently blogged about installing a 64-bit build of Emacs for Windows because I was dealing with a bunch of large and very large files.

While the 64-bit build definitely handled the really large files much better than the 32-bit build, there were still some performance issues. The main advantage of using the 64 bit build was that I could finally load a couple of files that I wasn’t able to load on the 32-bit build, but opening the files severely tested my patience.

The other problem that I didn’t think about initially was that if you happen to have desktop-save-mode turned on, Emacs will of course reload the large files every time you restart it, unless you remember to close large files before shutting down Emacs. That tends to take a considerable amount of time because either build of Emacs is rather slow when it comes loading files in the hundreds of megabytes or bigger category.

A commented in the previous post kindly suggested to use VLF for large files, which is what I’ve been doing instead. I keep forgetting to turn on the integration which should save me from having to remember to manually kick off vlf every time I want to open a large file, but other than that it just works.

Highly recommended if you need to deal with large files from time to time.

Using ELPA with pinned packages in GNU Emacs 24.4

Yes, I promise I’ll shut up about Emacs package management via ELPA any minute now.

Based on the feedback I had on my last post about using a combination of melpa and melpa-stable, I looked into using pinned packages via the package-pinned-packages variable that’s new in Emacs 24.4’s package.el. I couldn’t find any simple examples on how to use it, but a quick look at the source code and some playing around in ielm got me there. Well, after I finally upgraded to Emacs 24.4 on my main machine.

Using pinned packages via package-pinned-packages is actually pretty simple. First, you need a list of ELPA package archives to pull packages from. If you’re already using ELPA, you most likely have package-archives set up already. Second, you need to create the list of pinned packages. For me that was the majority of the work and it required a few round trips through the *Packages* buffer. It would have been smarter to uninstall all the base packages, run list-packages once and note which package should be installed from which archive instead of relying on my famously selective memory. Don’t do what I did, just do it once. Either way, I got there. Eventually.

Once you have both variables set up, you’re left with a massively simplified loop that really just takes a list of packages and calls package-install on them if they’re not already present in the system.

For those who like me learn better by looking at code but like me couldn’t find an example on how to use package-pinned-packages, here’s the updated code. It’s Emacs 24.4 only, partially for clarity and partially because I don’t need to support anything older than 24.4 in my .emacs anymore.

 
(require 'package)

(when (>= emacs-major-version 24)
  (setq package-archives '(("ELPA" . "http://tromey.com/elpa/")
                           ("gnu" . "http://elpa.gnu.org/packages/")
                           ("melpa" . "http://melpa.org/packages/"<span">)
                           ("melpa-stable" . "http://stable.melpa.org/packages/")
                           ("marmalade" . "http://marmalade-repo.org/packages/")
                           )))

;; Check if we're on Emacs 24.4 or newer, if so, use the pinned package feature
(when (boundp 'package-pinned-packages)
  (setq package-pinned-packages
                '((bm                 . "marmalade")
                  (smex               . "melpa-stable")
                  (zenburn-theme      . "melpa-stable")
                  (anti-zenburn-theme . "melpa-stable")
                  (zen-and-art-theme  . "marmalade")
                  (cider              . "melpa-stable")
                  (clojure-mode       . "melpa-stable")
                  (htmlize            . "marmalade")
                  (rainbow-delimiters . "melpa-stable")
                  ;; "unstable" package
                  (icicles            . "melpa"))))

(package-initialize t)

(defun install-required-packages (package-list)
  (when (>= emacs-major-version 24)
    (package-refresh-contents)
    (mapc (lambda (package)
            (unless (require package nil t)
              (package-install package)))
          package-list)))

(setq required-package-list '(bm icicles smex zenburn-theme zen-and-art-theme htmlize cider clojure-mode rainbow-delimiters))
 

You can see the whole code is a lot more compact even with the formatting I use and most importantly, it’s a lot more readable. Keep in mind that I stripped out all the code that made the melpa-stable/melpa combination work in Emacs 24.3; the code gets more complicated again when you’re trying to accommodate both versions. For my use, I was happy to just get everything working for Emacs 24.4 after I upgraded the installs on my various machines.

So what’s the take away here?

Package management is hard. Well, all of us who have worked on medium and large software systems know that from first hand experience. Emacs is no different. I think having package-pinned-packages available is a nice feature in my use case. I only use the above code to bootstrap my various Emacsen and individual Emacs instances usually have a few more packages installed. If I wanted to pin all the packages I use I’d probably be grumpy by the end of the exercise but for the basic set of packages I use, this works better than my previous attempts.

Set up Emacs to use both melpa and melpa-stable

I’ve blogged about a little elisp snippet I use to install my preferred base set of Emacs packages before. Thanks for all the feedback, it definitely helped improve the code.

One issue that kept annoying me is that there is no simple way to tell ELPA to mainly pull packages from melpa-stable and only fall back to melpa for those packages I can’t get on melpa-stable yet. I decided to extend my code to handle that situation with some manual inputs as I know which packages can’t be found on melpa-stable. It proved surprisingly easy to do so after mulling over the problem a little.

First, I updated my function install-required-packages so that it accepts an optional parameter containing a list of packages repositories. When the parameter is non-nil, I make a temporary copy of the existing packages-archives list to preserve my default settings and replace it with the list that’s been passed in. Then the function checks and install the packages as before and then restores the original package-archives variable. The code now looks like this:

 
(defun install-required-packages (package-list &optional package-archive-list)
  (when (>= emacs-major-version 24)
    (if package-archive-list
        (setq temp-package-archives package-archives
              package-archives package-archive-list))
    (package-refresh-contents)

    (mapc (lambda (package)
            (unless (require package nil t)
              (package-install package)))
          package-list)
    (if package-archive-list
        (setq package-archives temp-package-archives))))
 

As you can see, the function is now a just little more complicated thanks to the additional state preservation code. The big bonus  is that it now lets me specify which packages I don’t want to pull from my list of default repositories. To make things easier I also pre-populated the lists of my preferred ELPA repos:

 (setq stable-package-archives '(("ELPA" . "http://tromey.com/elpa/")
                                ("gnu" . "http://elpa.gnu.org/packages/")
                                ("melpa-stable" . "http://stable.melpa.org/packages/")
                                ("marmalade" . "http://marmalade-repo.org/packages/"))
      unstable-package-archives '(("melpa" . "http://melpa.org/packages/")))
 

Now I can simply tell ELPA which packages should be pulled from the default repositories and which ones come from the special repositories:

 
(install-required-packages '(smex zenburn-theme zen-and-art-theme htmlize cider clojure-mode rainbow-delimiters))
(install-required-packages '(bm icicles) unstable-package-archives)
 

Quite simple, isn’t it?

Obviously this is still work in progress. The whole approach feels clunky to me and that suggests there is room for improvement. Yes, it works and there is a lot to be said for that, but ideally I would like to build it out in such a way that I specify with package to pull from which repository and add functionality to semi-automatically update the packages as well. I don’t like the idea of fully automated upgrades – especially not from melpa – as I’ve ended up with broken packages before when I took that approach, but a manually triggered auto-update.

Isn’t it great that we spend hours customizing our tools to save five minutes?

The official GNU Emacs 24.4 build is available for Windows

Looks like the Windows build of Emacs 24.4 has been released to http://ftp.gnu.org/gnu/emacs/windows/ on November 15th. As usual, I appear to be a few days behind the times.

Time to upgrade and see how it compares to the unofficial 64 bit builds I’ve been using recently.

Top new feature in 24.4 for me so far is the new rectangle-mark-mode. I’m doing a bunch of code conversions right now – basically transferring a bunch of manual assignments into a big lookup table – and it’s seeing a lot of use because of it.

Finally trying out a 64 bit Emacs on Windows

I’ve been using the official GNU distribution of Emacs for Windows for the last few years and am very happy with it. Well, usually I am very happy with it until someone sends me a 25GB log file I need to analyse and the 32 bit Emacs refuses to play when faced with the enormity of the file in question.

This happened again recently so I finally decided to go look for a 64 bit Emacs for Windows, which led me to emacs-w64:

emacs-24-64-bit

I haven’t spent a lot of time with it (yet) but first impressions are very favourable. It starts quickly, seems responsive and, well, edits texts. In other words, I like it so far and I’ll keep using it for a while to see if first impressions hold up.

Oh, and there’s also the side benefit of being emacs-w64 already having a 24.4 build available. At the time of writing, the official GNU Windows build still appears to be 24.3.

MELPA has a new URL

MELPA has recently got its own domain (melpa.org) so it’s time to update your list of package repositories with the new URL.

Speaking of MELPA, I recently switched to their stable repository instead of their “regular” nightly build/snapshot repository after I accidentally ended up with a cider build that didn’t want to playing ball. This is not a complaint – if you use nightly builds etc you know what you’re getting yourself into – but it prompted me to switch over to using the stable package repository instead on those machines that I consider production machines. This of course require me to uninstall and reinstall a bunch of packages but that only took a few minutes.

Anyway, got update your ELPA/package.el configuration settings. And if you don’t use ELPA, spend the ten minutes or so to switch from manual package management to ELPA. Of course, if you are using a different package manager, feel free to ignore all my ramblings and keep using that one instead.

Install your basic Emacs packages via a single function call

If you, like me tend to carry around or “cloud around” a single .emacs file so you end up with similar environments wherever you have an Emacs install, you know it’s a little painful to ensure that you have the same set of basic packages installed on each one of your Emacs installations. As I had mentioned before I don’t use that many third party packages so my Emacs configurations aren’t that complicated, but I always prefer to have the computer remember things so I don’t have to.

As I started using ELPA last year, I decided to investigate if I could use ELPA and a little custom code to download and install the packages that I know I want everywhere. One evening I got bored enough to write a function that tests for the presence of a bunch of basic packages that I want to have on every Emacs install. It’s not trying to be fancy and automate things to the n-th degree as I didn’t want to spend the time each time I start Emacs, so I just have to remember to invoke the function after a new install. Here is the function, in all its simple glory:

(defun install-required-packages ()
  (if (>= emacs-major-version 24)
      (progn
        (setq package-archives '(("ELPA" . "http://tromey.com/elpa/")
                                 ("gnu" . "http://elpa.gnu.org/packages/")
                                 ("marmalade" . "http://marmalade-repo.org/packages/")
                                 ("melpa" . "http://melpa.milkbox.net/packages/")
                                 ))
        (package-refresh-contents)

        (when (not (require 'bm nil t))
          (package-install 'bm))
        (when (not (require 'icicles nil t))
          (package-install 'icicles))
        (when (not (require 'smex nil t))
          (package-install 'smex))
        (when (not (require 'zenburn-theme nil t))
          (package-install 'zenburn-theme))
        )))

Unique buffer names in Emacs

A common annoyance with Emacs when working on a code base that has duplicate file names is that the mode line tends to display the buffer names as “one.py:<1>”, “one.py:<2>” etc etc. That doesn’t help much with telling them apart and I find it confusing.

I was introduced to the Uniquify library a while ago. This library allows you some control over how you want to display the buffer names of buffers that contain files with duplicate names. I use the following configuration in my .emacs:

(require 'uniquify)
(setq uniquify-buffer-name-style 'post-forward uniquify-separator ":")

The above code will display duplicate buffers as “one.py:left” and “one.py:right”, with “left” and “right” being the directories that contain the file. The screenshot below shows Emacs with two files with the same name opened from two different locations:

Screen Shot 2014-08-23 at 3.49.46 PM