The Problem

Unit testing in Common Lisp is complicated by the presence of macros. For example, if you want to test the following macro:

(defmacro add (x) `(+ ,x 100))

You might define a test like this (in some imaginary testing framework, not any specific one):

(define-test test-add
   (ensure (= (add 3) 103)))

But what if the definiton of add changes:

(defmacro add (x) `(+ ,x 50))

The test should fail. On most Lisp implementations and testing frameworks, it will. But in some testing frameworks, running on SBCL, it doesn't.

What's Going On?

The problem stems from a peculiar quirk of SBCL's design: it is a compiler-only implementation. It has no interpreter at all. Expressions entered at the REPL are wrapped in a lambda which is then compiled before execution.

In normal Lisp implementations, macros get expanded at compile time or execution time. Macros in interpreted code are (usually) reevaluated each time that code is run. But in SBCL, compile time is the same as as read time. So, effectively, macros get expanded at read time.

Most Lisp testing frameworks grew out of the SUnit/JUnit/XUnit mold, in which tests are stored as anonymous functions (lambdas) or class methods. In a compiler-only implementation, those functions and methods get compiled too early. This means that when we defined our test, it got expanded to something like this:

(push (lambda ()
        (ensure (= (+ 3 100) 103)))
      *all-test-functions*)

This test will always succeed. The add symbol is not present in the compiled function, so changing its definiton has no effect.

Solutions

How do we get around this problem? The LiFT documentation (PDF) suggests testing the macroexpansion itself, like so:

(define-test test-add-expansion
  (ensure (equal (macroexpand '(add 3))
                 '(+ 3 100))))

But the same document admits that this only tests the definition of the macro, not its effects.

We could also wrap the macro in a function and test that:

(defun add-expander (x)
  (eval (macroexpand `(add ,x))))

(define-test test-add
  (ensure (= (add-expander 3) 103)))

But that's ugly and confusing.

A Better Way

The real solution is a different testing framework. We need to delay the compilation of test expressions until it is time to run them.

This is not as exotic as it sounds. As Lisp hackers are fond of saying, code is just data. All we need to do is switch from storing a function to storing the definition of that function. If our define-test macro (or whatever it's called) expands to something like this:

(push (quote (ensure (= (add 3) 103)))
      *all-tests*)

then the test is neither compiled nor evaluated but simply stored in memory as a raw S-expression. (This is almost exactly what LispUnit's define-test does.)

The Moral of the Story

If you want to test macros in SBCL, and you don't want to have to reload your tests every time you change something, use a testing framework that stores tests as quoted expressions. The frameworks I know of that do this are LispUnit, FReT, RT, and recent versions of FiveAM.

Here is a list of the frameworks I tried for this article, and whether or not they correctly support testing macros in SBCL: