Description

A high-level guide to the world of Common Lisp testing frameworks and tools.

Inspector

Gary King

Library Capabilities Overview

Here is a list of tools mentioned on the CLiki.

General Snapshot

Library	License	SUnit Based?	ASDF Installable?	Maintained	Last Updated	Review	Notes
JUnit style
CLUnit (CLiki)	LGPL	Y	N	N?	2002-11-27
FiveAM (CLiki)	BSD	N?	Y	Y	2004-11-15		Simple to use. Brief syntax. Reference documentation is complete but could use a tutorial. Can override the debugger, but will not report this as a failed test. --Stuart Sierra
FReT (CLiki)	BSD	Y	N	N	2004-07-09		Inspired by LIFT
LIFT (CLiki)	MIT	Y	Y	Y	2005-12-17
LispUnit (CLiki)	MIT	Y	N	Y	2006-04-07		Simple to use (just load one file). Thorough documentation. Tests return values, macro expansion, printed output, and errors. Supports test-first development. Supports redefining functions and macros without reloading tests. No setup/teardown facilities. Portable. --Stuart Sierra (Easy enough to write your own two line .asd file if you want to use asdf --Michael Price)
unit-test (CLiki)	?	Y	Y	?	2004-06-23
xlunit (CLiki)		Y	Y	?	2003-09-07		Based on JUnit, XPTest, and clos-unit.
RT / tester style
ptester (CLiki)	LGPL	N?	Y	Y	2003-09-07		"portable version of Franz's tester"
rt (CLiki)	MIT?	N	Y	Y	2003-09-07		An old standard? See also COVER test coverage tool
tester (CLiki)	LLGPL	N	N	Y	2002-09-17		See ptester too
Other style or style unknown
clickcheck (CLiki)	MIT	N?	? (has ASDF files...)	Y?	2005-09-13		"Automated random specification testing" [...] "within its own testing framework, because the existing Lisp testing libraries weren't quite to my taste."
EnclineTest (CLiki)	BSD	?	N	?	2004-03-31
MSL-Test (CLiki)	MIT	N?	N	?	2003-11-05

Probably deprecated

Library	License	SUnit Based?	ASDF Installable?	Maintained	Last Package Update	Last Review	Notes
clos-unit (CLiki)	"Retain copyright notice"	Y	N	N	2003-06-17		Possibly deprecated in favor of xlunit (but who said this originally?) On Feb 1 2005, "bsder@allcaps.org", aka Andrew P. Lentvorski, Jr., did; check the CLiki recent changes page (page 83, as of 1/18/06). xlunit does say it's based on JUnit, XPTest, and clos-unit, and it seems a bit more complete, polished, and recent. I tend to agree with Andrew. -- Larry Clapp
XPTest (CLiki)	DFSG	Y	N	N			Download link is dead. Probably deprecated in favor of xlunit.

Test Tool Features

What shall we measure? Here are some ideas:

• Interactive versus batch.
• Supports hierarchical test organization (cf. JUnit)
• Supports setup and teardown (cf. JUnit)
• Verbosity (how hard to write a test, how hard to run a test, etc.)
• Lisp integration (in a touchy feely way [smile])
• Special support for errors, warnings, and conditions
• Supports test-first development
• Need to reload tests after changes (especially with respect to macros)?
• Automatic result reporting--tests passed or failed, no need examine output.
• Number of external dependencies. A requirement to install and load a large number of external libraries might be a deterrent.
• Not too verbose --- try to avoid the Java-style "it takes a village full of objects to do the most trivial imaginable task."
• Installed and tested? On what implementations and platforms?
• Ease of determining which test-cases failed.
• How can result reporting be customized?

Real World Test Framework Comparisons (TFC)

TFC - non-hierarchical testing

How do various tools feel when defining simple tests with no setup or tear-down and with no defined hierarchy?

TFC - Hierarchical testing?

TFC - Random testing?

A note on Unit Tests, Macros, and SBCL

Add your wish list items here...

Itemized Library Details

Coming soon!

Tasks

• Complete non-hierarchical testing

  • CLUnit, LispUnit, MSL-Test, [p]tester, unit-test
  • I’m leaving out FReT because it doesn’t seem supported

• Find something where hierarchical testing makes sense (cl-containers?)

  • Write tests in all packages
  • Write up a report

• Find something where random testing makes sense

  • Maybe the original
  • Write tests in all packages that can do this (clickcheck, FiveAM, ?)
  • Write up a report

• Create list of the union of all the features we like and describe each tools support of these features

• Investigate other languages tools (JUnit, etc.)

  • Are we missing any good ideas?

Categories

Gardeners Reports