TFC - non-hierarchical testing

Simple testing

The following table shows the input and output for several different test suites. The tests are run against some of the string handling routines in Kevin Rosenberg's KMRCL. He uses RT for his tests so the following doesn't speak to the putative strengths of more complex tools like FiveAM or LIFT. It does, however, show what it is like to define simple, non-hierarchical tests and what sort of output each tool displays. Other reports can be found on the Test Frameworks page.

FiveAM
Notes: none.
Input Output 
(test test-strings
  "Test some KMRCL string tests"
  (is (string= (kl:substitute-chars-strings "" nil) ""))
  (is (string= (kl:substitute-chars-strings "abcd" nil) "abcd"))
  (is (string= (kl:substitute-chars-strings "abcd" nil) "abcde")))

(run! 'test-strings)
                
..f
Did 3 checks.
   Pass: 2 (66%)
   Skip: 0 ( 0%)
   Fail: 1 (33%)

Failure Details:
TEST-STRINGS [Test some KMRCL string tests]: 
   "abcde" was not STRING= to "abcd".
LIFT
Notes: by default, LIFT is interactive so we see the results of each test as it runs and then see the results again when we run all the tests.
Input Output 
(deftestsuite test-strings () ())
(addtest :str.0
  (ensure-same (substitute-chars-strings "" nil) ""))
(addtest :str.1
  (ensure-same (substitute-chars-strings "abcd" nil) "abcd"))
(addtest :str.2 
  (ensure-same (substitute-chars-strings "abcd" nil) "abcde"))

(run-tests)
        
#< TEST-STRINGS: no tests defined>
#< Test passed>
#< Test passed>
#< Test failed
Failure: test-stringsx : str.2
  Condition: Ensure-same: "abcd" is not EQUAL to "abcde"
  >
#< Results for TEST-STRINGS 3 Tests, 1 Failure.
Failure: test-stringsx : str.2
  Condition: Ensure-same: "abcd" is not EQUAL to "abcde"
  >
RT
Notes: by default, RT is non-interactive. This can be changed using *do-tests-when-defined*.
Input Output 
(rem-all-tests)

(deftest :str.0 (substitute-chars-strings "" nil) "")
(deftest :str.1 (substitute-chars-strings "abcd" nil) "abcd")
(deftest :str.2 (substitute-chars-strings "abcd" nil) "abcde")

(do-tests)
		
NIL
:STR.0
:STR.1
:STR.2
? 
Doing 3 pending tests of 3 tests total.
 :STR.0 :STR.1
Test :STR.2 failed
Form: (SUBSTITUTE-CHARS-STRINGS "abcd" NIL)
Expected value: "abcde"
Actual value: "abcd".
1 out of 3 total tests failed: :STR.2.
NIL
?
XLUnit
Notes: XLUnit provides timing information (but not consing).
Input Output 
(defclass string-test-case (test-case)
  ())
(def-test-method :str.0 ((test string-test-case) :run nil)
  (assert-true (string-equal (substitute-chars-strings "" nil) "")))
(def-test-method  :str.1 ((test string-test-case) :run nil)
  (assert-true (string-equal (substitute-chars-strings "abcd" nil) "abcd")))
(def-test-method  :str.1 ((test string-test-case) :run nil)
  (assert-true (string-equal (substitute-chars-strings "abcd" nil) "abcde")))

(textui-test-run (get-suite string-test-case))        
#
NIL
NIL
NIL
..F
Time: 0.001


There was 1 failure:
1) STR.1: Assert true: (STRING-EQUAL (SUBSTITUTE-CHARS-STRINGS "abcd" NIL)
                                     "abcde")


FAILURES!!!
Run: 2   Failures: 1   Errors: 0
#
?
CLUnit
Notes: none.
Input Output 
(deftest "str.0" 
    :test-fn (lambda ()
               (string= "" (substitute-chars-strings "" nil))))
(deftest "str.1"
    :test-fn (lambda ()
               (string= "abcd" (substitute-chars-strings "abcd" nil))))
(deftest "str.3"
    :test-fn (lambda ()
               (string= "abcd" (substitute-chars-strings "abcde" nil))))
(run-all-tests)

Output did not match expected output in test "str.3"
3 tests run; 2 tests passed; 1 test failed.
NIL
1
2
lisp-unit
Notes: none.
Input Output 
(define-test test-strings
  (assert-equal "" (kl:substitute-chars-strings "" nil))
  (assert-equal "abcd" (kl:substitute-chars-strings "abcd" nil))
  (assert-equal "abcde" (kl:substitute-chars-strings "abcd" nil)))

(run-tests test-strings)
 
TEST-STRINGS: (KMRCL:SUBSTITUTE-CHARS-STRINGS "abcd" NIL) failed: Expected "abcde" but saw "abcd"
TEST-STRINGS: 2 assertions passed, 1 failed.
msl-test
Notes: none.
Input Output 
(deftest str.0 (test-strings)
  (ensure (kl:substitute-chars-strings "" nil) => ""))
(deftest str.1 (test-strings)
  (ensure (kl:substitute-chars-strings "abcd" nil) => "abcd"))
(deftest str.2 (test-strings)
  (ensure (kl:substitute-chars-strings "abcde" nil) => "abcd"))
(run-tests 'test-strings)

** PASS: STR.0                     [00:00:00.00]
** PASS: STR.1                     [00:00:00.00]
Test STR.2 failed
Form: (KMRCL:SUBSTITUTE-CHARS-STRINGS "abcde" NIL)
Expected value: "abcd"
Actual value: "abcde".
** FAIL: STR.2                     [00:00:00.00]

Ran 3 of 3 tests in group TEST-STRINGS
The following tests failed:
  (STR.2)

Totals -- Passed: 2       67%
          Failed: 1       33%
unit-test
Notes: none.
Input Output 
(deftest :test-string "str.0"
  (test-assert (string= "" (kl:substitute-chars-strings "" nil))))
(deftest :test-string "str.1"
  (test-assert (string= "abcd" (kl:substitute-chars-strings "abcd" nil))))
(deftest :test-string "str.2"
  (test-assert (string= "abcde" (kl:substitute-chars-strings "abcd" nil))))

(run-all-tests :unit :test-string)

Running "str.0" ...
=> 1 passed.
Running "str.1" ...
=> 1 passed.
Running "str.2" ...
assert: (STRING= abcde (SUBSTITUTE-CHARS-STRINGS abcd NIL))                     FAILS.
=> 1 failed.

====================================
3 tests total.
   2 passed.
   1 failed.

Categories

Gardeners Reports

This page is linked from:

Test Frameworks
Other pages sharing this page's categories:

Test Frameworks XML Parser Libraries