Better exception output for test.check

29th September 2023

Here’s a hack for getting better output when test.check tests run via clojure.test throw exceptions, thanks to Dominic Monroe.

TL;DR

If you require the following code somewhere in your test suite, then you’ll still get shown the usual helpful test.check output - in particular, the :seed - if the code under test throws an exception:

(alter-var-root
 #'clojure.test.check.clojure-test/assert-check
 (constantly
  (fn [{:keys [result result-data] :as m}]
    (if-let [error (:clojure.test.check.properties/error result-data)]
      (clojure.test/do-report
       {:type :error
        :message (-> m
                     (dissoc :result :result-data)
                     (update :shrunk dissoc :result)
                     (update-in [:shrunk :result-data]
                                dissoc
                                :clojure.test.check.properties/error))
        :expected {:result true}
        :actual error})
      (clojure.test/is (clojure.test.check.clojure-test/check? m))))))

What problem does this fix?

If we run the following test.check test using clojure.test (via the defspec macro, which provides integration between the two)…

(require '[clojure.test.check.generators :as gen])
(require '[clojure.test.check.properties :as prop])
(require '[clojure.test.check.clojure-test :refer [defspec]])

(defn broken-identity [x]
  (if (= x 42) 0 x))

(defspec broken-identity-should-return-argument
  (prop/for-all
   [x gen/small-integer]
   (= x (broken-identity x))))

then we get the following useful output from test.check:

{:fail [42]
 :failed-after-ms 5
 :failing-size 85
 :num-tests 86
 :pass? false
 :result false
 :result-data nil
 :seed 1696006147623
 :shrunk {:depth 0
          :pass? false
          :result false
          :result-data nil
          :smallest [42]
          :time-shrinking-ms 0
          :total-nodes-visited 6}
 :test-var "broken-identity-should-return-argument"}

In particular, this includes the seed. This effectively lets us replay the test case if we wanted:

(require '[clojure.test.check :as tc])
(tc/quick-check
 100
 (prop/for-all
  [x gen/small-integer]
  (= x (broken-identity x)))
 {:seed 1696006147623})

However - if the bug with the code under test throws an exception, then sadly we get a fraction of the data we would ordinarily get:

(defn exploding-identity [x]
  (if (= x 42)
    (throw (Exception. "Boom!"))
    x))

(defspec exploding-identity-should-return-argument 200
  (prop/for-all
   [x gen/small-integer]
   (= x (exploding-identity x))))

;; Test output
{:clojure.test.check.clojure-test/params [42]
 :clojure.test.check.clojure-test/property
   #clojure.test.check.generators.Generator
    {:gen #object [clojure.test.check.generators$gen_fmap$fn__2247 0x2801827a
                   "clojure.test.check.generators$gen_fmap$fn__2247@2801827a"]}
 :type :clojure.test.check.clojure-test/shrunk}

We still get given the data that causes the problem, so that’s good. But it would still be nice to get the seed since that would be easier to copy-paste out of a large test report when working with much more complicated test data.

However, if we require Dominic’s code shown above, we can get some better test output again:

(alter-var-root
 #'clojure.test.check.clojure-test/assert-check
 ;;; etc
 )

;; Test output
{:fail [42]
 :failed-after-ms 6
 :failing-size 95
 :num-tests 96
 :pass? false
 :seed 1696007827585
 :shrunk {:depth 0
          :pass? false
          :result-data {}
          :smallest [42]
          :time-shrinking-ms 1
          :total-nodes-visited 6}
 :test-var exploding-identity-should-return-argument}

That’s better!

Also, test runners play a bit more nicely with it; in Cider, without the fix I just get the following output:

Error in exploding-identity-should-return-argument
Uncaught exception, not in assertion
   error: java.lang.Exception: Boom!

I can’t even see the problematic input, never mind the seed!

But with the fix in place, I get:

Error in exploding-identity-should-return-argument
{:shrunk {:total-nodes-visited 6, :depth 0, :pass? false, :result-data {},
          :time-shrinking-ms 0, :smallest [42]}
 :failed-after-ms 4, :num-tests 48, :seed 1696008029883,
 :fail [42], :failing-size 47,
 :pass? false,
 :test-var "exploding-identity-should-return-argument"}
 expected: {:result true}
   error: java.lang.Exception: Boom!

Nice!

It’s a shame that the Jira for this issue isn’t getting any attention - but at least there’s a workaround we can make use of.