Better exception output for test.check

Here’s a hack for getting better output when test.check tests run via clojure.test throw exceptions, thanks to Dominic Monroe.


If you require the following code somewhere in your test suite, then you’ll still get shown the usual helpful test.check output - in particular, the :seed - if the code under test throws an exception:

  (fn [{:keys [result result-data] :as m}]
    (if-let [error ( result-data)]
       {:type :error
        :message (-> m
                     (dissoc :result :result-data)
                     (update :shrunk dissoc :result)
                     (update-in [:shrunk :result-data]
        :expected {:result true}
        :actual error})
      (clojure.test/is (clojure.test.check.clojure-test/check? m))))))

What problem does this fix?

If we run the following test.check test using clojure.test (via the defspec macro, which provides integration between the two)…

(require '[clojure.test.check.generators :as gen])
(require '[ :as prop])
(require '[clojure.test.check.clojure-test :refer [defspec]])

(defn broken-identity [x]
  (if (= x 42) 0 x))

(defspec broken-identity-should-return-argument
   [x gen/small-integer]
   (= x (broken-identity x))))

then we get the following useful output from test.check:

{:fail [42]
 :failed-after-ms 5
 :failing-size 85
 :num-tests 86
 :pass? false
 :result false
 :result-data nil
 :seed 1696006147623
 :shrunk {:depth 0
          :pass? false
          :result false
          :result-data nil
          :smallest [42]
          :time-shrinking-ms 0
          :total-nodes-visited 6}
 :test-var "broken-identity-should-return-argument"}

In particular, this includes the seed. This effectively lets us replay the test case if we wanted:

(require '[clojure.test.check :as tc])
  [x gen/small-integer]
  (= x (broken-identity x)))
 {:seed 1696006147623})

However - if the bug with the code under test throws an exception, then sadly we get a fraction of the data we would ordinarily get:

(defn exploding-identity [x]
  (if (= x 42)
    (throw (Exception. "Boom!"))

(defspec exploding-identity-should-return-argument 200
   [x gen/small-integer]
   (= x (exploding-identity x))))
;; Test output
{:clojure.test.check.clojure-test/params [42]
    {:gen #object [clojure.test.check.generators$gen_fmap$fn__2247 0x2801827a
 :type :clojure.test.check.clojure-test/shrunk}

We still get given the data that causes the problem, so that’s good. But it would still be nice to get the seed since that would be easier to copy-paste out of a large test report when working with much more complicated test data.

However, if we require Dominic’s code shown above, we can get some better test output again:

 ;;; etc
;; Test output
{:fail [42]
 :failed-after-ms 6
 :failing-size 95
 :num-tests 96
 :pass? false
 :seed 1696007827585
 :shrunk {:depth 0
          :pass? false
          :result-data {}
          :smallest [42]
          :time-shrinking-ms 1
          :total-nodes-visited 6}
 :test-var exploding-identity-should-return-argument}

That’s better!

Also, test runners play a bit more nicely with it; in Cider, without the fix I just get the following output:

Error in exploding-identity-should-return-argument
Uncaught exception, not in assertion
   error: java.lang.Exception: Boom!

I can’t even see the problematic input, never mind the seed!

But with the fix in place, I get:

Error in exploding-identity-should-return-argument
{:shrunk {:total-nodes-visited 6, :depth 0, :pass? false, :result-data {},
          :time-shrinking-ms 0, :smallest [42]}
 :failed-after-ms 4, :num-tests 48, :seed 1696008029883,
 :fail [42], :failing-size 47,
 :pass? false,
 :test-var "exploding-identity-should-return-argument"}
 expected: {:result true}
   error: java.lang.Exception: Boom!


It’s a shame that the Jira for this issue isn’t getting any attention - but at least there’s a workaround we can make use of.