| C L A I M |
| L E N S E |
| A N I L E |
| I S L E T |
| M E E T S |
This library is not maintained any longer and will not be ported to Scala 3. Please consider switching to:
By default, when a ScalaCheck property fails, you'll see its inputs
(e.g. ARG_0: ...
) but not the expression that actually failed. In
many cases it is useful to be able to see the test or comparison that
failed.
This library provides a Claim(...)
macro which wraps any Boolean
expression and converts it into a labeled Prop
value. If the
property fails, ScalaCheck will show you this label.
Claimant supports Scala 2.11, 2.12, and 2.13. It is available from Sonatype.
To include Claimant in your projects, you can use the following
build.sbt
snippet:
libraryDependencies += "org.typelevel" %% "claimant" % "0.1.0"
Claimant also supports Scala.js. To use Claimant in your Scala.js
projects, include the following build.sbt
snippet:
libraryDependencies += "org.typelevel" %%% "claimant" % "0.1.0"
Please note that Claimant is still a very young project. While we
will try to keep basic source compatibility around the Claim(...)
macro itself, it's very likely that Claim's library internals will
change significantly between releases. No compatibility (binary or
otherwise) is guaranteed at this point.
Here's an example of using Claim(...)
to try to prove that Float
is associative:
package mytest
import org.scalacheck.{Prop, Properties}
import org.typelevel.claimant.Claim
object MyTest extends Properties("MyTest") {
property("float is associative") =
Prop.forAll { (x: Float, y: Float, z: Float) =>
Claim((x + (y + z)) == ((x + y) + z))
}
}
Unfortunately for us, this isn't true and ScalaCheck will quickly find a counter-example:
[info] ! MyTest.float is associative: Falsified after 22 passed tests.
[info] > Labels of failing property:
[info] falsified: 0.2962196 == 0.29621956
[info] > ARG_0: 0.29622045
[info] > ARG_1: -8.811786E-7
[info] > ARG_2: 1.0369974E-8
The Claim(...)
call inspects the expression and tries to determine
what kind of operator is being used. Finding ==
, it captures the
left- and right-hand sides of that operator. Since the values are not
equal, it labels the property with:
falsified: 0.2962196 == 0.29621956
This means that in addition to seeing which inputs cause a failure, we
also see how much we failed by (around 4e-8
in this case).
Similarly, in some cases we want to be sure that at least one of
several conditions is true. In this case, we want to be sure that
either n
is zero, or that n
is not equal to -n
.
package mytest
import org.scalacheck.{Prop, Properties}
import org.typelevel.claimant.Claim
object AnotherTest extends Properties("AnotherTest") {
property("ints have distinct inverses") = {
Prop.forAll { (n: Int) =>
Claim(n == 0 || n != -n)
}
}
}
Once again, we are out of luck! It turns out that Int.MinValue
is
its own negation (there is no positive value large enough to represent
its actual negation). ScalaCheck helpfully shows us this:
[info] ! AnotherTest.ints have distinct inverses: Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] falsified: (-2147483648 == 0 {false}) || (-2147483648 != -2147483648 {false})
[info] > ARG_0: -2147483648
In this case, Claim(_)
helpfully shows us the how the different
branches evaluate (summarizing each branch with {true}
or
{false}
). Being able to see that the right branch ended up testing
-2147483648 != -2147483648
cuts to the heart of the problem, and
doesn't leave the caller guessing about how the conditions were
evaluated.
The Claim(_)
macro recognizes many different kinds of Boolean
expressions:
==
and!=
(universal equality)eq
andne
(referential equality)<
,<=
,>
, and>=
(comparisons)&&
,&
,||
,|
,^
, and!
(boolean operators)isEmpty
andnonEmpty
startsWith
andendsWith
contains
,containsSlice
, andapply
isDefinedAt
,sameElements
, andsubsetOf
exists
andforall
(althoughFunction1
values can't be displayed)Equiv#equiv
The Claim(_)
macro also recognizes certain kinds of expressions
which it will attempt to annotate, such as:
size
andlength
compare
,compareTo
, andlengthCompare
min
andmax
Ordering#compare
andPartialOrdering#tryCompare
Ordering.Implicits.infixOrderingOps
(For examples of the labels produced by these, see ClaimTest
.)
It should be fairly straightforward to extend this to support other
shapes, both for Boolean
expressions and for general annotations.
Claimant uses its own Render[A]
typeclass to produce human-readable
representations of values. We get a number of benefits from this:
- We can provide a more useful representation of arrays than you
would get with
.toString
. - We can quote and escape
String
andChar
values to make it easier to see their exact value. - We support user-provided display strategies for types they don't control.
- We support all of the above recursively, allowing collections, tuples, case classes, etc. to take advantage of all of these together.
For types that don't have their own Render
instances, Claimant will
use a low-priority implicit value to provide an implementation based
on .toString
. Some attempt has been made to provide implementations
for most built-in Scala types, but PRs adding support for new
instances (along with tests exercising them) will be gladly accepted.
In particular, Claimant makes it easy to define Render
implementations for case classes. Simply do the following:
case class MyClass(...)
object MyClass {
implicit val renderForMyClass: Render[MyClass] =
Render.caseClass[MyClass]
}
The code above will define a render instance for MyClass
which will
render each of its fields with its corresponding instances. (In the
future Claimant may use Shapeless to derive Render
instances
automatically.)
Currently Claim(...)
will potentially evaluate its expression (or
sub-expressions) multiple times, in any order. In the future we could
be more uptight about preserving execution order and ensuring
sub-expressions are run exactly as they would be, but so far this
hasn't been a priority.
For example, assuming that the missilesFiredAt
method is
side-effecting, and returns the number of missiles that were just
fired, consider the following code:
def missilesFiredAt(target: String): Int = {
val num = scala.util.Random.nextInt(3) + 3
println(s"firing $num missiles at $target")
num
}
property("notTooManyMissiles") =
Claim((missilesFiredAt("moon") max missilesFiredAt("mars")) < 4)
Setting aside the questionable wisdom of launching missiles during a test, here's an example of the output we might see:
firing 5 missiles at moon
firing 3 missiles at mars
firing 4 missiles at moon
firing 4 missiles at mars
firing 4 missiles at moon
firing 3 missiles at mars
[info] ! MissileTest.notTooManyMissiles: Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] falsified: 4 max 4 {4} < 4
As we can see, Claimant is evaluating each expression multiple times.
The values we see in the test label (4
for the Moon and 4
for
Mars) aren't necessarily the same ones used to describe the test
failing, we also see that at various points we launched 5
missiles
at the Moon, and 3
missiles at Mars.
In cases where side-effects are unavoidable, consider evaluating them
before calling Claim(...)
:
property("notTooManyMissiles") = {
val x = missilesFiredAt("moon")
val y = missilesFiredAt("mars")
Claim((x max y) < 4)
}
This will result in more consistent test output:
firing 5 missiles at moon
firing 5 missiles at mars
[info] ! MissileTest.notTooManyMissiles: Falsified after 0 passed tests.
[info] > Labels of failing property:
[info] falsified: 5 max 5 {5} < 4
Currently Claim(...)
only expands a set of known methods. This means
that if you have methods which return Boolean
and write something
like Claim(Verifier.verify(dataSet))
your test failures will look
something like this:
[info] ! FancyTest.verify data sets: Falsified after 4 passed tests.
[info] > Labels of failing property:
[info] falsified: false
[info] > ARG_0: DataSet(...)
The ways to fix this are:
- Have
verify
return a richer result. - Inline the
verify
logic in theClaim(...)
call. - Extend Claimant to support
Verifier.verify
.
Another problem is that Claim(...)
inspects method calls based on
their AST shape. This means that type application, implicit
parameters, etc. need to be explicitly supported. This also means that
implicit enrichment (or bedazzlement) can muddy the waters a bit and
obscure the underlying values.
(For an example of how to deal with enrichment, see the support for
Ordering.Implicits.infixOrderingOps
.)
To measure code coverage for 2.12, do the following:
$ sbt '++ 2.12.12' coreJVM/clean coverage coreJVM/test coverageReport
Assuming everything works, the result should end up someplace like:
.jvm/target/scala-2.12/scoverage-report/index.html
To measure coverage in 2.11 you'd instead do:
$ sbt '++ 2.11.12' coreJVM/clean coverage coreJVM/test coverageReport
And the result would end up someplace like:
.jvm/target/scala-2.11/scoverage-report/index.html
There's at least some code in Claimant that is specific to either 2.11 or 2.12, making it unlikely that we'll achieve 100% coverage under either version independently.
There are a ton of possible improvements:
- Support more methods/shapes.
- Minimize recomputation in the macro.
- Consider using raw trees instead of quasi-quotes.
- Consider supporting fancy diagrams
- Consider supporting color output
- Consider an extensible/modular design
People are expected to follow the Scala Code of Conduct when discussing Claimant on GitHub, the Gitter channel, or other venues.
This library was inspired by the assert(...)
macro found in
ScalaTest.
All code is available to you under the Apache 2 license, available at https://opensource.org/licenses/Apache-2.0.
Copyright Erik Osheim, 2019.