A simple testing framework for Scala
uTest (pronounced micro-test) is a simple, intuitive testing library for Scala. Its key features are:
must_==/
must beEqual/
should beopertors
mainmethod, or in Ammonite Scripts)
Unlike traditional testing libraries for Scala (like Scalatest or Specs2) uTest aims to be simple enough you will never "get lost" in its codebase and functionality, so you can focus on what's most important: your tests.
While uTest has many fewer features than other libraries, the features that it does provide are polished and are enough to build and maintain test suites of any size. uTest is used for countless projects, from the 1-file test suite for Fansi to the 50-file 9,000-line test suite for Ammonite
If you use uTest and like it, please support it by donating to our Patreon:
Most people coming to uTest will be running tests through SBT. Add the following to your
build.sbtand you can immediately begin defining and running tests programmatically.
libraryDependencies += "com.lihaoyi" %% "utest" % "0.7.7" % "test"testFrameworks += new TestFramework("utest.runner.Framework")
To use it with Scala.js or Scala-Native:
libraryDependencies += "com.lihaoyi" %%% "utest" % "0.7.7" % "test"testFrameworks += new TestFramework("utest.runner.Framework")
For Scala-Native, you will also need
nativeLinkStubs := true
Put this in your
src/test/scala/folder:
package test.utest.examplesimport utest._
object HelloTests extends TestSuite{ val tests = Tests{ test("test1"){ throw new Exception("test1") } test("test2"){ 1 } test("test3"){ val a = List[Byte](1, 2) a(10) } } }
You can then run this via
sbt myproject/test
Which should produce this output:
-------------------------------- Running Tests -------------------------------- Setting up CustomFramework X test.utest.examples.HelloTests.test1 4ms java.lang.Exception: test1 test.utest.examples.HelloTests$.$anonfun$tests$2(HelloTests.scala:7) + test.utest.examples.HelloTests.test2.inner 0ms 1 X test.utest.examples.HelloTests.test3 0ms java.lang.IndexOutOfBoundsException: 10 scala.collection.LinearSeqOptimized.apply(LinearSeqOptimized.scala:63) scala.collection.LinearSeqOptimized.apply$(LinearSeqOptimized.scala:61) scala.collection.immutable.List.apply(List.scala:86) test.utest.examples.HelloTests$.$anonfun$tests$5(HelloTests.scala:16) Tearing down CustomFramework Tests: 3, Passed: 1, Failed: 2
The tests are run one at a time, and any tests that fail with an exception have their stack trace printed. If the number of tests is large, a separate results-summary and failures-summary will be shown after all tests have run.
Note that tests within the suite can nested within each other, but only directly. E.g. you cannot define tests within
if-statements or
for-loops. uTest relies on the test structure to be statically known at compile time. They can be nested arbitrarily deep:
package test.utest.examplesimport utest._
object NestedTests extends TestSuite{ val tests = Tests{ val x = 1 test("outer1"){ val y = x + 1
test("inner1"){ assert(x == 1, y == 2) (x, y) } test("inner2"){ val z = y + 1 assert(z == 3) } } test("outer2"){ test("inner3"){ assert(x > 1) } }
} }
Here, we have a tree of nested blocks, with three tests at the inner-most blocks of this tree:
innerest,
inner2and
inner3. Test blocks can be nested arbitrary deeply to help keep things neat, and only the inner-most blocks are considered to be tests.
When this suite is run with
sbt myproject/test, it is those three tests that get executed:
------------------------------- Running Tests ------------------------------- + test.utest.examples.NestedTests.outer1.inner1 21ms (1,2) + test.utest.examples.NestedTests.outer1.inner2 0ms + test.utest.examples.NestedTests.outer2.inner3 0ms
You can see also that
test.utest.examples.NestedTests.outer1.inner1displays the value of
(x, y)returned from the test:
(1,2). Returning a value from a test is useful if you want to skim the test's results after a run to perform manual sanity-checks on some computed value.
If you find yourself wanting to define a test in a for, loop, e.g.
// Doesn't work! val tests = Tests{ for(fileNameYou can instead factor out the common code into a function, and call that from each distinct test case:
// Works! val tests = Tests{ def runTestChecks(fileName: String) = { // lots of code using fileName } test("hello"){ runTestChecks("hello") } test("world"){ runTestChecks("world") } test("i"){ runTestChecks("i") } test("am"){ runTestChecks("am") } test("cow"){ runTestChecks("cow") } }Or even using the TestPath that is available implicitly to every test, you can remove the duplication between the test name and the call to
runTestChecks():# Also works! val tests = Tests{ def runTestChecks()(implicit path: utest.framework.TestPath) = { val fileName = path.value.last // lots of code using fileName } test("hello"){ runTestChecks() } test("world"){ runTestChecks() } test("i"){ runTestChecks() } test("am"){ runTestChecks() } test("cow"){ runTestChecks() } }Running Tests
Apart from running all tests at once using
sbt myproject/testYou can also run individual tests using their full path e.g.
sbt 'myproject/test-only -- test.utest.examples.NestedTests.outer1.inner1' sbt 'myproject/test-only -- test.utest.examples.NestedTests.outer2.inner2' sbt 'myproject/test-only -- test.utest.examples.NestedTests.outer2.inner3'You can also wrap the test selector in double quotes, which lets you run test whose path segments contain spaces or other special characters:
sbt 'myproject/test-only -- "test.utest.examples.NestedTests.segment with spaces.inner"'You can run groups of tests by providing the path to the block enclosing all of them:
# runs both tests `inner1` and `inner2` sbt 'myproject/test-only -- test.utest.examples.NestedTests.outer1'runs all tests in the
NestedTests
test suitesbt 'myproject/test-only -- test.utest.examples.NestedTests'
runs all tests in
NestedTests
and any other suites withintest.utest.examples
sbt 'myproject/test-only -- test.utest.examples'
You can also use the
{foo,bar}syntax to specify exactly which tests you would like to run:# runs both tests `inner2` and `inner3`, explicitly sbt 'myproject/test-only -- test.examples.NestedTests.outer1.{inner1,inner2}'runs
outer1.inner1
andouter2.inner3
but notouter1.inner2
sbt 'myproject/test-only -- test.examples.NestedTests.{outer1.inner1,outer2.inner3}'
also runs
inner1
andinnerest
(and any other test insideinner1
) but notinner2
sbt 'myproject/test-only -- test.examples.NestedTests.{outer1.inner1,outer2}'
The same syntax can be used to pick and choose specific
TestSuites to run, or tests within those test suites:# Run every test in `HelloTests` and `NestedTests` sbt 'myproject/test-only -- test.examples.{HelloTests,NestedTests}'Runs
HelloTests.test1
,NestedTests.outer1.inner2
andNestedTests.outer2.inner3
sbt 'myproject/test-only -- test.examples.{HelloTests.test1,NestedTests.outer2}' sbt 'myproject/test-only -- {test.examples.HelloTests.test1,test.examples.NestedTests.outer2}'
In general, it doesn't really matter if you are running individual tests, groups of tests within a
TestSuite, individualTestSuites, or packages containingTestSuite. These all form one a single large tree of tests that you can run, using the same uniform syntax.By default, SBT runs multiple test suites in parallel, so the output from those suites may be interleaved. You can set
parallelExecution in Test := falsein your SBT config to make the tests execute sequentially, so the output from each suite will be grouped together in the terminal.uTest defaults to emitting ANSI-colored terminal output describing the test run. You can configure the colors by overriding methods on your test suite, or disable it altogether with
override def formatColor = false.Sharing Setup Code, and Sharing Setup Objects
As you saw in the previous section, you can define blocks tests to group them together, and have them share common initialization code in the enclosing block. You can also define mutable values, or "fixtures" in the shared initialization code, and each nested test with a
Testsblock gets its own copy of any mutable variables defined within it:package test.utest.examplesimport utest._
object SeparateSetupTests extends TestSuite{ val tests = Tests{ var x = 0 test("outer1"){ x += 1 test("inner1"){ x += 2 assert(x == 3) // += 1, += 2 x } test("inner2"){ x += 3 assert(x == 4) // += 1, += 3 x } } test("outer2"){ x += 4 test("inner3"){ x += 5 assert(x == 9) // += 4, += 5 x } } } }
Here, you can see that the
xavailable in each inner test block (inner1,inner2,inner3) is separate and independent: each test gets a new copy ofx, modified only by that test's enclosing blocks. This helps avoid inter-test interference (where a test ends up implicitly depending on some state initialized by another test running earlier in the block) while still making it convenient to share common setup code between the various tests in your suite.If you want the mutable fixtures to really-truly be shared between individual tests (e.g. because they are expensive to repeatedly initialize) define it outside the
Tests{}block in the enclosing object:package test.utest.examplesimport utest._
object SharedFixturesTests extends TestSuite{ var x = 0 val tests = Tests{ test("outer1"){ x += 1 test("inner1"){ x += 2 assert(x == 3) // += 1, += 2 x } test("inner2"){ x += 3 assert(x == 7) // += 1, += 2, += 1, += 3 x } } test("outer2"){ x += 4 test("inner3"){ x += 5 assert(x == 16) // += 1, += 2, += 1, += 3, += 4, += 5 x } } } }
Here you see that the changes to
xare being shared between the invocations of all the tests. If you are initializing something expensive to share between tests, this allows you to avoid paying that cost multiple times, but you need to be careful the tests aren't mutating shared state that could cause other tests to fail!Other Ways of Naming tests
You can also use the
test("symbol") -syntax, if your tests are simply forwarding to a separate helper method to do the real testing:test("test1") - processFileAndCheckOutput("input1.txt", "expected1.txt") test("test2") - processFileAndCheckOutput("input2.txt", "expected2.txt") test("test3") - processFileAndCheckOutput("input3.txt", "expected3.txt")The
test("string"){...}andtest("symbol")...syntaxes are equivalent.The last way of defining tests is with the
utest.*symbol, e.g. these tests from the Fansi project:test("parsing"){ def check(frag: fansi.Str) = { val parsed = fansi.Str(frag.render) assert(parsed == frag) parsed } test{ check(fansi.Color.True(255, 0, 0)("lol")) } test{ check(fansi.Color.True(1, 234, 56)("lol")) } test{ check(fansi.Color.True(255, 255, 255)("lol")) } test{ (for(iTests defined using the
*symbol are give the numerical names "0", "1", "2", etc.. This is handy if you have a very large number of very simple test cases, don't really care what each one is called, but still want to be able to run them and collect their result separately.Asynchronous Tests
val tests = Tests { test("testSuccess"){ Future { assert(true) } } test("testFail"){ Future { assert(false) } } test("normalSuccess"){ assert(true) } test("normalFail"){ assert(false) } }TestRunner.runAsync(tests).map { results => val leafResults = results.leaves.toSeq assert(leafResults(0).value.isSuccess) // root assert(leafResults(1).value.isSuccess) // testSuccess assert(leafResults(2).value.isFailure) // testFail assert(leafResults(3).value.isSuccess) // normalSuccess }
You can have tests which return (have a last expression being) a
Future[T]instead of a normal value. You can run the suite using.runAsyncto return aFutureof the results, or you can continue using.runwhich will wait for all the futures to complete before returning.In Scala.js, calling
.runon a test suite with futures in it throws an error instead of waiting, since you cannot wait for asynchronous results in Scala.js.When running the test suites from SBT, you do not need worry about any of this
runvsrunAsyncstuff: the test runner will handle it for you and provide the correct results.Smart Asserts
val x = 1 val y = "2" assert( x > 0, x == y )// utest.AssertionError: x == y // x: Int = 1 // y: String = 2
uTest comes with a macro-powered smart
asserts that provide useful debugging information in the error message. These take one or more boolean expressions, and when they fail, will print out the names, types and values of any local variables used in the expression that failed. This makes it much easier to see what's going on than Scala's defaultassert, which gives you the stack trace and nothing else.uTest also wraps any exceptions thrown within the assert, to help trace what went wrong:
val x = 1L val y = 0L assert(x / y == 10)// utest.AssertionError: assert(x / y == 10) // caused by: java.lang.ArithmeticException: / by zero // x: Long = 1 // y: Long = 0
The origin exception is stored as the
causeof theutest.AssertionError, so the original stack trace is still available for you to inspect.uTest's smart asserts live on the
utestpackage, and are brought into scope viaimport utest._. If you want to continue using the built-in Scala asserts, e.g. if you want custom messages if the assert fails, those remain available asPredef.assert.Arrow Asserts
1 ==> 1 // passes Array(1, 2, 3) ==> Array(1, 2, 3) // passes try{ 1 ==> 2 // throws }catch{case e: java.lang.AssertionError => e }You can use
a ==> bas a shorthand forassert(a == b). This results in pretty code you can easily copy-paste into documentation.Intercept
val e = intercept[MatchError]{ (0: Any) match { case _: String => } } println(e)// scala.MatchError: 0 (of class java.lang.Integer)
interceptallows you to verify that a block raises an exception. This exception is caught and returned so you can perform further validation on it, e.g. checking that the message is what you expect. If the block does not raise one, anAssertionErroris raised.As with
assert,interceptadds debugging information to the error messages if theinterceptfails or throws an unexpected Exception.Eventually and Continually
val x = Seq(12) eventually(x == Nil)// utest.AssertionError: eventually(x == Nil) // x: Seq[Int] = List(12)
In addition to a macro-powered
assert, uTest also provides macro-powered versions ofeventuallyandcontinually. These are used to test asynchronous concurrent operations:
eventually(tests: Boolean*): ensure that the boolean values of
testsall become true at least once within a certain period of time.
continually(tests: Boolean*): ensure that the boolean values of
testsall remain true and never become false within a certain period of time.
These are implemented via a retry-loop, with a default retry interval of 0.1 second and retries up to a total of 1 second. If you want to change this behavior, you can shadow the implicit values
retryIntervaland
retryMax, for example this:
implicit val retryMax = RetryMax(300.millis) implicit val retryInterval = RetryInterval(50.millis)
Would set the retry-loop to happen every 50ms up to a max of 300ms.
Together, these two operations allow you to easily test asynchronous operations. You can use them to help verify Liveness properties (that condition must eventually be met) and Safety properties (that a condition is never met)
As with
assert,
eventuallyand
continuallyadd debugging information to the error messages if they fail.
assertMatch(Seq(1, 2, 3)){case Seq(1, 2) =>} // AssertionError: Matching failed Seq(1, 2, 3)
assertMatchis a convenient way of checking that a value matches a particular shape, using Scala's pattern matching syntax. This includes support for use of
|or
_or
if-guards within the pattern match. This gives you additional flexibility over a traditional
assert(a == Seq(1, 2)), as you can use
_as a wildcard e.g. using
assertMatch(a){case Seq(1, _)=>}to match any 2-item
Seqwhose first item is
1.
As with
assert,
assertMatchadds debugging information to the error messages if the value fails to match or throws an unexpected Exception while evaluating.
compileError("true * false") // CompileError.Type("value * is not a member of Boolean")compileError("(}") // CompileError.Parse("')' expected but '}' found.")
compileErroris a macro that can be used to assert that a fragment of code (given as a literal String) fails to compile.
compileErrorwill fail the compilation run with a message.
compileErrorwill return an instance of
CompileError, one of
CompileError.Type(pos: String, msgs: String*)or
CompileError.Parse(pos: String, msgs: String*)to represent typechecker errors or parser errors
In general,
compileErrorworks similarly to
intercept, except it does its checks (that a snippet of code fails) and errors (if it doesn't fail) at compile-time rather than run-time. If the code fails as expected, the failure message is propagated to runtime in the form of a
CompileErrorobject. You can then do whatever additional checks you want on the failure message, such as verifying that the failure message contains some string you expect to be there.
The
compileErrormacro compiles the given string in the local scope and context. This means that you can refer to variables in the enclosing scope, i.e. the following example will fail to compile because the variable
xexists.
val x = 0compileError("x + x"), // [error] compileError check failed to have a compilation error
The returned
CompileErrorobject also has a handy
.checkmethod, which takes a position-string indicating where the error is expected to occur, as well as zero-or-more messages which are expected to be part of the final error message. This is used as follows:
compileError("true * false").check( """ compileError("true * false").check( ^ """, "value * is not a member of Boolean" )
Note that the position-string needs to exactly match the line of code the compile-error occured on. This includes any whitespace on the left, as well as any unrelated code or comments sharing the same line as the
compileErrorexpression.
uTest provides a range of test utilities that aren't strictly necessary, but aim to make your writing of tests much more convenient and DRY.
package test.utest.examplesimport utest._
object TestPathTests extends TestSuite{ val tests = Tests{ 'testPath{ 'foo { assert(implicitly[utest.framework.TestPath].value == Seq("testPath", "foo")) } } } }
uTest exposes the path to the current test to the body of the test via the
utest.framework.TestPathimplicit. This can be used to print debugging information while the test is running ("test foo.bar.baz 50% done") or to avoid repetition between the test name and test body.
One example is the Fastparse test suite, which uses the name of the test to provide the repository that it needs to clone and parse:
def checkRepo(filter: String => Boolean = _ => true) (implicit testPath: utest.framework.TestPath) = { val url = "https://github.com/" + testPath.value.last import sys.process._ val name = url.split("/").last if (!Files.exists(Paths.get("target", "repos", name))){ println("CLONING") Seq("git", "clone", url, "target/repos/"+name, "--depth", "1").! } checkDir("target/repos/"+name, filter) }"lihaoyi/fastparse" - checkRepo() "scala-js/scala-js" - checkRepo() "scalaz/scalaz" - checkRepo() "milessabin/shapeless" - checkRepo() "akka/akka"- checkRepo() "lift/framework" - checkRepo() "playframework/playframework" - checkRepo() "PredictionIO/PredictionIO" - checkRepo() "apache/spark" - checkRepo()
This allows us to keep the tests DRY - avoiding having to repeat the name of the repo in the name of the test for every test we define - as well as ensuring that they always stay in sync.
If you need additional metadata such as line-numbers or file-paths or class or package names, you can use the SourceCode library's implicits to pull them in for you.
object LocalRetryTests extends utest.TestSuite{ val flaky = new FlakyThing def tests = Tests{ test("hello") - retry(3){ flaky.run } } }
You can wrap individual tests, or even individual expressions, in a
retryblock to make them retry a number of times before failing. That is very useful for dealing with small points of flakiness within your test suite. A
retryblock simply retries its body up to the specified number of times; the first run that doesn't throw an exception returns the value returned by that run.
You can also use Suite Retries if you want to configure retries more globally across your test suite.
To configure things which affect an entire test run, and not any individual
TestSuiteobject, you can create your own subclass of
utest.runner.Frameworkand override the relevant methods.
For example, if you need to perform some action (initialize a database, cleanup the filesystem, etc.) not just per-test but per-run, you can do that by defining a custom
utest.runner.Frameworkand overriding the
setupand
teardownmethods:
class CustomFramework extends utest.runner.Framework{ override def setup() = { println("Setting up CustomFramework") } override def teardown() = { println("Tearing down CustomFramework") } }
And then telling SBT to run tests using the custom framework:
testFrameworks += new TestFramework("test.utest.CustomFramework"),
This is handy for setup/teardown that is necessary but too expensive to do before/after every single test, which would be the case if you used Test Wrapping to do it.
Apart from setup and teardown, there are other methods on
utest.runner.Frameworkthat you can override to customize:
/** * Override to run code before tests start running. Useful for setting up * global databases, initializing caches, etc. */ def setup() = ()/** * Override to run code after tests finish running. Useful for shutting * down daemon processes, closing network connections, etc. */ def teardown() = ()
/** * How many tests need to run before uTest starts printing out the test * results summary and test failure summary at the end of a test run. For * small sets of tests, these aren't necessary since all the output fits * nicely on one screen; only when the number of tests gets large and their * output gets noisy does it become valuable to show the clean summaries at * the end of the test run. */ def showSummaryThreshold = 30
/** * Whether to use SBT's test-logging infrastructure, or just println. * * Defaults to println because SBT's test logging doesn't seem to give us * anything that we want, and does annoying things like making a left-hand * gutter and buffering input by default */ def useSbtLoggers = false
def resultsHeader = BaseRunner.renderBanner("Results") def failureHeader = BaseRunner.renderBanner("Failures")
def startHeader(path: String) = BaseRunner.renderBanner("Running Tests" + path)
You can control how the output of tests gets printed via overriding methods on the
Frameworkclass, described above. For example, this snippet overrides the
exceptionStackFrameHighlightermethod to select which stack frames in a stack trace are more relevant to you, so they can be rendered more brightly:
class CustomFramework extends utest.runner.Framework{ override def exceptionStackFrameHighlighter(s: StackTraceElement) = { s.getClassName.contains("utest.") } }
This results in stack traces being rendered as such:
the
utest.runner.Frameworkprovides a wide range of hooks you can use to customize how the uTest output is colored, wrapped, truncated or formatted:
def formatColor: Boolean = true def formatTruncateHeight: Int = 15 def formatWrapWidth: Int = 100def formatValue(x: Any) = testValueColor(x.toString)
def toggledColor(t: ufansi.Attrs) = if(formatColor) t else ufansi.Attrs.Empty def testValueColor = toggledColor(ufansi.Color.Blue) def exceptionClassColor = toggledColor(ufansi.Underlined.On ++ ufansi.Color.LightRed) def exceptionMsgColor = toggledColor(ufansi.Color.LightRed) def exceptionPrefixColor = toggledColor(ufansi.Color.Red) def exceptionMethodColor = toggledColor(ufansi.Color.LightRed) def exceptionPunctuationColor = toggledColor(ufansi.Color.Red) def exceptionLineNumberColor = toggledColor(ufansi.Color.LightRed) def exceptionStackFrameHighlighter(s: StackTraceElement) = true
def formatResultColor(success: Boolean) = toggledColor( if (success) ufansi.Color.Green else ufansi.Color.Red )
def formatMillisColor = toggledColor(ufansi.Bold.Faint)
Any methods overriden on your own custom
Frameworkapply to every
TestSuitethat is run. If you want to further customize how a single
TestSuite's output is formatted, you can override
utestFormatteron that test suite.
Note that uTest uses an internal copy of the Fansi library, vendored at
utest.ufansi, in order to avoid any compatibility problems with any of your other dependencies. You can use
ufansito construct the colored
ufansi.Strs that these methods require, or you could just return colored
java.lang.Stringobjects containing ANSI escapes, created however you like, and they will be automatically parsed into the correct format.
You can mix in the
TestSuite.Retriestrait to any
TestSuiteand define the
utestRetryCountint to enable test-level retries for all tests within a suite:
object SuiteRetryTests extends TestSuite with TestSuite.Retries{ override val utestRetryCount = 3 val flaky = new FlakyThing def tests = Tests{ 'hello{ flaky.run } } }
You can also use Local Retries if you want to only retry within specific tests or expressions instead of throughout the entire suite.
uTest offers the
utestBeforeEachand
utestAfterEachmethods that you can override on any
TestSuite, these methods are invoked before and after running each test.
def utestBeforeEach(path: Seq[String]): Unit = () def utestAfterEach(path: Seq[String]): Unit = ()
These are equivalent to
utestWrapbut easier to use for simple cases.
package test.utest.examplesimport utest._ object BeforeAfterEachTest extends TestSuite { var x = 0 override def utestBeforeEach(path: Seq[String]): Unit = { println(s"on before each x: $x") x = 0 } override def utestAfterEach(path: Seq[String]): Unit = println(s"on after each x: $x")
val tests = Tests{ test("outer1"){ x += 1 test("inner1"){ x += 2 assert(x == 3) // += 1, += 2 x } test("inner2"){ x += 3 assert(x == 4) // += 1, += 3 x } } test("outer2"){ x += 4 test("inner3"){ x += 5 assert(x == 9) // += 4, += 5 x } } } }
-------------------------------- Running Tests -------------------------------- Setting up CustomFramework on before each x: 0 on after each x: 3
Both
utestBeforeEachand
utestAfterEachruns inside
utestWrap's
bodycallback.
If you need something fancier than what
utestBeforeEachor
utestAfterEachprovide, e.g. passing initialized objects into the main test case or tearing them down after the test case has completed, feel free to define your test wrapper/initialization function and use it for each test case:
def myTest[T](func: Int => T) = { val fixture = 1337 // initialize some value val res = func(fixture) // make the value available in the test case assert(fixture == 1337) // properly teardown the value later res }test("test") - myTest{ fixture => // do stuff with fixture }
The above
myTestfunction can also take a TestPath implicit if it wants access to the current test's path.
If you're looking for something like
utestBeforeAll, you can add your code to the object body, and you can also use lazy val to delay the initialization until the test suite object is created.
uTest offers the
utestAfterAllmethod that you can override on any test suite, this method is invoked after running the entire test suite.
def utestAfterAll(): Unit = ()
package test.utest.examplesimport utest._ object BeforeAfterAllSimpleTests extends TestSuite { println("on object body, aka: before all")
override def utestAfterAll(): Unit = { println("on after all") }
val tests = Tests { test("outer1"){ test("inner1"){ 1 } test("inner2"){ 2 } } } }
-------------------------------- Running Tests -------------------------------- Setting up CustomFramework on object body, aka: before all
uTest exposes the
utestWrapfunction that you can override on any test suite:
def utestWrap(path: Seq[String], runBody: => concurrent.Future[Any]) (implicit ec: ExecutionContext): concurrent.Future[Any]
This is a flexible function that wraps every test call; you can use it to:
runBody
runBodyvia
runBody.onComplete
runBodymultiple times
pathof that test, e.g.
Seq("outer", "inner1", "innerest")for the test
outer.inner1.innerest
Generally, if you want to perform messy before/after logic around every individual test, override
utestWrap. Please remember to call the
utestBeforeEachand
utestAfterEachmethods when needed.
runBodyis a future to support asynchronous testing, which is the only way to test things like Ajax calls in Scala.js
uTest is completely compatible with Scala.js and Scala-Native: the above sections on defining a test suite, asserts and the test-running API all work unchanged under ScalaJS, with minor differences:
ExecutionContextslike
utest.ExecutionContext.runNowor
scala.scalajs.concurrent.JSExecutionContext.runNowwork. When run via SBT,
--parallelhas no effect.
Apart from these differences, there should be no problem compiling uTest TestSuites via Scala.js and running them on Node.js, in the browser, or (with Scala-Native) on LLVM.
Note that Scala-Native support, like Scala-Native, is experimental. While it is tested in a few projects (uTest's own test suite runs in Scala-Native), it does have it's own quirks (e.g.
NullPointerExceptions appear to be fatal) and does not have the weight of third-party usage that the Scala-JVM and Scala.js versions of uTest have.
uTest exposes a straightforward API that allows you to run tests, receive results and format them in a nice way when used standalone, without using SBT's test integration. The following code snippet demonstrates how to define tests and run them directly:
import utest._ val tests = Tests{ test("test1"){ // throw new Exception("test1") } test("test2"){ test("inner"){ 1 } } test("test3"){ val a = List[Byte](1, 2) // a(10) } }// Run and return results val results1 = TestRunner.run(tests)
The
TestRunner.runcall can be customized with additional arguments to control how the code is run, or can be replaced with
.runAsyncto handle asynchronous testing, or
.runAndPrint/
.runAndPrintAsyncif you want to print the results of tests as they complete using the normal output formatter:
// Run, return results, and print streaming output with the default formatter val results2 = TestRunner.runAndPrint( tests, "MyTestSuiteA" ) // Run, return results, and print output with custom formatter and executor val results3 = TestRunner.runAndPrint( tests, "MyTestSuiteA", executor = new utest.framework.Executor{ override def utestWrap(path: Seq[String], runBody: => Future[Any]) (implicit ec: ExecutionContext): Future[Any] = { println("Getting ready to run " + path.mkString(".")) utestBeforeEach() runBody.andThen { case _ => utestAfterEach() } } }, formatter = new utest.framework.Formatter{ override def formatColor = false } )
Lastly, you can also run
TestSuiteobjects in the same way:
// Run `TestSuite` object directly without using SBT, and use // its configuration for execution and output formatting object MyTestSuite extends TestSuite{ val tests = Tests{ test("test1"){ // throw new Exception("test1") } test("test2"){ test("inner"){ 1 } } test("test3"){ val a = List[Byte](1, 2) // a(10) } } }val results4 = TestRunner.runAndPrint( MyTestSuite.tests, "MyTestSuiteB", executor = MyTestSuite, formatter = MyTestSuite )
uTest provides helpers to render the standard summary reports at the end of a test run, once all your results are in:
// Show summary and exit val (summary, successes, failures) = TestRunner.renderResults( Seq( "MySuiteA" -> results1, "MySuiteA" -> results2, "MySuiteA" -> results3, "MySuiteB" -> results4 ) ) if (failures > 0) sys.exit(1)
You can thus use uTest anywhere you can run Scala code, even when not using SBT: in a normal
mainmethod, within Ammonite scripts, or elsewhere.
uTest aims to give you the things that everyone needs to run tests, giving you one obvious way to do common tasks while testing. uTest is simple and compact, letting you avoid thinking about it so you can focus on what's most important: you tests.
Hence uTest provides:
A simple, uniform syntax for delimiting tests and grouping tests together as blocks:
test("test"){ ... }
A simple, uniform syntax for running tests:
foo.BarTests,
foo.BarTests.baz.qux,
foo.BarTests.{baz,qux}or
foo.{BarTests,BazTests}
A simple, uniform syntax for Smart Asserts, that save you the day-to-day drudgery of printing out the value of variables when things fail
Isolation of tests within the same suite, to avoid inter-test interference due to mutable state
uTest tries to provide things that every developer needs, in their minimal, essential form. It intentionally avoids redundant/unnecessary features or syntaxes that bloat the library and make it harder to developers to pick up, which I find to be common in other popular testing libraries like Scalatest or Specs2:
Fluent English-like code: matchers like
shouldBeor
should not beor
mustbe_==don't really add anything, and it doesn't really matter whether you name each test block using
should,
when,
can,
must,
feature("...")or
it should "..."
Multiple redundant ways of defining test suites, individual tests and blocks of related tests
Legacy code, like ScalaTests time package, now obsolete with the introduction of scala.concurrent.duration.
While uTest has and will continue to slowly grow and add more features, it is unlikely that it will ever reach the same level of complexity that other testing libraries are currently at.
test("foo"){...},
test{...}. Old syntax is now deprecated.
Testsmacro
TestUtilswith portable-scala-reflect
IllegalStateException: Unknown opcode 5exceptions occuring with Scala.JS.
nullfrom a test case no longer blows up
utest.runner.MillFramework
Migrate formatting-related configuration options from
utest.TestSuiteonto the
utest.runner.Framework, as described here.
Added the
exceptionStackFrameHighlighterformatting hook, letting you choose which stack frames in an exception are of interest to you so they can be rendered more brightly.
Cross-publish for Scala 2.13.0-M2, Scala.js 0.6.20, Scala-Native 0.3.3
Stack traces for chained exceptions (i.e. those with a
.getCause != null) are now properly displayed when tests fail
Portions of stack traces caused by the internals of the
assertmacros are now hidden, since they aren't relevant to any failure in user code
Revamped test output format, motivated by drhumlen's PR 113, which should be much easier to read and skim
Much smarter test-output-value truncation, now based on lines-of-output (including wrapping) rather than number-of-characters
Line-wrapping of output is now indentation aware: it wraps to the next line with the same indentation, preserving the outline of indentation on the left making it easier to skim
How long tests take (im milliseconds) is now displayed in the standard test output format
Hierarchical test summary and failure-summary are now only shown when you have a large number of tests, since it's not use when running individual or small numbers of tests. The default threshold is 20, which can be configured by defining a custom framework overriding
showSummaryThreshold
Revamped test-query system, now allowing you to run multiple groups of tests at once via
test-only -- mypackage.{foo,bar,baz}or
{mypackage, myotherpackage}or
mypackage.{foo,bar.{baz,qux}}
Overhauled the execution model of test suites: now only the inner-most blocks within your
TestSuite{...}block count as tests, and the surrounding blocks do not. Thus the surrounding blocks no longer show pass/fail status, return a test result-value, or get run independently.
Various user errors (non-existent test, non-existing test suite, invalid test query syntax) now give slightly more friendly error messages
Using uTest with
fork in Test := truein SBT no longer gives an incorrect results summmary
Fix problem with lazy vals in test blocks crashing the compiler #67. Note that the issue is only fixed on 2.12.3, and not on Scala 2.10.x or 2.11.x.
Deprecated the
'foo{...}syntax for defining tests. We should standardize on
test("foo"){...}or
test("foo"){...}
The
TestSuite{...}for defining a block of tests is now deprecated; it never actually defined a
TestSuiteobject, so the name was pretty misleading. The new syntax is
Tests{...}
Remove the
MultipleErrorswrapper when you test more than one expression in an
assertand multiple of them fail. Now only the first expression in the
assertcall which fails will have it's exception thrown.
Moved smart asserts from being embedded into the
TestSuiteobjects directly onto the
utestpackage, so they are pulled in by the
import utest._import.
-opt:l:method)
getStackTraceStringdeprecated in Scala 2.13
==>asserts to work on
Arrays
utest.*from
utest.framework.*
acyclichack variable
foo: @Showannotation lets you tell uTest to print out arbitrary expressions within the test suite when things fail, in addition to the default of local variables
You can use
a ==> bto assert equality within a test suite, in a form that's pretty enough to use as documentation
compileErrornow properly passes when an expression would fail to compile due to
@compileTimeOnlyannotations
Configuring uTest has been overhauled.
Scala
2.12.0-M3support
Fix some warnings appearing when the
-Ydead-codeflag is used
Added TestPath implicit to make the path to the current test available for usage inside the test body or helpers.
JsCrossBuildnow that Scala.js supports cross-building via
crossProject
compileTimeOnlyhas been re-introduced, so invalid use of test DSL should fail with nice errors
--throwflag in favor of "native" SBT error reporting
Future.
CompileError.check(pos: String, msgs: String*)to simplify the common pattern of checking that the error occurs in the right place and with the message you expect.
JsCrossBuild, to expect the shared files to be in
js/shared/and
jvm/shared/, rather than in
shared/. This is typically achieved via symlinks, and should make the cross-build play much more nicely with IDEs.
utestJsSettingspulling in the wrong version of uTest
compileErrormacro to allow testing of compilation errors.
*symbol, which can be used in place of a test name to get sequentially numbered test names.
utestJvmSettingsand
utestJsSettingsfor use outside the
JsCrossBuildplugin, for people who don't want to use the plugin.