= JUnit Platform Test Kit

The `junit-platform-testkit` artifact provides support for executing a test plan on the
JUnit Platform and then verifying the expected results. This support is currently limited
to the execution of a single `TestEngine` (see <<engine>>).

[[engine]]
== Engine Test Kit

The `{testkit-engine-package}` package provides support for discovering and executing a
`{TestPlan}` for a given `{TestEngine}` running on the JUnit Platform and then accessing
the results via convenient result objects. For execution, a fluent API may be used to
verify the expected execution events were received. The key entry point into this API is
the `{EngineTestKit}` which provides static factory methods named `engine()`,
`discover()`, and `execute()`. It is recommended that you select one of the `engine()`
variants to benefit from the fluent API for building a `LauncherDiscoveryRequest`.

NOTE: If you prefer to use the `LauncherDiscoveryRequestBuilder` from the `Launcher` API
to build your `LauncherDiscoveryRequest`, you must use one of the `discover()` or
`execute()` variants in `EngineTestKit`.

The following test class written using JUnit Jupiter will be used in subsequent examples.

[[engine-ExampleTestCase]]
[source,java,indent=0]
----
include::example$java/example/ExampleTestCase.java[tags=user_guide]
----

For the sake of brevity, the following sections demonstrate how to test JUnit's own
`JupiterTestEngine` whose unique engine ID is `"junit-jupiter"`. If you want to test your
own `TestEngine` implementation, you need to use its unique engine ID. Alternatively, you
may test your own `TestEngine` by supplying an instance of it to the
`EngineTestKit.engine(TestEngine)` static factory method.

[[engine-discovery]]
== Verifying Test Discovery

The following test demonstrates how to verify that a `TestPlan` was discovered as expected
by the JUnit Jupiter `TestEngine`.

[source,java,indent=0]
----
include::example$java/example/testkit/EngineTestKitDiscoveryDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the <<engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Discover the `TestPlan`.
<4> Assert engine root descriptor has expected display name.
<5> Assert no discovery issues were encountered.

[[engine-statistics]]
== Asserting Execution Statistics

One of the most common features of the Test Kit is the ability to assert statistics
against events fired during the execution of a `TestPlan`. The following tests demonstrate
how to assert statistics for _containers_ and _tests_ in the JUnit Jupiter `TestEngine`.
For details on what statistics are available, consult the Javadoc for `{EventStatistics}`.

[source,java,indent=0]
----
include::example$java/example/testkit/EngineTestKitStatisticsDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the <<engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _container_ events.
<5> Assert statistics for _container_ events.
<6> Filter by _test_ events.
<7> Assert statistics for _test_ events.

NOTE: In the `verifyJupiterContainerStats()` test method, the counts for the `started` and
`succeeded` statistics are `2` since the `JupiterTestEngine` and the
<<engine-ExampleTestCase, `ExampleTestCase`>> class are both considered containers.

[[engine-events]]
== Asserting Events

If you find that <<engine-statistics, asserting statistics>> alone is insufficient
for verifying the expected behavior of test execution, you can work directly with the
recorded `{Event}` elements and perform assertions against them.

For example, if you want to verify the reason that the `skippedTest()` method in
<<engine-ExampleTestCase, `ExampleTestCase`>> was skipped, you can do that as
follows.

[TIP]
====
The `assertThatEvents()` method in the following example is a shortcut for
`org.assertj.core.api.Assertions.assertThat(events.list())` from the {AssertJ} assertion
library.

For details on what _conditions_ are available for use with AssertJ assertions against
events, consult the Javadoc for `{EventConditions}`.
====

[source,java,indent=0]
----
include::example$java/example/testkit/EngineTestKitSkippedMethodDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the `skippedTest()` method in the <<engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _test_ events.
<5> Save the _test_ `Events` to a local variable.
<6> Optionally assert the expected statistics.
<7> Assert that the recorded _test_ events contain exactly one skipped test named
    `skippedTest` with `"for demonstration purposes"` as the _reason_.

If you want to verify the type of exception thrown from the `failingTest()` method in
<<engine-ExampleTestCase, `ExampleTestCase`>>, you can do that as follows.

[TIP]
====
For details on what _conditions_ are available for use with AssertJ assertions against
events and execution results, consult the Javadoc for `{EventConditions}` and
`{TestExecutionResultConditions}`, respectively.
====

[source,java,indent=0]
----
include::example$java/example/testkit/EngineTestKitFailedMethodDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the <<engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _test_ events.
<5> Assert that the recorded _test_ events contain exactly one failing test named
    `failingTest` with an exception of type `ArithmeticException` and an error message
    that ends with `"/ by zero"`.

Although typically unnecessary, there are times when you need to verify **all** of the
events fired during the execution of a `TestPlan`. The following test demonstrates how to
achieve this via the `assertEventsMatchExactly()` method in the `EngineTestKit` API.

[TIP]
====
Since `assertEventsMatchExactly()` matches conditions exactly in the order in which the
events were fired, <<engine-ExampleTestCase, `ExampleTestCase`>> has been
annotated with `@TestMethodOrder(OrderAnnotation.class)` and each test method has been
annotated with `@Order(...)`. This allows us to enforce the order in which the test
methods are executed, which in turn allows our `verifyAllJupiterEvents()` test to be
reliable.
====

If you want to do a _partial_ match _with_ or _without_ ordering requirements, you can use
the methods `assertEventsMatchLooselyInOrder()` and `assertEventsMatchLoosely()`,
respectively.

[source,java,indent=0]
----
include::example$java/example/testkit/EngineTestKitAllEventsDemo.java[tags=user_guide]
----
<1> Select the JUnit Jupiter `TestEngine`.
<2> Select the <<engine-ExampleTestCase, `ExampleTestCase`>> test class.
<3> Execute the `TestPlan`.
<4> Filter by _all_ events.
<5> Print all events to the supplied `writer` for debugging purposes. Debug information
    can also be written to an `OutputStream` such as `System.out` or `System.err`.
<6> Assert _all_ events in exactly the order in which they were fired by the test engine.

The `debug()` invocation from the preceding example results in output similar to the
following.

[source,options="nowrap"]
----
All Events:
	Event [type = STARTED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.082280Z, payload = null]
	Event [type = STARTED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.089339Z, payload = null]
	Event [type = SKIPPED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:skippedTest()], timestamp = 2018-12-14T12:45:14.094314Z, payload = 'for demonstration purposes']
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.095182Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:succeedingTest()], timestamp = 2018-12-14T12:45:14.104922Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.106121Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:abortedTest()], timestamp = 2018-12-14T12:45:14.109956Z, payload = TestExecutionResult [status = ABORTED, throwable = org.opentest4j.TestAbortedException: Assumption failed: abc does not contain Z]]
	Event [type = STARTED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.110680Z, payload = null]
	Event [type = FINISHED, testDescriptor = TestMethodTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase]/[method:failingTest()], timestamp = 2018-12-14T12:45:14.111217Z, payload = TestExecutionResult [status = FAILED, throwable = java.lang.ArithmeticException: / by zero]]
	Event [type = FINISHED, testDescriptor = ClassTestDescriptor: [engine:junit-jupiter]/[class:example.ExampleTestCase], timestamp = 2018-12-14T12:45:14.113731Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
	Event [type = FINISHED, testDescriptor = JupiterEngineDescriptor: [engine:junit-jupiter], timestamp = 2018-12-14T12:45:14.113806Z, payload = TestExecutionResult [status = SUCCESSFUL, throwable = null]]
----
