build-aux/docs/testing.md
common.mk includes a built-in test harness that allows you to plug
in your own tests, and have them aggregated and summarized, like:
$ make check
...
PASS: go-test 5 - github.com/datawire/apro/cmd/amb-sidecar.TestAppNoToken
PASS: go-test 6 - github.com/datawire/apro/cmd/amb-sidecar.TestAppBadToken
PASS: go-test 7 - github.com/datawire/apro/cmd/amb-sidecar.TestAppBadCookie
PASS: go-test 8 - github.com/datawire/apro/cmd/amb-sidecar.TestAppCallback
PASS: go-test 9 - github.com/datawire/apro/cmd/amb-sidecar.TestAppCallbackNoCode
PASS: tests/local/apictl.tap.gen 1 - check_version
============================================================================
test-suite summary
============================================================================
# TOTAL: 10
# SKIP: 0
# PASS: 10
# XFAIL: 0
# FAIL: 0
# XPASS: 0
# ERROR: 0
============================================================================
make[1]: Leaving directory '/home/lukeshu/src/apro'
(If you were viewing that in a terminal, it would also be pretty and colorized.)
Each test-case can have one of 6 results:
pass, fail, skip: You can guess what these mean.xfail ("expected fail"): The test failed, but it was expected to
fail. Perhaps you wrote a test for a feature for before
implementing the feature. Perhaps you wrote a test that captures a
bug report, but haven't written the fix yet. Makes
Test-Driven-Development possible.xpass ("unexpected pass"): The test passed, but it was expected
to xfail. This either means you did a poor job implementing the
test, and it doesn't really check what you think it checks, or it
means you've implemented the fix, but forgot to replace xfail
with fail in the test.error: It isn't that the test decided that there's bug in the
code-under-test, it's that the test itself encountered an error.You can use any test framework or language you like, as long as it can
emit TAP, the Test Anything Protocol (or something that can be
translated to TAP). Both TAP version 12 and version 13 are supported.
not ok # TODO is "xfail", while , ok # TODO is "xpass"
Side-Note: pytest-tap emits
ok # TODOfor both xfail and xpass, which is wrong, and I consider to be a bug in pytest-tap.
By default, common.mk knows how to run test cases of 2 types (but
you can easily add more):
*.test GNU Automake-compatible standalone test cases. One file
is one test case. FOO.test must be an executable file. It is
run with no arguments, stdout and stderr are ignored (but logged to
FOO.log); it is the exit code that determines the test result:
It is not possible to xfail or xpass with this type of test.
*.tap.gen TAP-emitting test cases. You may have multiple test
cases per file. FOO.tap.gen must be an executable file. It is
run with no arguments; stdout and stderr are merged, and are taken
to be a TAP stream (both v12 and v13 are supported). The exit code
is ignored.
common.mk does not scan your source directory for tests (but
go-*.mk will scan for go test tests). In your Makefile You must
explicitly tell it about any .test or .tap.gen files that you
would like it to include in make check. For example, if you would
like it to include any .test or .tap.gen files in the ./tests/
directory, you could write:
test-suite.tap: $(patsubst %.test,%.tap,$(wildcard tests/*.test))
test-suite.tap: $(patsubst %.tap.gen,%.tap,$(wildcard tests/*.tap.gen))
To add a new test runner, you just need a command that emits TAP:
tee it to a .tap file, and pipe that to $(tools/tap-driver) stream -n TEST_GROUP_NAME to pretty-print the results as they happen:
# Tell Make how to run the test command, and stream the results to
# `$(tools/tap-driver) stream` to pretty-print the results as they happen.
my-test.tap: my-test.input $(tools/tap-driver) FORCE
@SOME_COMMAND_THAT_EMITS_TAP 2>&1 | tee $@ | $(tools/tap-driver) stream -n my-test
# Tell Make to include 'my-test' in `make check`
test-suite.tap: my-test.tap
For example, to use BATS (Bash Automated Testing System), you would write:
%.tap: %.bats $(tools/tap-driver) FORCE
@bats --tap $< | tee $@ | $(tools/tap-driver) stream -n $<
# Automatically include `./tests/*.bats`
test-suite.tap: $(patsubst %.bats,%.tap,$(wildcard tests/*.bats))
If your test framework of choice doesn't support TAP output, you can
pipe it to a helper program that can translate it. For example, go test doesn't support TAP output, but go test -json output is
parsable, so we pipe that to gotest2tap, which
translates it to TAP.
If you set SHELL = sh -o pipefail in your Makefile (the pros and
cons of which I won't comment on here), you should be sure that if
your test-runner indicates success or failure with an exit code, that
you ignore that exit code:
%.tap: %.bats $(tools/tap-driver) FORCE
@{ bats --tap $< || true; } | tee $@ | $(tools/tap-driver) stream -n $<
It is assumed that all tests depend on make build. To add a
dependency shared by all tests, to declare a dependency that all tests
should depend on, declare it as a dependency of check itself. For
example, common.mk says:
check: lint build
As another example, the Makefile for Ambassador Pro says:
check: $(if $(HAVE_DOCKER),deploy proxy)
To declare a dependency for an individual test is a little trickier,
because you must keep in mind what type of test it is. For .tap.gen
tests, you must declare it both on the .tap file and (if the
dependency is not a .tap) on check itself:
test-suite.tap: tests/cluster/oauth-e2e.tap
check tests/cluster/oauth-e2e.tap: tests/cluster/oauth-e2e/node_modules
If that were a .test test, you would need to declare it on the
.log file instead of .tap:
test-suite.tap: tests/cluster/oauth-e2e.tap
check tests/cluster/oauth-e2e.log: tests/cluster/oauth-e2e/node_modules
If you need one test to depend on another test, write the dependency
using the .tap suffix (not .log). You do not need to write the
depenency for check, since it will already depend on the .tap
through test-suite.tap:
test-suite.tap: foo.tap bar.tap
foo.log: bar.tap