Rework test/run_tests.pl to support selective verbosity and TAP copy

This includes a complete rework of how we use TAP::Harness, by adding
a TAP::Parser subclass that allows additional callbacks to be passed
to perform what we need.  The TAP::Parser callbacks we add are:

    ALL         to print all the TAP output to a file (conditionally)
                to collect all the TAP output to an array (conditionally)
    EOF         to print all the collected TAP output (if there is any)
                if any subtest failed

To get TAP output to file, the environment variable HARNESS_TAP_COPY
must be defined, with a file name as value.  That file will be
overwritten unconditionally.

To get TAP output displayed on failure, the make variable VERBOSE_FAILURE
or VF must be defined with a non-emoty value.

Additionally, the output of test recipe names has been changed to only
display its basename.

Reviewed-by: Matt Caswell <matt@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/9862)
This commit is contained in:
Richard Levitte 2019-09-11 11:05:11 +02:00
parent f3503cb0f6
commit e3d9a6b5f0
4 changed files with 176 additions and 61 deletions

View file

@ -9,6 +9,15 @@
Changes between 1.1.1 and 3.0.0 [xx XXX xxxx]
*) Extended testing to be verbose for failing tests only. The make variables
VERBOSE_FAILURE or VF can be used to enable this:
$ make VF=1 test # Unix
$ mms /macro=(VF=1) test ! OpenVMS
$ nmake VF=1 test # Windows
[Richard Levitte]
*) For built-in EC curves, ensure an EC_GROUP built from the curve name is
used even when parsing explicit parameters, when loading a serialized key
or calling `EC_GROUP_new_from_ecpkparameters()`/