This means that the user will see the estimation of full benchmark
running time when it is available, unlike now when it often only
ends up flushed after the benchmark is fully finished.
This means that the user will almost immediately see the start
of table like this
```
benchmark name samples iterations estimated
mean low mean high mean
std dev low std dev high std dev
-------------------------------------------------------------------------------
Fill vector generated 100 54 3.0834 ms
```
This presents significant improvement in user experience especially
for long running benchmarks.
The CompactReporter changes save 21 (430764 -> 430743) allocations
when running the SelfTest binary in default configuration. They
save about 500 allocations when running the binary with `-s`.
* Add new SKIP macro for skipping tests at runtime
This adds a new `SKIP` macro for dynamically skipping tests at runtime.
The "skipped" status of a test case is treated as a first-class citizen,
like "succeeded" or "failed", and is reported with a new color on the
console.
* Don't show "skipped assertions" in console/compact reporters
Also extend skip tests to cover a few more use cases.
* Return exit code 4 if all test cases are skipped
* Use LightGrey for the skip colour
This isn't great, but is better than the deep blue that was borderline
invisible on dark backgrounds. The fix is to redo the colouring
a bit, including introducing light-blue that is actually visible.
* Add support for explicit skips in all reporters
* --allow-running-no-tests also allows all tests to be skipped
* Add docs for SKIP macro, deprecate IEventListener::skipTest
Co-authored-by: Martin Hořeňovský <martin.horenovsky@gmail.com>
This changes the compact reporter's summary of test run totals to use
the same format as the console reporter. This means that while output is
no longer on a single line (two instead), it now includes totals for
`failedButOk` test cases and assertions, which were previously missing.
Not all reporters use a format that supports this, so TeamCity
and Automake reporters still do not report it. The console
reporter now reports it even on successful runs, where before
it only reported the rng seed in the header, which was showed
either for failed run, or for run with `-s`.
CLoses#2065
This ended up being a surprisingly large refactoring, motivated
by removing a `const_cast` from `Config`'s handling of reporter
streams, forced by previous commit.
Forcing it to be engaged explicitly, either via `op<<`, or by
`ColourGuard::engage`, fixes an issue with multiple `ColourGuard`s
being constructed in a single expression. Because the construction
of the `ColourGuard` instances can happen in arbitrary order,
colours would be applied in arbitrary order too. However, a chain
of `op<<`s has strict call orders, fixing this issue.
* POSIX colour impl is now compiled for all platforms.
* Deciding whether a colour impl should be picked is now stream
dependent, and thus incompatible implementations can be removed
immediately, rather than checking when the colour is being used.
This opens path to per-reporter colour output customization,
and fixes multiple issues with the old colour implementation.
Under the old implementation, using Win32-backed colouring
would always change the colour used by the console, even if the
actual output was written elsewhere, such as a file passed by
the `--out` flag. This will no longer happen, as the reporter's
colour impl will check that the reporter's stream is pointed
to console before trying to change the colours.
POSIX/ANSI colour implementation suffered a similar-ish issue,
in that it only wrote the colour escape codes into the default
output stream, even if the reporter asking for colouring was
actually writing to a completely different output stream.
This will become useful when reworking colour support, because
Win32 colour support requires checking whether the output is
stdout, which is done through the `IStream` wrapper.
The problem came from the console reporter trying to provide a
fancy linebreaking (primarily for things like `SCENARIO` or the
BDD macros), so that new lines start with extra indentation if
the text being line broken starts as "{text}: ".
The console reporter did not properly take into account cases
where the ": " part would already be in a later line, in which
case it would ask for non-sensical level of indentation (larger
than single line length).
We fixed this by also enforcing that the special indentation case
only triggers if the ": " is found early enough in the line, so
that we also avoid degenerate cases like this:
```
blablabla: F
a
n
c
y
.
.
.
```
Fixes#2309
They now take `StringRef` as the argument, and are virtual only
in the basic interface.
Also cleaned out the various reporters and their overrides
of these members which were often empty or delegating up.
In some places the `std::flush` was not added, as it was sufficiently
obvious that the flush semantics are not intended. There are likely
other places where the flush semantics aren't intended, but that
is a cleanup for later.
Due to also adding a new TU, there is no improvement to the
compilation times of the static library, but it improves the
compilation times of consumer's reporter TUs.
* Added some missing `noexcept`s on custom destructors.
* Fixed `std::move` being called on a const-reference.
* Initialized `ScopedMessage::m_moved` in class definition, instead
of doing so in constructors explicitly.
* Turned some `enum`s into `enum class`es.
* Initialized `StreamingReporterBase::currentTestCaseInfo` in class
definition.
* Some cleanups in SelfTest code.
The includes were an artifact of the old design where the built-in
reporter TUs used autoregistration to add the reporters to the
factory. Because the current design is to add them explicitly in
the central reporter factory TU, the includes are useless.
Now that it has its own header, various reporter TUs that want to
format text do not have to also include Clara. Together with
outlining implementations from a header into a separate TU, this
has noticeably improved the compilation times of the testing impl.
As part of this split, I also implemented some improvements to the
TextFlow code in comparison to the upstream code. These are:
* Replaced the `Spacer` type with a free function that constructs
special `Column` that does the same thing.
* Generic performance improvements, such as eliminating needless
allocations, reserving space in needed allocations, and using smarter
algorithms in some places.
* Because `Column` only ever stored 1 string in its vector, it now
holds the string directly instead.
This means that code that uses it no longer has to include all of
catch_config.hpp, which seems to provide significant reduction in
size of unoptimized compilation of the final static library.
A test runner already has a --durations option to print durations.
However, this isn't entirely satisfactory.
When there are many tests, this produces output spam which makes it hard
to find the test failure output. Nevertheless, it is helpful to be
informed of tests which are unusually slow.
Therefore, introduce a new option --min-duration that causes all
durations above a certain threshold to be printed. This allows slow
tests to be visible without mentioning every test.
This is both a really big and a really small commit. It is small in
that it only contains renaming, moving and modification of include
directives caused by this.
It is really big in the obvious way of touching something like 200
files.
The new rules for naming files is simple: headers use the `.hpp`
extension. The rules for physical file layout is still kinda in
progress, but the basics are also simple:
* Significant parts of functionality get their own subfolder
* Benchmarking is in `catch2/benchmark`
* Matchers are in `catch2/matchers`
* Generators are in `catch2/generators`
* Reporters are in `catch2/reporters`
* Baseline testing facilities are in `catch2/`
* Various top level folders also contain `internal` subfolder,
with files that users probably do not want to include directly,
at least not until they have to write something like their own
reporter.
* The exact files in these subfolders is likely to change later
on
Note that while some includes were cleaned up in this commit, it
is only the low hanging fruit and further cleanup using automatic
tooling will happen later.
Also note that various include guards, copyright notices and file
headers will also be standardized later, rather than in this commit.
Now that Catch2 is a proper library, we can always build the full
library (comparatively minor slowdown) and the user can avoid
including benchmarking headers to avoid the compilation slowdown.
