Update documentation for --order

Catch assumes std::uncaught_exceptions is available whenever C++17 is
available, but for macOS versions older than 10.12 this is not the case.

Instead of checking the C++ version, use a macro to check whether the
feature is available.

docs/cmake-integration.md

@@ -6,6 +6,7 @@
 [Automatic test registration](#automatic-test-registration)<br>
 [CMake project options](#cmake-project-options)<br>
 [Installing Catch2 from git repository](#installing-catch2-from-git-repository)<br>
+[Installing Catch2 from vcpkg](#installing-catch2-from-vcpkg)<br>
 
 Because we use CMake to build Catch2, we also provide a couple of
 integration points for our users.
@@ -220,6 +221,19 @@ when configuring the build, and then modify your calls to
 [find_package](https://cmake.org/cmake/help/latest/command/find_package.html)
 accordingly.
 
+## Installing Catch2 from vcpkg
+
+Alternatively, you can build and install Catch2 using [vcpkg](https://github.com/microsoft/vcpkg/) dependency manager:
+```
+git clone https://github.com/Microsoft/vcpkg.git
+cd vcpkg
+./bootstrap-vcpkg.sh
+./vcpkg integrate install
+./vcpkg install catch2
+```
+
+The catch2 port in vcpkg is kept up to date by microsoft team members and community contributors.
+If the version is out of date, please [create an issue or pull request](https://github.com/Microsoft/vcpkg) on the vcpkg repository.
 
 ---
 

docs/command-line.md

@@ -243,15 +243,23 @@ This option lists all available tests in a non-indented form, one on each line.
 
 Test cases are ordered one of three ways:
 
-
 ### decl
-Declaration order (this is the default order if no --order argument is provided). The order the tests were originally declared in. Note that ordering between files is not guaranteed and is implementation dependent.
+Declaration order (this is the default order if no --order argument is provided).
+Tests in the same TU are sorted using their declaration orders, different
+TUs are in an implementation (linking) dependent order.
+
 
 ### lex
-Lexicographically sorted. Tests are sorted, alpha-numerically, by name.
+Lexicographic order. Tests are sorted by their name, their tags are ignored.
+
 
 ### rand
-Randomly sorted. Test names are sorted using ```std::random_shuffle()```. By default the random number generator is seeded with 0 - and so the order is repeatable. To control the random seed see <a href="#rng-seed">rng-seed</a>.
+
+Randomly sorted. The order is dependent on Catch2's random seed (see
+[`--rng-seed`](#rng-seed)), and is subset invariant. What this means
+is that as long as the random seed is fixed, running only some tests
+(e.g. via tag) does not change their relative order.
+
 
 <a id="rng-seed"></a>
 ## Specify a seed for the Random Number Generator

docs/opensource-users.md

@@ -20,18 +20,21 @@ Listing a project here does not imply endorsement and the plan is to keep these
 ### [ApprovalTests.cpp](https://github.com/approvals/ApprovalTests.cpp)
 C++11 implementation of Approval Tests, for quick, convenient testing of legacy code.
 
+### [args](https://github.com/Taywee/args)
+A simple header-only C++ argument parser library.
+
 ### [Azmq](https://github.com/zeromq/azmq)
 Boost Asio style bindings for ZeroMQ.
 
 ### [Cataclysm: Dark Days Ahead](https://github.com/CleverRaven/Cataclysm-DDA)
 Post-apocalyptic survival RPG.
 
-### [ChakraCore](https://github.com/Microsoft/ChakraCore)
-The core part of the Chakra JavaScript engine that powers Microsoft Edge.
-
 ### [ChaiScript](https://github.com/ChaiScript/ChaiScript)
 A, header-only, embedded scripting language designed from the ground up to directly target C++ and take advantage of modern C++ development techniques.
 
+### [ChakraCore](https://github.com/Microsoft/ChakraCore)
+The core part of the Chakra JavaScript engine that powers Microsoft Edge.
+
 ### [Clara](https://github.com/philsquared/Clara)
 A, single-header-only, type-safe, command line parser - which also prints formatted usage strings.
 
@@ -65,9 +68,6 @@ A small C++ library wrapper for the native C ODBC API.
 ### [Nonius](https://github.com/libnonius/nonius)
 A header-only framework for benchmarking small snippets of C++ code.
 
-### [SOCI](https://github.com/SOCI/soci)
-The C++ Database Access Library.
-
 ### [polymorphic_value](https://github.com/jbcoe/polymorphic_value)
 A polymorphic value-type for C++.
 
@@ -77,18 +77,21 @@ A C++ client library for Consul. Consul is a distributed tool for discovering an
 ### [Reactive-Extensions/ RxCpp](https://github.com/Reactive-Extensions/RxCpp)
 A library of algorithms for values-distributed-in-time.
 
-### [thor](https://github.com/xorz57/thor)
-Wrapper Library for CUDA.
+### [SOCI](https://github.com/SOCI/soci)
+The C++ Database Access Library.
 
 ### [TextFlowCpp](https://github.com/philsquared/textflowcpp)
 A small, single-header-only, library for wrapping and composing columns of text.
 
+### [thor](https://github.com/xorz57/thor)
+Wrapper Library for CUDA.
+
+### [toml++](https://github.com/marzer/tomlplusplus)
+A header-only TOML parser and serializer for modern C++.
+
 ### [Trompeloeil](https://github.com/rollbear/trompeloeil)
 A thread-safe header-only mocking framework for C++14.
 
-### [args](https://github.com/Taywee/args)
-A simple header-only C++ argument parser library.
-
 ## Applications & Tools
 
 ### [ArangoDB](https://github.com/arangodb/arangodb)
@@ -103,6 +106,9 @@ MAME originally stood for Multiple Arcade Machine Emulator.
 ### [Newsbeuter](https://github.com/akrennmair/newsbeuter)
 Newsbeuter is an open-source RSS/Atom feed reader for text terminals.
 
+### [PopHead](https://github.com/SPC-Some-Polish-Coders/PopHead)
+A 2D, Zombie, RPG game which is being made on our own engine.
+
 ### [raspigcd](https://github.com/pantadeusz/raspigcd)
 Low level CLI app and library for execution of GCODE on Raspberry Pi without any additional microcontrolers (just RPi + Stepsticks).
 
@@ -112,9 +118,6 @@ SpECTRE is a code for multi-scale, multi-physics problems in astrophysics and gr
 ### [Standardese](https://github.com/foonathan/standardese)
 Standardese aims to be a nextgen Doxygen.
 
-### [PopHead](https://github.com/SPC-Some-Polish-Coders/PopHead)
-A 2D, Zombie, RPG game which is being made on our own engine.
-
 ---
 
 [Home](Readme.md#top)

extras/ParseAndAddCatchTests.cmake

@@ -144,7 +144,7 @@ function(ParseAndAddCatchTests_ParseFile SourceFile TestTarget)
         if("${TestType}" STREQUAL "SCENARIO")
             set(Name "Scenario: ${Name}")
         endif()
-        if(PARSE_CATCH_TESTS_ADD_FIXTURE_IN_TEST_NAME AND TestFixture)
+        if(PARSE_CATCH_TESTS_ADD_FIXTURE_IN_TEST_NAME AND "${TestType}" MATCHES "(CATCH_)?TEST_CASE_METHOD" AND TestFixture )
             set(CTestName "${TestFixture}:${Name}")
         else()
             set(CTestName "${Name}")

src/catch2/internal/catch_compiler_capabilities.hpp

@@ -39,7 +39,7 @@
 
 #endif
 
-#if defined(CATCH_CPP17_OR_GREATER)
+#if defined(__cpp_lib_uncaught_exceptions)
 #  define CATCH_INTERNAL_CONFIG_CPP17_UNCAUGHT_EXCEPTIONS
 #endif
 
@@ -66,7 +66,20 @@
 #    define CATCH_INTERNAL_START_WARNINGS_SUPPRESSION _Pragma( "clang diagnostic push" )
 #    define CATCH_INTERNAL_STOP_WARNINGS_SUPPRESSION  _Pragma( "clang diagnostic pop" )
 
-#    define CATCH_INTERNAL_IGNORE_BUT_WARN(...) (void)__builtin_constant_p(__VA_ARGS__)
+// As of this writing, IBM XL's implementation of __builtin_constant_p has a bug
+// which results in calls to destructors being emitted for each temporary,
+// without a matching initialization. In practice, this can result in something
+// like `std::string::~string` being called on an uninitialized value.
+//
+// For example, this code will likely segfault under IBM XL:
+// ```
+// REQUIRE(std::string("12") + "34" == "1234")
+// ```
+//
+// Therefore, `CATCH_INTERNAL_IGNORE_BUT_WARN` is not implemented.
+#  if !defined(__ibmxl__)
+#    define CATCH_INTERNAL_IGNORE_BUT_WARN(...) (void)__builtin_constant_p(__VA_ARGS__) /* NOLINT(cppcoreguidelines-pro-type-vararg) */
+#  endif
 
 
 #    define CATCH_INTERNAL_SUPPRESS_GLOBALS_WARNINGS \

src/catch2/internal/catch_debugger.cpp

@@ -14,8 +14,7 @@
 
 #if defined(CATCH_PLATFORM_MAC) || defined(CATCH_PLATFORM_IPHONE)
 
-#  include <assert.h>
-#  include <stdbool.h>
+#  include <cassert>
 #  include <sys/types.h>
 #  include <unistd.h>
 #  include <cstddef>

src/catch2/internal/catch_test_case_registry_impl.cpp

@@ -16,27 +16,70 @@
 #include <catch2/catch_test_case_info.hpp>
 #include <catch2/catch_test_spec.hpp>
 
+#include <algorithm>
+#include <iterator>
+#include <random>
 #include <sstream>
 
 namespace Catch {
 
-    std::vector<TestCaseHandle> sortTests( IConfig const& config, std::vector<TestCaseHandle> const& unsortedTestCases ) {
-
-        std::vector<TestCaseHandle> sorted = unsortedTestCases;
-
-        switch( config.runOrder() ) {
-            case RunTests::InLexicographicalOrder:
-                std::sort( sorted.begin(), sorted.end() );
-                break;
-            case RunTests::InRandomOrder:
-                seedRng( config );
-                std::shuffle( sorted.begin(), sorted.end(), rng() );
-                break;
-            case RunTests::InDeclarationOrder:
-                // already in declaration order
-                break;
+namespace {
+    struct HashTest {
+        explicit HashTest(SimplePcg32& rng) {
+            basis = rng();
+            basis <<= 32;
+            basis |= rng();
         }
-        return sorted;
+
+        uint64_t basis;
+
+        uint64_t operator()(TestCaseInfo const& t) const {
+            // Modified FNV-1a hash
+            static constexpr uint64_t prime = 1099511628211;
+            uint64_t hash = basis;
+            for (const char c : t.name) {
+                hash ^= c;
+                hash *= prime;
+            }
+            return hash;
+        }
+    };
+} // end anonymous namespace
+
+    std::vector<TestCaseHandle> sortTests( IConfig const& config, std::vector<TestCaseHandle> const& unsortedTestCases ) {
+        switch (config.runOrder()) {
+        case RunTests::InDeclarationOrder:
+            return unsortedTestCases;
+
+        case RunTests::InLexicographicalOrder: {
+            std::vector<TestCaseHandle> sorted = unsortedTestCases;
+            std::sort(sorted.begin(), sorted.end());
+            return sorted;
+        }
+        case RunTests::InRandomOrder: {
+            seedRng(config);
+            HashTest h(rng());
+            std::vector<std::pair<uint64_t, TestCaseHandle>> indexed_tests;
+            indexed_tests.reserve(unsortedTestCases.size());
+
+            for (auto const& handle : unsortedTestCases) {
+                indexed_tests.emplace_back(h(handle.getTestCaseInfo()), handle);
+            }
+
+            std::sort(indexed_tests.begin(), indexed_tests.end());
+
+            std::vector<TestCaseHandle> randomized;
+            randomized.reserve(indexed_tests.size());
+
+            for (auto const& indexed : indexed_tests) {
+                randomized.push_back(indexed.second);
+            }
+
+            return randomized;
+        }
+        }
+
+        CATCH_INTERNAL_ERROR("Unknown test order value!");
     }
 
     bool isThrowSafe( TestCaseHandle const& testCase, IConfig const& config ) {

src/catch2/internal/catch_test_spec_parser.cpp

@@ -170,6 +170,7 @@ namespace Catch {
          m_pos = m_arg.size();
          m_substring.clear();
          m_patternName.clear();
+         m_realPatternPos = 0;
          return false;
       }
       endMode();
@@ -188,6 +189,7 @@ namespace Catch {
         }
 
         m_patternName.clear();
+        m_realPatternPos = 0;
 
         return token;
     }

src/catch2/reporters/catch_reporter_console.cpp

@@ -363,7 +363,7 @@ ConsoleReporter::ConsoleReporter(ReporterConfig const& config)
         else
         {
             return{
-                { "benchmark name", CATCH_CONFIG_CONSOLE_WIDTH - 32, ColumnInfo::Left },
+                { "benchmark name", CATCH_CONFIG_CONSOLE_WIDTH - 43, ColumnInfo::Left },
                 { "samples      mean       std dev", 14, ColumnInfo::Right },
                 { "iterations   low mean   low std dev", 14, ColumnInfo::Right },
                 { "estimated    high mean  high std dev", 14, ColumnInfo::Right }

tests/CMakeLists.txt

@@ -198,6 +198,9 @@ set_tests_properties(TagAlias PROPERTIES
   FAIL_REGULAR_EXPRESSION "0 matching test cases"
 )
 
+add_test(NAME RandomTestOrdering COMMAND ${PYTHON_EXECUTABLE}
+  ${CATCH_DIR}/tests/TestScripts/testRandomOrder.py $<TARGET_FILE:SelfTest>)
+
 if (CATCH_USE_VALGRIND)
     add_test(NAME ValgrindRunTests COMMAND valgrind --leak-check=full --error-exitcode=1 $<TARGET_FILE:SelfTest>)
     add_test(NAME ValgrindListTests COMMAND valgrind --leak-check=full --error-exitcode=1 $<TARGET_FILE:SelfTest> --list-tests --verbosity high)

tests/SelfTest/Baselines/automake.sw.approved.txt

@@ -13,6 +13,7 @@ This would not be caught previously
 Nor would this
 :test-result: FAIL #1514: stderr/stdout is not captured in tests aborted by an exception
 :test-result: PASS #1548
+:test-result: PASS #1905 -- test spec parser properly clears internal state between compound tests
 :test-result: XFAIL #748 - captures with unexpected exceptions
 :test-result: PASS #809
 :test-result: PASS #833

tests/SelfTest/Baselines/compact.sw.approved.txt

@@ -24,6 +24,9 @@ This would not be caught previously
 Nor would this
 Tricky.tests.cpp:<line number>: failed: explicitly with 1 message: '1514'
 Compilation.tests.cpp:<line number>: passed: std::is_same<TypeList<int>, TypeList<int>>::value for: true
+CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec . char")) for: true
+CmdLine.tests.cpp:<line number>: passed: spec.matches(*fakeTestCase("spec , char")) for: true
+CmdLine.tests.cpp:<line number>: passed: !(spec.matches(*fakeTestCase(R"(spec \, char)"))) for: !false
 Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42' with 1 message: 'expected exception'
 Exception.tests.cpp:<line number>: failed: unexpected exception with message: 'answer := 42'; expression was: thisThrows() with 1 message: 'expected exception'
 Exception.tests.cpp:<line number>: passed: thisThrows() with 1 message: 'answer := 42'

tests/SelfTest/Baselines/console.std.approved.txt

@@ -1380,6 +1380,6 @@ due to unexpected exception with message:
   Why would you throw a std::string?
 
 ===============================================================================
-test cases:  332 |  258 passed |  70 failed |  4 failed as expected
-assertions: 1873 | 1721 passed | 131 failed | 21 failed as expected
+test cases:  333 |  259 passed |  70 failed |  4 failed as expected
+assertions: 1876 | 1724 passed | 131 failed | 21 failed as expected
 

tests/SelfTest/Baselines/console.sw.approved.txt

@@ -197,6 +197,27 @@ Compilation.tests.cpp:<line number>: PASSED:
 with expansion:
   true
 
+-------------------------------------------------------------------------------
+#1905 -- test spec parser properly clears internal state between compound tests
+-------------------------------------------------------------------------------
+CmdLine.tests.cpp:<line number>
+...............................................................................
+
+CmdLine.tests.cpp:<line number>: PASSED:
+  REQUIRE( spec.matches(*fakeTestCase("spec . char")) )
+with expansion:
+  true
+
+CmdLine.tests.cpp:<line number>: PASSED:
+  REQUIRE( spec.matches(*fakeTestCase("spec , char")) )
+with expansion:
+  true
+
+CmdLine.tests.cpp:<line number>: PASSED:
+  REQUIRE_FALSE( spec.matches(*fakeTestCase(R"(spec \, char)")) )
+with expansion:
+  !false
+
 -------------------------------------------------------------------------------
 #748 - captures with unexpected exceptions
   outside assertions
@@ -14658,6 +14679,6 @@ Misc.tests.cpp:<line number>
 Misc.tests.cpp:<line number>: PASSED:
 
 ===============================================================================
-test cases:  332 |  242 passed |  86 failed |  4 failed as expected
-assertions: 1890 | 1721 passed | 148 failed | 21 failed as expected
+test cases:  333 |  243 passed |  86 failed |  4 failed as expected
+assertions: 1893 | 1724 passed | 148 failed | 21 failed as expected
 

tests/SelfTest/Baselines/console.swa4.approved.txt

@@ -197,6 +197,27 @@ Compilation.tests.cpp:<line number>: PASSED:
 with expansion:
   true
 
+-------------------------------------------------------------------------------
+#1905 -- test spec parser properly clears internal state between compound tests
+-------------------------------------------------------------------------------
+CmdLine.tests.cpp:<line number>
+...............................................................................
+
+CmdLine.tests.cpp:<line number>: PASSED:
+  REQUIRE( spec.matches(*fakeTestCase("spec . char")) )
+with expansion:
+  true
+
+CmdLine.tests.cpp:<line number>: PASSED:
+  REQUIRE( spec.matches(*fakeTestCase("spec , char")) )
+with expansion:
+  true
+
+CmdLine.tests.cpp:<line number>: PASSED:
+  REQUIRE_FALSE( spec.matches(*fakeTestCase(R"(spec \, char)")) )
+with expansion:
+  !false
+
 -------------------------------------------------------------------------------
 #748 - captures with unexpected exceptions
   outside assertions
@@ -377,6 +398,6 @@ Condition.tests.cpp:<line number>: FAILED:
   CHECK( true != true )
 
 ===============================================================================
-test cases: 20 | 15 passed | 3 failed | 2 failed as expected
-assertions: 43 | 36 passed | 4 failed | 3 failed as expected
+test cases: 21 | 16 passed | 3 failed | 2 failed as expected
+assertions: 46 | 39 passed | 4 failed | 3 failed as expected
 

tests/SelfTest/Baselines/junit.sw.approved.txt

@@ -1,7 +1,7 @@
 <?xml version="1.0" encoding="UTF-8"?>
 <testsuitesloose text artifact
 >
-  <testsuite name="<exe-name>" errors="17" failures="132" tests="1891" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
+  <testsuite name="<exe-name>" errors="17" failures="132" tests="1894" hostname="tbd" time="{duration}" timestamp="{iso8601-timestamp}">
     <properties>
       <property name="filters" value="~[!nonportable]~[!benchmark]~[approvals] *"/>
       <property name="random-seed" value="1"/>
@@ -31,6 +31,7 @@ Nor would this
       </system-err>
     </testcase>
     <testcase classname="<exe-name>.global" name="#1548" time="{duration}"/>
+    <testcase classname="<exe-name>.global" name="#1905 -- test spec parser properly clears internal state between compound tests" time="{duration}"/>
     <testcase classname="<exe-name>.global" name="#748 - captures with unexpected exceptions/outside assertions" time="{duration}">
       <error type="TEST_CASE">
 FAILED:

tests/SelfTest/Baselines/sonarqube.sw.approved.txt

@@ -2,6 +2,7 @@
 <testExecutions version="1"loose text artifact
 >
   <file path="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp">
+    <testCase name="#1905 -- test spec parser properly clears internal state between compound tests" duration="{duration}"/>
     <testCase name="Parse test names and tags/Empty test spec should have no filters" duration="{duration}"/>
     <testCase name="Parse test names and tags/Test spec from empty string should have no filters" duration="{duration}"/>
     <testCase name="Parse test names and tags/Test spec from just a comma should have no filters" duration="{duration}"/>

tests/SelfTest/Baselines/tap.sw.approved.txt

@@ -46,6 +46,12 @@ Nor would this
 not ok {test-number} - explicitly with 1 message: '1514'
 # #1548
 ok {test-number} - std::is_same<TypeList<int>, TypeList<int>>::value for: true
+# #1905 -- test spec parser properly clears internal state between compound tests
+ok {test-number} - spec.matches(*fakeTestCase("spec . char")) for: true
+# #1905 -- test spec parser properly clears internal state between compound tests
+ok {test-number} - spec.matches(*fakeTestCase("spec , char")) for: true
+# #1905 -- test spec parser properly clears internal state between compound tests
+ok {test-number} - !(spec.matches(*fakeTestCase(R"(spec \, char)"))) for: !false
 # #748 - captures with unexpected exceptions
 not ok {test-number} - unexpected exception with message: 'answer := 42' with 1 message: 'expected exception'
 # #748 - captures with unexpected exceptions
@@ -3772,5 +3778,5 @@ ok {test-number} - q3 == 23. for: 23.0 == 23.0
 ok {test-number} -
 # xmlentitycheck
 ok {test-number} -
-1..1882
+1..1885
 

tests/SelfTest/Baselines/teamcity.sw.approved.txt

@@ -28,6 +28,8 @@ Tricky.tests.cpp:<line number>|nexplicit failure with message:|n  "1514"']
 ##teamcity[testFinished name='#1514: stderr/stdout is not captured in tests aborted by an exception' duration="{duration}"]
 ##teamcity[testStarted name='#1548']
 ##teamcity[testFinished name='#1548' duration="{duration}"]
+##teamcity[testStarted name='#1905 -- test spec parser properly clears internal state between compound tests']
+##teamcity[testFinished name='#1905 -- test spec parser properly clears internal state between compound tests' duration="{duration}"]
 ##teamcity[testStarted name='#748 - captures with unexpected exceptions']
 Exception.tests.cpp:<line number>|nunexpected exception with messages:|n  "answer := 42"|n  "expected exception"- failure ignore as test marked as |'ok to fail|'|n']
 Exception.tests.cpp:<line number>|nunexpected exception with messages:|n  "answer := 42"|n  "expected exception"|n  REQUIRE_NOTHROW( thisThrows() )|nwith expansion:|n  thisThrows()|n- failure ignore as test marked as |'ok to fail|'|n']

tests/SelfTest/Baselines/xml.sw.approved.txt

@@ -202,6 +202,33 @@ Nor would this
       </Expression>
       <OverallResult success="true"/>
     </TestCase>
+    <TestCase name="#1905 -- test spec parser properly clears internal state between compound tests" tags="[command-line][test-spec]" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+      <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+        <Original>
+          spec.matches(*fakeTestCase("spec . char"))
+        </Original>
+        <Expanded>
+          true
+        </Expanded>
+      </Expression>
+      <Expression success="true" type="REQUIRE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+        <Original>
+          spec.matches(*fakeTestCase("spec , char"))
+        </Original>
+        <Expanded>
+          true
+        </Expanded>
+      </Expression>
+      <Expression success="true" type="REQUIRE_FALSE" filename="tests/<exe-name>/IntrospectiveTests/CmdLine.tests.cpp" >
+        <Original>
+          !(spec.matches(*fakeTestCase(R"(spec \, char)")))
+        </Original>
+        <Expanded>
+          !false
+        </Expanded>
+      </Expression>
+      <OverallResult success="true"/>
+    </TestCase>
     <TestCase name="#748 - captures with unexpected exceptions" tags="[!shouldfail][!throws][.][failing]" filename="tests/<exe-name>/UsageTests/Exception.tests.cpp" >
       <Section name="outside assertions" filename="tests/<exe-name>/UsageTests/Exception.tests.cpp" >
         <Info>
@@ -17568,7 +17595,7 @@ loose text artifact
       </Section>
       <OverallResult success="true"/>
     </TestCase>
-    <OverallResults successes="1721" failures="149" expectedFailures="21"/>
+    <OverallResults successes="1724" failures="149" expectedFailures="21"/>
   </Group>
-  <OverallResults successes="1721" failures="148" expectedFailures="21"/>
+  <OverallResults successes="1724" failures="148" expectedFailures="21"/>
 </Catch>

tests/SelfTest/IntrospectiveTests/CmdLine.tests.cpp

@@ -296,6 +296,17 @@ TEST_CASE( "Parse test names and tags", "[command-line][test-spec]" ) {
     }
 }
 
+TEST_CASE("#1905 -- test spec parser properly clears internal state between compound tests", "[command-line][test-spec]") {
+    using Catch::parseTestSpec;
+    using Catch::TestSpec;
+    // We ask for one of 2 different tests and the latter one of them has a , in name that needs escaping
+    TestSpec spec = parseTestSpec(R"("spec . char","spec \, char")");
+
+    REQUIRE(spec.matches(*fakeTestCase("spec . char")));
+    REQUIRE(spec.matches(*fakeTestCase("spec , char")));
+    REQUIRE_FALSE(spec.matches(*fakeTestCase(R"(spec \, char)")));
+}
+
 TEST_CASE( "Process can be configured on command line", "[config][command-line]" ) {
 
     using namespace Catch::Matchers;

tests/TestScripts/testRandomOrder.py

@@ -0,0 +1,62 @@
+#!/usr/bin/env python3
+
+"""
+This test script verifies that the random ordering of tests inside
+Catch2 is invariant in regards to subsetting. This is done by running
+the binary 3 times, once with all tests selected, and twice with smaller
+subsets of tests selected, and verifying that the selected tests are in
+the same relative order.
+"""
+
+import subprocess
+import sys
+import random
+import xml.etree.ElementTree as ET
+
+def list_tests(self_test_exe, tags, rng_seed):
+    cmd = [self_test_exe, '--reporter', 'xml', '--list-tests', '--order', 'rand',
+            '--rng-seed', str(rng_seed)]
+    tags_arg = ','.join('[{}]'.format(t) for t in tags)
+    if tags_arg:
+        cmd.append(tags_arg + '~[.]')
+    process = subprocess.Popen(
+            cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
+    stdout, stderr = process.communicate()
+    if stderr:
+        raise RuntimeError("Unexpected error output:\n" + process.stderr)
+
+    root = ET.fromstring(stdout)
+    result = [elem.text for elem in root.findall('./TestCase/Name')]
+
+    if len(result) < 2:
+        raise RuntimeError("Unexpectedly few tests listed (got {})".format(
+            len(result)))
+    return result
+
+def check_is_sublist_of(shorter, longer):
+    assert len(shorter) < len(longer)
+    assert len(set(longer)) == len(longer)
+
+    indexes_in_longer = {s: i for i, s in enumerate(longer)}
+    for s1, s2 in zip(shorter, shorter[1:]):
+        assert indexes_in_longer[s1] < indexes_in_longer[s2], (
+                '{} comes before {} in longer list.\n'
+                'Longer: {}\nShorter: {}'.format(s2, s1, longer, shorter))
+
+def main():
+    self_test_exe, = sys.argv[1:]
+
+    # We want a random seed for the test, but want to avoid 0,
+    # because it has special meaning
+    seed = random.randint(1, 2 ** 32 - 1)
+
+    list_one_tag = list_tests(self_test_exe, ['generators'], seed)
+    list_two_tags = list_tests(self_test_exe, ['generators', 'matchers'], seed)
+    list_all = list_tests(self_test_exe, [], seed)
+
+    # First, verify that restricting to a subset yields the same order
+    check_is_sublist_of(list_two_tags, list_all)
+    check_is_sublist_of(list_one_tag, list_two_tags)
+
+if __name__ == '__main__':
+    sys.exit(main())