JSON and TEXT fields with binary collations support.

JSON fields are now read as strings instead of blobs.
TEXT fields with binary collations (e.g. utf8_bin or
    the JSON type in MariaDB) are now read as strings instead of blobs.
Added the column_type::json type.
The CI script now loads all data into the databases. There is no
    longer need to rebuild DB containers when test data changes.
This commit is contained in:
Ruben Perez 2023-03-04 10:21:36 +01:00
parent 21d90d4407
commit 643e39f85e
37 changed files with 244 additions and 120 deletions

View File

@ -1,12 +1,12 @@
#
# Copyright (c) 2019-2020 Ruben Perez Hidalgo (rubenperez038 at gmail dot com)
# Copyright (c) 2019-2023 Ruben Perez Hidalgo (rubenperez038 at gmail dot com)
#
# Distributed under the Boost Software License, Version 1.0. (See accompanying
# file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
#
_triggers = { "branch": [ "master", "develop", "drone*", "bugfix/*", "feature/*", "fix/*", "pr/*" ] }
_container_tag = 'a2479d03ed00a86cb7f916c9155e0dbbeee59325'
_container_tag = '65e51d3af7132dcb1001249629c24cc59b934cb6'
def _image(name):

View File

@ -1,5 +1,5 @@
#
# Copyright (c) 2019-2020 Ruben Perez Hidalgo (rubenperez038 at gmail dot com)
# Copyright (c) 2019-2023 Ruben Perez Hidalgo (rubenperez038 at gmail dot com)
#
# Distributed under the Boost Software License, Version 1.0. (See accompanying
# file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)

View File

@ -71,7 +71,6 @@
[def __DECIMAL__ [mysqllink fixed-point-types.html `DECIMAL`]]
[def __NUMERIC__ [mysqllink fixed-point-types.html `NUMERIC`]]
[def __BIT__ [mysqllink bit-type.html `BIT`]]
[def __GEOMETRY__ [mysqllink spatial-type-overview.html `GEOMETRY`]]
[def __CHAR__ [mysqllink char.html `CHAR`]]
[def __VARCHAR__ [mysqllink char.html `VARCHAR`]]
[def __BINARY__ [mysqllink binary-varbinary.html `BINARY`]]
@ -80,6 +79,8 @@
[def __BLOB__ [mysqllink blob.html `BLOB`]]
[def __ENUM__ [mysqllink enum.html `ENUM`]]
[def __SET__ [mysqllink set.html `SET`]]
[def __JSON__ [mysqllink json.html `JSON`]]
[def __GEOMETRY__ [mysqllink spatial-type-overview.html `GEOMETRY`]]
[def __USE__ [mysqllink use.html `USE`]]

View File

@ -231,7 +231,7 @@ Every MySQL type is mapped to a single C++ type. The following table shows these
`std::string` for `field`
]
[
__CHAR__, __VARCHAR__, __TEXT__ (all sizes), __ENUM__, __SET__, __DECIMAL__, __NUMERIC__
__CHAR__, __VARCHAR__, __TEXT__ (all sizes), __ENUM__, __SET__, __DECIMAL__, __NUMERIC__, __JSON__
]
[[refmemunq field_view is_string]]
[[refmemunq field_view as_string]]

View File

@ -68,7 +68,8 @@ This library performs no character set conversion on strings. All strings are pr
as the server sends them. If you've issued a `"SET NAMES <charset-name>"` statement,
strings will be encoded according to `<charset-name>`. For details, see [link mysql.charsets this section].
The type `DECIMAL` is also provided as a string, to avoid losing precision.
The `JSON` type is represented as a string containing the serialized JSON value.
The `DECIMAL` type is also provided as a string, to avoid losing precision.
[endsect]
@ -421,6 +422,18 @@ column shows what [refmem metadata type] would return for a column of that type.
Character string with a fixed set of possible values (many possible).
]
]
[
[__JSON__]
[`string`]
[[reflink string_view] or `std::string`]
[]
[`json` (MySQL) or `text` (MariaDB)]
[
A serialized JSON value of any type.
Note that [refmem metadata type] is different depending on the DB system. MySQL has a dedicated `JSON` type, while in MariaDB `JSON` is an alias for `LONGTEXT`. JSON values are represented as strings by this library in both cases.
]
]
[
[__DECIMAL__/__NUMERIC__]
[`string`]
@ -501,7 +514,7 @@ is interpreted by MySQL, depeding on `v`'s type:
[
[`std::string`, [reflink string_view], `std::string_view`, `const char*`]
[`VARCHAR`]
[`CHAR`, `VARCHAR`, `TEXT` (all sizes), `ENUM`, `SET`, `DECIMAL`, `NUMERIC`]
[`CHAR`, `VARCHAR`, `TEXT` (all sizes), `ENUM`, `SET`, `JSON`, `DECIMAL`, `NUMERIC`]
]
[
[[reflink blob], [reflink blob_view]]

View File

@ -32,8 +32,8 @@ Examples make use of a database named `boost_mysql_examples`.
The server hostname and credentials (username and password) are passed
to the examples via the command line.
If you're using docker, you can use the `ghcr.io/anarthal-containers/mysql8` container,
which already ships with all the required configuration:
If you're using docker, you can use the `ghcr.io/anarthal-containers/mysql8` container
to simplify the process:
[!teletype]
```
@ -43,6 +43,10 @@ which already ships with all the required configuration:
# If you're on a system that does not support UNIX sockets
> docker run -p 3306:3306 -d ghcr.io/anarthal-containers/mysql8
# All the required data can be loaded by running example/db_setup.sql.
# If you're using the above container, the root user has a blank password
> mysql -u root < example/db_setup.sql
```
Please note that this container is just for demonstrative purposes,

View File

@ -37,12 +37,16 @@ You can run the containers as follows:
> docker run -p 3306:3306 -d ghcr.io/anarthal-containers/mysql8
```
If you are using your own database server, you will need to perform the following steps:
You then need to run the following setup files, e.g. by running `mysql -u root < path/to/setup.sql`:
* Run `example/db_setup.sql` and `test/integration/db_setup.sql` as the root user. If you are running a MySQL 8.x server,
run also `test/integration/db_setup_sha256.sql`.
* Install the SSL certificates in `tools/ssl` in your MySQL server and change your config file so that your server uses them.
More information [mysqllink using-encrypted-connections.html here].
* `example/db_setup.sql`
* `test/integration/db_setup.sql`
* If you are running a MySQL 8.x server, run also `test/integration/db_setup_sha256.sql`.
If you are using your own database server, you will need to install the SSL certificates in `tools/ssl`
in your MySQL server and change your config file so that your server uses them.
More information [mysqllink using-encrypted-connections.html here].
Next, define the following environment variables:
@ -50,7 +54,7 @@ Next, define the following environment variables:
If you are using the Docker image as provided in this document, you don't need this.
* If your system does not support UNIX sockets or your socket path is different than MySQL's default (`/var/run/mysqld/mysqld.sock`),
define `BOOST_MYSQL_NO_UNIX_SOCKET_TESTS=1`.
* If you are using MySQL 5.x or MariaDB, define `BOOST_MYSQL_NO_SHA256_TESTS=1`. These servers don't support part of the functionality we test.
* Define `BOOST_MYSQL_TEST_DB` to either `mysql5`, `mysql8` or `mariadb`, depending on the server you're running.
If you are using `b2`, you can build the targets `boost/mysql/example//boost_mysql_all_examples`,
`boost/mysql/test/integration//boost_mysql_integrationtests` and `boost/mysql/test` to build and run the tests.

View File

@ -48,6 +48,7 @@ enum class column_type
blob, ///< `BLOB` types (`TINYBLOB`, `MEDIUMBLOB`, `BLOB` and `LONGBLOB`)
enum_, ///< `ENUM`
set, ///< `SET`
json, ///< `JSON`
geometry, ///< `GEOMETRY`
unknown, ///< None of the known types; maybe a new MySQL type we have no knowledge of.
};
@ -81,6 +82,7 @@ inline std::ostream& operator<<(std::ostream& os, column_type t)
case column_type::blob: return os << "blob";
case column_type::enum_: return os << "enum_";
case column_type::set: return os << "set";
case column_type::json: return os << "json";
case column_type::geometry: return os << "geometry";
default: return os << "<unknown field type>";
}

View File

@ -35,6 +35,7 @@ enum class protocol_field_type : std::uint8_t
year = 0x0d, // YEAR
varchar = 0x0f, // Apparently not sent
bit = 0x10, // BIT
json = 0xf5, // JSON
newdecimal = 0xf6, // DECIMAL
enum_ = 0xf7, // Apparently not sent
set = 0xf8, // Apperently not sent
@ -76,6 +77,9 @@ constexpr std::uint8_t auth_switch_request_header = 0xfe;
constexpr std::uint8_t auth_more_data_header = 0x01;
constexpr string_view fast_auth_complete_challenge = make_string_view("\3");
// The binary collation number, used to distinguish blobs from strings
constexpr std::uint16_t binary_collation = 63;
// Column flags
namespace column_flags {
@ -122,9 +126,8 @@ constexpr std::size_t secs_sz = 2;
constexpr std::size_t date_sz = year_sz + month_sz + day_sz + 2; // delimiters
constexpr std::size_t time_min_sz = hours_min_sz + mins_sz + secs_sz + 2; // delimiters
constexpr std::size_t time_max_sz = time_min_sz + max_decimals +
3; // sign, period, hour extra character
constexpr std::size_t datetime_min_sz = date_sz + time_min_sz + 1; // delimiter
constexpr std::size_t time_max_sz = time_min_sz + max_decimals + 3; // sign, period, hour extra character
constexpr std::size_t datetime_min_sz = date_sz + time_min_sz + 1; // delimiter
constexpr std::size_t datetime_max_sz = datetime_min_sz + max_decimals + 1; // period
constexpr unsigned time_max_hour = 838;

View File

@ -298,7 +298,8 @@ inline boost::mysql::detail::deserialize_errc boost::mysql::detail::deserialize_
case column_type::text:
case column_type::enum_:
case column_type::set:
case column_type::decimal: return deserialize_binary_field_string(ctx, output, buffer_first, false);
case column_type::decimal:
case column_type::json: return deserialize_binary_field_string(ctx, output, buffer_first, false);
// Blobs and anything else
case column_type::binary:
case column_type::varbinary:

View File

@ -294,7 +294,8 @@ inline boost::mysql::detail::deserialize_errc boost::mysql::detail::deserialize_
case column_type::text:
case column_type::enum_:
case column_type::set:
case column_type::decimal: return deserialize_text_value_string(from, output, buffer_first, false);
case column_type::decimal:
case column_type::json: return deserialize_text_value_string(from, output, buffer_first, false);
// Blobs and anything else
case column_type::binary:
case column_type::varbinary:

View File

@ -13,40 +13,39 @@
#include <boost/mysql/metadata.hpp>
#include <boost/mysql/detail/auxiliar/access_fwd.hpp>
#include <boost/mysql/detail/protocol/constants.hpp>
namespace boost {
namespace mysql {
namespace detail {
inline column_type compute_field_type_string(std::uint32_t flags)
inline column_type compute_field_type_string(std::uint32_t flags, std::uint16_t collation)
{
if (flags & column_flags::set)
return column_type::set;
else if (flags & column_flags::enum_)
return column_type::enum_;
else if (flags & column_flags::binary)
else if (collation == binary_collation)
return column_type::binary;
else
return column_type::char_;
}
inline column_type compute_field_type_var_string(std::uint32_t flags)
inline column_type compute_field_type_var_string(std::uint16_t collation)
{
if (flags & column_flags::binary)
return column_type::varbinary;
else
return column_type::varchar;
return collation == binary_collation ? column_type::varbinary : column_type::varchar;
}
inline column_type compute_field_type_blob(std::uint32_t flags)
inline column_type compute_field_type_blob(std::uint16_t collation)
{
if (flags & column_flags::binary)
return column_type::blob;
else
return column_type::text;
return collation == binary_collation ? column_type::blob : column_type::text;
}
inline column_type compute_field_type(protocol_field_type protocol_type, std::uint32_t flags)
inline column_type compute_field_type(
protocol_field_type protocol_type,
std::uint32_t flags,
std::uint16_t collation
)
{
switch (protocol_type)
{
@ -66,9 +65,10 @@ inline column_type compute_field_type(protocol_field_type protocol_type, std::ui
case protocol_field_type::timestamp: return column_type::timestamp;
case protocol_field_type::time: return column_type::time;
case protocol_field_type::year: return column_type::year;
case protocol_field_type::string: return compute_field_type_string(flags);
case protocol_field_type::var_string: return compute_field_type_var_string(flags);
case protocol_field_type::blob: return compute_field_type_blob(flags);
case protocol_field_type::json: return column_type::json;
case protocol_field_type::string: return compute_field_type_string(flags, collation);
case protocol_field_type::var_string: return compute_field_type_var_string(collation);
case protocol_field_type::blob: return compute_field_type_blob(collation);
default: return column_type::unknown;
}
}
@ -85,7 +85,7 @@ boost::mysql::metadata::metadata(const detail::column_definition_packet& msg, bo
org_name_(copy_strings ? msg.org_name.value : string_view()),
character_set_(msg.character_set),
column_length_(msg.column_length),
type_(detail::compute_field_type(msg.type, msg.flags)),
type_(detail::compute_field_type(msg.type, msg.flags, msg.character_set)),
flags_(msg.flags),
decimals_(msg.decimals)
{

View File

@ -9,6 +9,7 @@
#define BOOST_MYSQL_TEST_COMMON_CREATE_META_HPP
#include <boost/mysql/metadata.hpp>
#include <boost/mysql/mysql_collations.hpp>
#include <boost/mysql/detail/auxiliar/access_fwd.hpp>
#include <boost/mysql/detail/protocol/common_messages.hpp>
@ -28,13 +29,15 @@ inline metadata create_meta(const detail::column_definition_packet& coldef, bool
inline metadata create_meta(
detail::protocol_field_type type,
std::uint16_t flags = 0,
std::uint8_t decimals = 0
std::uint8_t decimals = 0,
std::uint16_t collation = mysql_collations::utf8mb4_general_ci
)
{
detail::column_definition_packet coldef{};
coldef.type = type;
coldef.flags = flags;
coldef.decimals = decimals;
coldef.character_set = collation;
return create_meta(coldef, true);
}

View File

@ -35,6 +35,7 @@
#include "metadata_validator.hpp"
#include "printing.hpp"
#include "safe_getenv.hpp"
#include "tcp_network_fixture.hpp"
#include "test_common.hpp"
@ -50,6 +51,8 @@ using boost::mysql::detail::stringize;
BOOST_AUTO_TEST_SUITE(test_database_types)
// Helpers
bool is_mariadb() { return safe_getenv("BOOST_MYSQL_TEST_DB", "mysql8") == "mariadb"; }
using flagsvec = std::vector<meta_validator::flag_getter>;
const flagsvec flags_unsigned{&metadata::is_unsigned};
@ -539,17 +542,32 @@ table types_string()
res.add_meta("field_text", column_type::text);
res.add_meta("field_mediumtext", column_type::text);
res.add_meta("field_longtext", column_type::text);
res.add_meta("field_text_bincol", column_type::text);
res.add_meta("field_enum", column_type::enum_);
res.add_meta("field_set", column_type::set);
// clang-format off
res.add_row("regular", "test_char", "test_varchar", "test_tinytext", "test_text", "test_mediumtext", "test_longtext", "red", "red,green");
res.add_row("utf8", "\xc3\xb1", "\xc3\x91", "\xc3\xa1", "\xc3\xa9", "\xc3\xad", "\xc3\xb3", nullptr, nullptr);
res.add_row("empty", "", "", "", "", "", "", nullptr, "");
res.add_row("regular", "test_char", "test_varchar", "test_tinytext", "test_text", "test_mediumtext", "test_longtext", "test_bincol", "red", "red,green");
res.add_row("utf8", "\xc3\xb1", "\xc3\x91", "\xc3\xa1", "\xc3\xa9", "\xc3\xad", "\xc3\xb3", "\xc3\xba", nullptr, nullptr);
res.add_row("empty", "", "", "", "", "", "", "", nullptr, "");
// clang-format on
return res;
}
// MariaDB doesn't have a dedicated column type, so there is a difference in metadata.
// Values should be the same, though.
table types_json()
{
table res("types_json");
res.add_meta("field_json", is_mariadb() ? column_type::text : column_type::json);
res.add_row("regular", R"([null, 42, false, "abc", {"key": "value"}])");
res.add_row("unicode_escape", R"(["\\u0000value\\u0000"])");
res.add_row("utf8", "[\"adi\xc3\xb3os\"]");
res.add_row("empty", "{}");
return res;
}
table types_binary()
{
table res("types_binary");

View File

@ -397,7 +397,6 @@ INSERT INTO types_time VALUES
("max", "838:59:59", "838:59:58.9", "838:59:58.99", "838:59:58.999", "838:59:58.9999", "838:59:58.99999", "838:59:58.999999")
;
CREATE TABLE types_string(
id VARCHAR(50) NOT NULL PRIMARY KEY,
field_char CHAR(20),
@ -406,13 +405,28 @@ CREATE TABLE types_string(
field_text TEXT,
field_mediumtext MEDIUMTEXT,
field_longtext LONGTEXT,
field_text_bincol TEXT CHARACTER SET utf16 COLLATE utf16_bin,
field_enum ENUM("red", "green", "blue"),
field_set SET("red", "green", "blue")
);
INSERT INTO types_string VALUES
("regular", "test_char", "test_varchar", "test_tinytext", "test_text", "test_mediumtext", "test_longtext", "red", "red,green"),
("utf8", "ñ", "Ñ", "á", "é", "í", "ó", NULL, NULL),
("empty", "", "", "", "", "", "", NULL, "")
("regular", "test_char", "test_varchar", "test_tinytext", "test_text", "test_mediumtext", "test_longtext", "test_bincol", "red", "red,green"),
("utf8", "ñ", "Ñ", "á", "é", "í", "ó", "ú", NULL, NULL),
("empty", "", "", "", "", "", "", "", NULL, "")
;
-- JSON is handled different by MySQL and MariaDB, so a separate table helps tests
-- Having arrays in the json fields guarantees order, avoiding requiring parsing
-- the json object to verify equality.
CREATE TABLE types_json(
id VARCHAR(50) NOT NULL PRIMARY KEY,
field_json JSON
);
INSERT INTO types_json VALUES
("regular", '[null, 42, false, "abc", {"key": "value"}]'),
("unicode_escape", '["\\u0000value\\u0000"]'),
("utf8", '["adiós"]'),
("empty", '{}')
;
CREATE TABLE types_binary(

View File

@ -0,0 +1,33 @@
//
// Copyright (c) 2019-2023 Ruben Perez Hidalgo (rubenperez038 at gmail dot com)
//
// Distributed under the Boost Software License, Version 1.0. (See accompanying
// file LICENSE_1_0.txt or copy at http://www.boost.org/LICENSE_1_0.txt)
//
#ifndef BOOST_MYSQL_TEST_INTEGRATION_UTILS_INCLUDE_SAFE_GETENV_HPP
#define BOOST_MYSQL_TEST_INTEGRATION_UTILS_INCLUDE_SAFE_GETENV_HPP
namespace boost {
namespace mysql {
namespace test {
inline std::string safe_getenv(const char* name, const char* default_value)
{
// MSVC doesn't like getenv
#ifdef BOOST_MSVC
#pragma warning(push)
#pragma warning(disable : 4996)
#endif
const char* res = std::getenv(name);
#ifdef BOOST_MSVC
#pragma warning(pop)
#endif
return res ? res : default_value;
}
} // namespace test
} // namespace mysql
} // namespace boost
#endif

View File

@ -15,29 +15,15 @@
#include <cstdlib>
#include "get_endpoint.hpp"
#include "safe_getenv.hpp"
namespace {
std::string get_host()
{
// MSVC doesn't like getenv
#ifdef BOOST_MSVC
#pragma warning(push)
#pragma warning(disable : 4996)
#endif
const char* hostname = std::getenv("BOOST_MYSQL_SERVER_HOST");
#ifdef BOOST_MSVC
#pragma warning(pop)
#endif
return std::string(hostname ? hostname : "localhost");
}
// Get the endpoint to use for TCP from an environment variable.
// Required as CI MySQL doesn't run on loocalhost
boost::asio::ip::tcp::endpoint get_tcp_valid_endpoint()
{
std::string hostname = get_host();
std::string hostname = boost::mysql::test::safe_getenv("BOOST_MYSQL_SERVER_HOST", "localhost");
boost::asio::io_context ctx;
boost::asio::ip::tcp::resolver resolver(ctx.get_executor());
auto results = resolver.resolve(hostname, boost::mysql::default_port_string);
@ -46,8 +32,7 @@ boost::asio::ip::tcp::endpoint get_tcp_valid_endpoint()
} // namespace
boost::asio::ip::tcp::endpoint boost::mysql::test::endpoint_getter<
boost::asio::ip::tcp>::operator()()
boost::asio::ip::tcp::endpoint boost::mysql::test::endpoint_getter<boost::asio::ip::tcp>::operator()()
{
static auto res = get_tcp_valid_endpoint();
return res;

View File

@ -8,9 +8,11 @@
#include <boost/mysql/blob_view.hpp>
#include <boost/mysql/date.hpp>
#include <boost/mysql/datetime.hpp>
#include <boost/mysql/mysql_collations.hpp>
#include <boost/mysql/detail/auxiliar/access_fwd.hpp>
#include <boost/mysql/detail/auxiliar/stringize.hpp>
#include <boost/mysql/detail/protocol/constants.hpp>
#include <boost/mysql/detail/protocol/deserialize_binary_field.hpp>
#include <boost/mysql/detail/protocol/deserialize_errc.hpp>
@ -45,6 +47,7 @@ struct success_sample
field_view expected;
protocol_field_type type;
std::uint16_t flags;
std::uint16_t collation;
template <class T>
success_sample(
@ -52,13 +55,15 @@ struct success_sample
std::vector<std::uint8_t> from,
T&& expected_value,
protocol_field_type type,
std::uint16_t flags = 0
std::uint16_t flags = 0,
std::uint16_t collation = boost::mysql::mysql_collations::utf8mb4_general_ci
)
: name(std::move(name)),
from(std::move(from)),
expected(std::forward<T>(expected_value)),
type(type),
flags(flags)
flags(flags),
collation(collation)
{
}
};
@ -98,6 +103,7 @@ void add_string_samples(std::vector<success_sample>& output)
column_flags::set
));
output.push_back(success_sample("decimal", {0x02, 0x31, 0x30}, "10", protocol_field_type::newdecimal));
output.push_back(success_sample("json", {0x02, 0x7b, 0x7d}, "{}", protocol_field_type::json));
}
void add_blob_samples(std::vector<success_sample>& output)
@ -109,30 +115,34 @@ void add_blob_samples(std::vector<success_sample>& output)
{0x04, 0x01, 0x00, 0x73, 0x74},
blob_view(buff),
protocol_field_type::var_string,
column_flags::binary
column_flags::binary,
binary_collation
));
output.push_back(success_sample(
"binary",
{0x04, 0x01, 0x00, 0x73, 0x74},
blob_view(buff),
protocol_field_type::string,
column_flags::binary
column_flags::binary,
binary_collation
));
output.push_back(success_sample(
"blob",
{0x04, 0x01, 0x00, 0x73, 0x74},
blob_view(buff),
protocol_field_type::blob,
column_flags::binary
column_flags::binary,
binary_collation
));
output.push_back(success_sample(
"geometry",
{0x04, 0x01, 0x00, 0x73, 0x74},
blob_view(buff),
protocol_field_type::geometry
protocol_field_type::geometry,
binary_collation
));
// Anything we don't know what it is, we interpret as a string
// Anything we don't know what it is, we interpret as a blob
output.push_back(success_sample(
"unknown_protocol_type",
{0x04, 0x01, 0x00, 0x73, 0x74},
@ -644,7 +654,7 @@ std::vector<success_sample> make_all_samples()
BOOST_DATA_TEST_CASE(test_deserialize_binary_value_ok, data::make(make_all_samples()))
{
auto meta = create_meta(sample.type, sample.flags);
auto meta = create_meta(sample.type, sample.flags, 0, sample.collation);
field_view actual_value;
const bytestring& buffer = sample.from;
deserialization_context ctx(buffer.data(), buffer.data() + buffer.size(), capabilities());

View File

@ -11,10 +11,12 @@
#include <boost/mysql/date.hpp>
#include <boost/mysql/datetime.hpp>
#include <boost/mysql/field_view.hpp>
#include <boost/mysql/mysql_collations.hpp>
#include <boost/mysql/detail/auxiliar/access_fwd.hpp>
#include <boost/mysql/detail/auxiliar/string_view_offset.hpp>
#include <boost/mysql/detail/auxiliar/stringize.hpp>
#include <boost/mysql/detail/protocol/constants.hpp>
#include <boost/mysql/detail/protocol/deserialize_text_field.hpp>
#include <boost/test/data/monomorphic/collection.hpp>
@ -50,8 +52,9 @@ struct success_sample
std::string from;
field_view expected;
protocol_field_type type;
unsigned decimals;
std::uint16_t flags;
unsigned decimals;
std::uint16_t collation;
template <class T>
success_sample(
@ -60,14 +63,16 @@ struct success_sample
T&& expected_value,
protocol_field_type type,
std::uint16_t flags=0,
unsigned decimals=0
unsigned decimals=0,
std::uint16_t collation=boost::mysql::mysql_collations::utf8mb4_general_ci
) :
name(std::move(name)),
from(std::move(from)),
expected(std::forward<T>(expected_value)),
type(type),
flags(flags),
decimals(decimals),
flags(flags)
collation(collation)
{
}
};
@ -89,21 +94,20 @@ void add_string_samples(std::vector<success_sample>& output)
output.emplace_back("enum", "value", "value", protocol_field_type::string, column_flags::enum_);
output.emplace_back("set", "value1,value2", "value1,value2", protocol_field_type::string, column_flags::set);
output.emplace_back("decimal", "1", "1", protocol_field_type::newdecimal);
output.emplace_back("json", "{}", "{}", protocol_field_type::json);
}
void add_blob_samples(std::vector<success_sample>& output)
{
static constexpr std::uint8_t buff [] = { 0x00, 0x01, 0x02, 0x03 };
std::string from { 0x00, 0x01, 0x02, 0x03 };
output.emplace_back("varbinary_non_empty", from, blob_view(buff), protocol_field_type::var_string, column_flags::binary);
output.emplace_back("varbinary_empty", "", blob_view(), protocol_field_type::var_string, column_flags::binary);
output.emplace_back("binary", from, blob_view(buff), protocol_field_type::string, column_flags::binary);
output.emplace_back("blob", from, blob_view(buff), protocol_field_type::blob, column_flags::binary);
output.emplace_back("varbinary_non_empty", from, blob_view(buff), protocol_field_type::var_string, column_flags::binary, 0, binary_collation);
output.emplace_back("varbinary_empty", "", blob_view(), protocol_field_type::var_string, column_flags::binary, 0, binary_collation);
output.emplace_back("binary", from, blob_view(buff), protocol_field_type::string, column_flags::binary, 0, binary_collation);
output.emplace_back("blob", from, blob_view(buff), protocol_field_type::blob, column_flags::binary, 0, binary_collation);
output.emplace_back("geometry", from, blob_view(buff), protocol_field_type::geometry,
column_flags::binary | column_flags::blob);
column_flags::binary | column_flags::blob, 0, binary_collation);
// Anything we don't know what it is, we interpret as a blob
output.emplace_back("unknown_protocol_type", from,
@ -429,7 +433,7 @@ std::vector<success_sample> make_all_samples()
BOOST_DATA_TEST_CASE(ok, data::make(make_all_samples()))
{
auto meta = create_meta(sample.type, sample.flags, static_cast<std::uint8_t>(sample.decimals));
auto meta = create_meta(sample.type, sample.flags, static_cast<std::uint8_t>(sample.decimals), sample.collation);
const std::uint8_t* buffer_first = reinterpret_cast<const std::uint8_t*>(sample.from.data());
field_view actual_value;

View File

@ -25,6 +25,13 @@ def _run(args: List[str]) -> None:
subprocess.run(args, check=True)
def _run_piped_stdin(args: List[str], fname: Path) -> None:
with open(str(fname), 'rt', encoding='utf8') as f:
content = f.read()
print('+ ', args, '(with < {})'.format(fname), flush=True)
subprocess.run(args, input=content.encode(), check=True)
def _add_to_path(path: Path) -> None:
sep = ';' if _is_windows else ':'
os.environ['PATH'] = '{}{}{}'.format(path, sep, os.environ["PATH"])
@ -121,6 +128,20 @@ def _install_boost(
_run(['b2', 'headers'])
def _run_sql_file(fname: Path) -> None:
_run_piped_stdin(['mysql', '-u', 'root'], fname)
def _db_setup(
source_dir: Path,
db: str = 'mysql8'
) -> None:
_run_sql_file(source_dir.joinpath('example', 'db_setup.sql'))
_run_sql_file(source_dir.joinpath('test', 'integration', 'db_setup.sql'))
if db == 'mysql8':
_run_sql_file(source_dir.joinpath('test', 'integration', 'db_setup_sha256.sql'))
def _doc_build(
source_dir: Path,
clean: bool = False,
@ -158,7 +179,8 @@ def _b2_build(
stdlib: str = 'native',
address_model: str = '64',
clean: bool = False,
boost_branch: str = 'develop'
boost_branch: str = 'develop',
db: str = 'mysql8'
) -> None:
# Config
if _is_windows:
@ -172,6 +194,9 @@ def _b2_build(
branch=boost_branch
)
# Setup DB
_db_setup(source_dir, db)
# Invoke b2
_run([
'b2',
@ -206,6 +231,7 @@ def _cmake_build(
build_type: str = 'Debug',
cxxstd: str = '20',
boost_branch: str = 'develop',
db: str = 'mysql8'
) -> None:
# Config
home = Path(os.path.expanduser('~'))
@ -228,6 +254,9 @@ def _cmake_build(
branch=boost_branch
)
# Setup DB
_db_setup(source_dir, db)
# Generate "pre-built" b2 distro
if standalone_tests:
_run([
@ -438,7 +467,8 @@ def main():
stdlib=args.stdlib,
address_model=args.address_model,
clean=args.clean,
boost_branch=boost_branch
boost_branch=boost_branch,
db=args.db
)
elif args.build_kind == 'cmake':
_cmake_build(
@ -453,7 +483,8 @@ def main():
install_tests=args.cmake_install_tests,
build_type=args.cmake_build_type,
cxxstd=args.cxxstd,
boost_branch=boost_branch
boost_branch=boost_branch,
db=args.db
)
else:
_doc_build(

View File

@ -15,7 +15,8 @@ RUN \
libssl-dev \
git \
python3 \
python-is-python3 && \
python-is-python3 \
mysql-client && \
ln -s /usr/bin/clang++-11 /usr/bin/clang++ && \
ln -s /usr/bin/clang-11 /usr/bin/clang

View File

@ -20,7 +20,8 @@ RUN \
ninja-build \
cmake \
python3 \
python-is-python3 && \
python-is-python3 \
mysql-client && \
ln -s /usr/bin/clang++-14 /usr/bin/clang++ && \
ln -s /usr/bin/clang-14 /usr/bin/clang

View File

@ -14,7 +14,8 @@ RUN \
ca-certificates \
libssl-dev \
git \
python3 && \
python3 \
mysql-client && \
add-apt-repository -y ppa:ubuntu-toolchain-r/test && \
apt-get --no-install-recommends -y install clang-3.6 && \
ln -s /usr/bin/clang++-3.6 /usr/bin/clang++ && \

View File

@ -15,7 +15,8 @@ RUN \
libssl-dev \
git \
python3 \
python-is-python3 && \
python-is-python3 \
mysql-client && \
ln -s /usr/bin/clang++-7 /usr/bin/clang++ && \
ln -s /usr/bin/clang-7 /usr/bin/clang

View File

@ -18,7 +18,8 @@ RUN \
gnupg-curl \
libssl-dev \
ninja-build \
curl && \
curl \
mysql-client && \
ln -s /usr/bin/python3 /usr/bin/python
RUN \

View File

@ -16,7 +16,8 @@ RUN \
libssl-dev \
git \
python3 \
python-is-python3 && \
python-is-python3 \
mysql-client && \
ln -s /usr/bin/g++-10 /usr/bin/g++ && \
ln -s /usr/bin/gcc-10 /usr/bin/gcc

View File

@ -26,7 +26,8 @@ RUN \
git \
gpg \
cmake \
gpg-agent && \
gpg-agent \
mysql-client && \
unlink /usr/bin/gcc && \
ln -s /usr/bin/g++-11 /usr/bin/g++ && \
ln -s /usr/bin/gcc-11 /usr/bin/gcc && \

View File

@ -14,7 +14,8 @@ RUN \
ca-certificates \
libssl-dev \
git \
python3 && \
python3 \
mysql-client && \
add-apt-repository -y ppa:ubuntu-toolchain-r/test && \
apt-get --no-install-recommends -y install gcc-5 g++-5 && \
ln -s /usr/bin/g++-5 /usr/bin/g++ && \

View File

@ -14,7 +14,8 @@ RUN \
ca-certificates \
libssl-dev \
git \
python3 && \
python3 \
mysql-client && \
add-apt-repository -y ppa:ubuntu-toolchain-r/test && \
apt-get --no-install-recommends -y install gcc-6 g++-6 && \
ln -s /usr/bin/g++-6 /usr/bin/g++ && \

View File

@ -10,15 +10,8 @@ FROM ${BASE_IMAGE}
COPY tools/win-ci.cnf C:/my.cnf
COPY tools/ssl C:/ssl
COPY test/integration/*.sql C:/db_setup/integration/
COPY example/db_setup.sql C:/db_setup/example/
RUN choco install --no-progress -y mysql --version 8.0.20 && \
call refreshenv && \
CHCP 65001 && \
mysql -u root < C:/db_setup/integration/db_setup.sql && \
mysql -u root < C:/db_setup/integration/db_setup_sha256.sql && \
mysql -u root < C:/db_setup/example/db_setup.sql
RUN choco install --no-progress -y mysql --version 8.0.20
RUN powershell -Command \
$ErrorActionPreference = 'Stop' ; \

View File

@ -17,5 +17,6 @@ RUN \
ninja-build \
cmake \
python3 \
python-is-python3
python-is-python3 \
mysql-client

View File

@ -10,8 +10,6 @@ FROM mariadb:10.11.2
ENV MYSQL_ALLOW_EMPTY_PASSWORD=1
ENV MYSQL_ROOT_PASSWORD=
COPY example/db_setup.sql /docker-entrypoint-initdb.d/example_db_setup.sql
COPY test/integration/db_setup.sql /docker-entrypoint-initdb.d/
COPY tools/docker/mysql_entrypoint.sh /
COPY tools/docker/ssl.cnf /etc/mysql/conf.d/
COPY tools/ssl/*.pem /etc/ssl/certs/mysql/

View File

@ -10,8 +10,6 @@ FROM mysql:5.7.41
ENV MYSQL_ALLOW_EMPTY_PASSWORD=1
ENV MYSQL_ROOT_PASSWORD=
COPY example/db_setup.sql /docker-entrypoint-initdb.d/example_db_setup.sql
COPY test/integration/db_setup.sql /docker-entrypoint-initdb.d/
COPY tools/docker/mysql_entrypoint.sh /
COPY tools/docker/ssl.cnf /etc/mysql/conf.d/
COPY tools/ssl/*.pem /etc/ssl/certs/mysql/

View File

@ -10,8 +10,6 @@ FROM mysql:8.0.32
ENV MYSQL_ALLOW_EMPTY_PASSWORD=1
ENV MYSQL_ROOT_PASSWORD=
COPY example/db_setup.sql /docker-entrypoint-initdb.d/example_db_setup.sql
COPY test/integration/*.sql /docker-entrypoint-initdb.d/
COPY tools/docker/mysql_entrypoint.sh /
COPY tools/docker/ssl.cnf /etc/mysql/conf.d/
COPY tools/ssl/*.pem /etc/ssl/certs/mysql/

View File

@ -8,12 +8,12 @@
set -e
BK=docs
IMAGE=build-docs
BK=b2
IMAGE=build-gcc11
CONTAINER=builder-$IMAGE-$BK
FULL_IMAGE=ghcr.io/anarthal-containers/$IMAGE
docker start mysql8
docker start mariadb
docker start $CONTAINER || docker run -dit \
--name $CONTAINER \
-v ~/workspace/mysql:/opt/boost-mysql \
@ -33,7 +33,9 @@ docker exec $CONTAINER python /opt/boost-mysql/tools/ci.py --source-dir=/opt/boo
--cmake-add-subdir-tests=1 \
--cmake-install-tests=1 \
--cmake-build-type=Release \
--stdlib=native
--stdlib=native \
--server-host=mariadb \
--db=mariadb
if [ "$BK" == "docs" ]; then
cp -r ~/workspace/mysql/doc/html ~/workspace/boost-root/libs/mysql/doc/

View File

@ -29,9 +29,8 @@ BASE_FOLDERS = [
'.github'
]
BASE_FILES = [
'.appveyor.yml',
'CMakeLists.txt',
'Jamfile'
'.drone.star'
]
HTML_GEN_PATH = path.join(REPO_BASE, 'doc', 'html')
@ -160,6 +159,7 @@ FILE_PROCESSORS : List[Tuple[str, BaseProcessor]] = [
('.yml', hash_processor),
('.cnf', hash_processor),
('.dockerfile', hash_processor),
('.star', hash_processor),
('.cpp', cpp_processor),
('.hpp', HppProcessor()),
('.ipp', HppProcessor()),
@ -168,6 +168,7 @@ FILE_PROCESSORS : List[Tuple[str, BaseProcessor]] = [
('.svg', IgnoreProcessor()),
('valgrind_suppressions.txt', IgnoreProcessor()),
('.pem', IgnoreProcessor()),
('.md', IgnoreProcessor()),
]
def process_file(fpath: str):
@ -193,6 +194,8 @@ def process_all_files():
continue
for fname in files:
process_file(path.join(curdir, fname))
for fname in BASE_FILES:
process_file(path.join(REPO_BASE, fname))
# Check that cmake and b2 test source files are equal

View File

@ -24,8 +24,3 @@ sudo chmod 777 /var/run/mysqld/
# Start the server
mysql.server start
# Load the data
mysql -u root < test/integration/db_setup.sql
mysql -u root < test/integration/db_setup_sha256.sql
mysql -u root < example/db_setup.sql