Static JSON parsing in C++
The DAW JSON Link library provides multiple ways to serialization/deserialization JSON documents in C++. The primary one is parsing of JSON directly to your C++ data structures. This allows the known structure of the document to be exploited for greater checking and performance. Alternatively, there is an event passing(SAX) interface that can parse to generic types(double, string, bool,...) or can use the same type restricted parsers as the static parser previously mentioned. A generic DOM(lazy) based parser is provided that can be iterate over the document structure too, again it can use the generic parsers or the type based restricted versions. One can mix the three modes of parsing to form more complicated systems. For serialization, the first static mapping method is required, there is no json value type in the library. The library is, also, non-intrusive into your data structures and does not require member's to be declared/defined within them. This allows keeping the mapping in a separate header file from the data structures themselves.
The library is using the BSL licensed
When the structure of the JSON document is known, parsing is like the following:
c++ MyThing thing = daw::json::from_json( json_string );or for array documents, where the root of the document is an array, there is a helper method to make it easier and it can be parsed like the following:
c++ std::vector things = daw::json::from_json_array( json_string2 );If the structure of the JSON document is unknown, one can construct a
json_valuethat acts as a container and allows iteration and parsing on demand. It is a lazy parser and will only parse when asked to. The following is an example of opening a
json_valuefrom JSON data:
c++ json_value val = daw::json::json_value( json_string );
The
from_jsonand
to_jsonmethods allow access most of the parsing needs.
The event based parser(SAX) can be called via
daw::json::json_event_parser. It takes two arguments, a json document and an event handler. The event handler can opt into events by having the following members: * handleonvalue * handleonarraystart * handleonarrayend * handleonclassstart * handleonclassend * handleonnumber * handleonbool * handleonstring * handleonnull * handleonerror
Mapping of your classes to JSON documents is done by specializing the trait
daw::json::json_data_contract. A class that is mapped does not need to be mapped again if it is a member of another mapped class. There are two parts to the trait
json_data_contract, first is a type alias named
typethat maps the JSON members to our class's constructor. This gets around needing private access to the class, assuming that data we would serialize would also be needed to construct the class. For example:
c++ struct Thing { int a; int b; };The construct for
Thingrequires 2 integers and if we had the following JSON:
json { "a": 42, "b": 1234 }We could do the mapping like the following:
c++ namespace daw::json { template<> struct json_data_contract { static constexpr char const a[] = "a"; static constexpr char const b[] = "b"; using type = json_member_list< json_number, json_number >; }; }This says that the JSON class will have at least two members "a", and "b" that will be numbers that are integers. They will be passed to the constructor of
Thingwhen
daw::json::from_json( json_doc );is called, or that another class has a
json_classmember mapping. The above is the C++17 mapping method for the names, it works in future C++ versions too. But, in C++20 and later the names can be inline in the mapping e.g.
json_number. The above is all that is needed for parsing JSON, for serializing a static member function is needed in the trait. Taking the previous example and extending it we could serialize
Thingwith: ```c++ namespace daw::json { template<> struct jsondatacontract { static constexpr char const a[] = "a"; static constexpr char const b[] = "b"; using type = jsonmemberlist< jsonnumber, jsonnumber >; };
static auto tojsondata( Thing const & v ) { return std::forwardastuple( v.a, v.b ); } } ``
The ordering of the members returned as a tuple need to match the mapping in the type aliastype
. This allows for passing the result of accessor methods too, if the data members are not public.
The parsers work by constructing each argument in place in the call to the classes constructor. The individual argument parsers can be tuned for the specified circumstances of the data(e.g. floating point and integral numbers). Then with our type trait defining the arguments needed to construct the C++ class and their order we are able to look at each member in the JSON. Now we construct the value with the result of each parser; similar to
T{ parse<0, json_string>( data ), parse<1, json_number>( data ), parse>( data )}. For each member, the data stream will be moved forward until we find the member we need to parse, storing interested locations for later parsing. This process allows us to parse other classes as members too via the
json_classmapping type. So that each mapping trait only has to deal with it's specific members and not their details.
In unnamed contexts, such as the root value, array elements, some key value types, and variant element lists where the name would be
no_name, one can use some native C++ data types instead of the the JSON mapping types. This includes, integer, floating point, bool, std::string, std::string_view, and previously mapped classes.
For example, to map an array of string's.
c++ template<> struct daw::json::json_data_contract { using type = json_member_list>; };
To use dawjsonlink in your cmake projects, adding the following should allow it to pull it in along with the dependencies:
cmake include( FetchContent ) FetchContent_Declare( daw_json_link GIT_REPOSITORY https://github.com/beached/daw_json_link GIT_TAG release ) FetchContent_MakeAvailable(daw_json_link)Then in the targets that need it:
cmake target_link_libraries( MyTarget daw::json_link )
On a system with bash, it is similar on other systems too, the following can install for the system
bash git clone https://github.com/beached/daw_json_link cd daw_json_link mkdir build cd build cmake .. cmake --install .
The following will build and run the tests.
bash git clone https://github.com/beached/daw_json_link cd daw_json_link mkdir build cd build cmake -DDAW_ENABLE_TESTING=On .. cmake --build . ctest .After the build there the individual examples can be tested too.
city_test_binrequires the path to the cities JSON file.
bash ./tests/city_test_bin ../test_data/cities.json
The order of the members in the data structures should generally match that of the JSON data. The parser is faster if it doesn't have to back track for values. Optional values, when missing in the JSON data, can slow down the parsing too. If possible have them sent as null. The parser does not allocate. The parsed to data types may and this allows one to use custom allocators or a mix as their data structures will do the allocation. The defaults for arrays is to use the std::vector and if this isn't desirable, you must supply the type.
The library, currently, does not unescape/escape member names when serializing, they are expected to be valid and unescaped. This may be a future optional addition, as it does have a cost.
There are slight differences between C++17 and C++20
namespace daw::json { template<> struct json_data_contract { static constexpr char const member_name[] = "memberName"; using type = json_member_list>; }; }
When compiled within C++20 compiler, in addition to passing a
char const *as in C++17, the member names can be specified as string literals directly. C++20 compiler support is still really early and here be dragons. There are known issues with g++9.x and it's only tested with g++10. Here be dragons
c++ namespace daw::json { template<> struct json_data_contract { using type = json_member_list>; }; }
Once a data type has been mapped with a
json_data_contract, the library provides methods to parse JSON to them
MyClass my_class = from_json( json_str );
Alternatively, if the input is trusted, the less checked version can be faster
c++ MyClass my_class = from_json( json_str );
JSON documents with array root's use the
from_json_arrayfunction to parse
c++ std::vector my_data = from_json_array( json_str );Alternatively, if the input is trusted, the less checked version can be faster
c++ std::vector my_data = from_json_array, NoCommentSkippingPolicyUnchecked>( json_str );
If you want to work from JSON array data you can get an iterator and use the std algorithms to Iterating over array's in JSON data can be done via the
json_array_iterator
c++ using iterator_t = json_array_iterator; auto pos = std::find( iterator_t( json_str ), iterator_t( ), MyClass( ... ) );Alternatively, if the input is trusted you can called the less checked version
c++ using iterator_t = daw::json::json_array_iterator_trusted; auto pos = std::find( iterator_t( json_str ), iterator_t( ), MyClass( ... ) );
If you want to serialize to JSON
std::string my_json_data = to_json( MyClass{} );
Or serialize a collection of things
c++ std::vector arry = ...; std::string my_json_data = to_json_array( arry );
Error checking can be modified on a per parse basis. the fromjson/fromjson_array calls can be supplied a Parser Policy. The current policies are
NoCommentSkippingPolicyChecked- No comments allowed, checks enabled
NoCommentSkippingPolicyUnchecked- No comments allowed, assumes perfect JSON
CppCommentSkippingPolicyChecked- C++ style comments
/* commment */and
// comment until end of line, checks enabled
CppCommentSkippingPolicyUnchecked- C++ style comments
/* commment */and
// comment until end of line, assumes perfect JSON
HashCommentSkippingPolicyChecked- Hash style comments
# comment until end of line, checks enabled
HashCommentSkippingPolicyUnchecked- Hash style comments
# comment until end of line, assumes perfect JSON
The unchecked variants can sometimes provide a 5-15% performance increase, but at great risk when the data isn't perfect.
There are two possible ways of handling errors. The default is to throw a
daw::json::json_exceptionon an error in the data.
json_exceptionhas a member function
std::string_view reason( ) constakin to
std::exception's
what( ). Second, calling
std::terminate( );on an error in data. If you want to disable exceptions in an environment that has them, you can defined
DAW_JSON_DONT_USE_EXCEPTIONSto disable exception throwing by the library.
This can be accomplished by writing a function called jsondatacontract_for with a single argument that is your type. The library is only concerned with it's return value. For example:
#includestruct TestClass { int i = 0; double d = 0.0; bool b = false; daw::string_view s{}; std::vector y{};
TestClass( int Int, double Double, bool Bool, daw::string_view S, std::vector Y ) : i( Int ) , d( Double ) , b( Bool ) , s( S ) , y( Y ) {} };
namespace daw::json { template<> struct json_data_contract { using type = json_member_list< json_number, json_number, json_bool, json_string, json_array >; }; }
int main( ) { std::string test_001_t_json_data = R"({ "i":5, "d":2.2e4, "b":false, "s":"hello world", "y":[1,2,3,4] })"; std::string json_array_data = R"([{ "i":5, "d":2.2e4, "b":false, "s":"hello world", "y":[1,2,3,4] },{ "i":4, "d":122e4, "b":true, "s":"goodbye world", "y":[4,3,1,4] }])";
TestClass test_class = daw::json::from_json( test_001_t_json_data ); std::vector arry_of_test_class = daw::json::from_json_array( test_001_t_json_data ); }
Both aggregate and user constructors are supported. The description provides the values needed to construct your type and the order. The order specified is the order they are placed into the constructor. There are customization points to provide a way of constructing your type too(TODO discuss customization points) A class like:
#includestruct AggClass { int a{}; double b{}; };
namespace daw::json { template<> struct json_data_contract { using type = json_member_list< json_number, json_number >; }; }
Works too. Same but C++17 ```c++
struct AggClass { int a{}; double b{}; };
namespace daw::json { template<> struct jsondatacontract { static inline constexpr char const a[] = "a"; static inline constexpr char const b[] = "b"; using type = jsonmemberlist< jsonnumber, jsonnumber >; }; } ``
The class descriptions are recursive with their submembers. Using the previousAggClass` one can include it as a member of another class
// See above for AggClass struct MyClass { AggClass other; std::string_view some_name; };namespace daw::json { template<> struct json_data_contract { using type = json_member_list< json_class, json_string >; }; }
The above maps a class MyClass that has another class that is described AggClass. Also, you can see that the member names of the C++ class do not have to match that of the mapped JSON names and that strings can use
std::string_viewas the result type. This is an important performance enhancement if you can guarantee the buffer containing the JSON file will exist as long as the class does.
Iterating over JSON arrays. The input iterator
daw::json::json_array_iteratorallows one to iterator over the array of JSON elements. It is technically an input iterator but can be stored and reused like a forward iterator. It does not return a reference but a value. ```c++
struct AggClass { int a{}; double b{}; };
namespace daw::json { template<> struct jsondatacontract { using type = jsonmemberlist< jsonnumber<"a", int>, jsonnumber<"b"> >; }; }
int main( ) { std::string jsonarraydata = R"([ {"a":5,"b":2.2}, {"a":5,"b":3.14}, {"a":5,"b":0.122e44}, {"a":5334,"b":34342.2} ])"; using iteratort = daw::json::jsonarrayiterator; auto pos = std::findif( iteratort( jsonarraydata ), iteratort( ), { return element.b > 1000.0; } ); if( pos == iterator_t( ) ) { std::cout << "Not found\n"; } else { std::cout << "Found\n"; } } ```
Parsing can begin at a specific member. An optional member path to
from_json_array,
from_json_array_unchecked,
from_json_array, or
from_json_array_uncheckedcan be specified. The format is a dot separated list of member names and optionally an array index such as
member0.member1or
member0[5].member1.
Comments are supported when the parser policy for them is used. Currently there are two forms of comment policies. C++ style
//and
/* */. Comments can be placed anywhere there is whitespace allowed
Hash style
{ # This is a comment "a" #this is also a comment : "a's value" }
C++ style
{ // This is a comment "a" /*this is also a comment*/: "a's value" }To change the parser policy, you add another argument to
from_jsonand call like
from_json( json_data )
To enable serialization one must create an additional function in your specialization of
json_data_contractcalled
to_json_data( Thing const & );It will provide a mapping from your type to the arguments provided in the class description. To serialize to a JSON string, one calls
to_json( my_thing );where value is a registered type or one of the fundamental types like string, bool, and numbers. The result of
to_json_data( Thing const & )is a tuple who's elements match order in jsondatacontract's type alias
type. Using the example above lets add that
#include #includestruct AggClass { int a{}; double b{}; };
namespace daw::json { template<> struct json_data_contract { using type = json_member_list< json_number, json_number >;
static inline auto to_json_data( AggClass const & value ) { return std::forward_as_tuple( value.a, value.b ); }
}; } //... AggData value = //...; std::string test_001_t_json_data = to_json( value );
// or std::vector values = //...; std::string json_array_data = to_json_array( values );
Alternatively there is an optional
iostreamsinterface. In you types
json_data_constractadd a type alias named
opt_into_iostreamsthe type it aliases doesn't matter, and include
daw_json_iostream.h. For example
struct AggClass { int a{}; double b{}; };
namespace daw::json { template<> struct jsondatacontract { using optintoiostreams = void; using type = jsonmemberlist< jsonnumber<"a", int>, jsonnumber<"b"> >;
static inline auto to_json_data( AggClass const & value ) { return std::forward_as_tuple( value.a, value.b ); }
}; } //... AggData value = //...; std::cout << value << '\n';
// or std::vector values = //...; std::cout << values << '\n'; ``` A working example can be found at dawjsoniostream_test.cpp
There are a few defines that affect how JSON Link operates *
DAW_JSON_DONT_USE_EXCEPTIONS- Controls if exceptions are allowed. If they are not, an
std::terminate()on errors will occur *
DAW_ALLOW_SSE42- Allow experimental SSE3 mode *
DAW_JSON_NO_CONST_EXPR- This can be used to allow classes without move/copy special members to be constructed from JSON data prior to C++ 20. This mode does not work in a constant expression prior to C++20 when this flag is no longer needed.
Darrell Wright [email protected]
json_valuesee or as a
json_key_valuethat is mapped to a
std::multimapor a
std::vectorwith a pair of key type(
string) and value type(s). Cookbook Key Values demonstrates these methods. If a
json_key_valueis used and the mapped data type does not support duplicate keys, it will insert for each key. This may result in the last item being the value reflected after serializing. If the duplicate member is the tag type in a
json_tagged_variant, it is undefined what the behaviour for parsing is.