Property fuzzing for OCaml
Crowbar is a library for testing code, combining QuickCheck-style property-based testing and the magical bug-finding powers of afl-fuzz.
There are some examples.
Some brief hints:
opam sw 4.04.0+afl).
afl-fuzz -i in -o out -- ./_build/myprog.exe @@.
To test your software, come up with a property you'd like to test, then decide on the input you'd like for Crowbar to vary. A Crowbar test is some invocation of
let identity x = Crowbar.check_eq x x
and instructions for running the test with generated items with
let () = Crowbar.(add_test ~name:"identity function" [int] (fun i -> identity i))
There are more examples available, with varying levels complexity.
crowbarin your list of dependencies via your favorite build system. The resulting executable is a Crowbar test. (Be sure to build a native-code executable, not bytecode.)
To build tests that run under AFL, you'll need to build your tests with a compiler that has AFL instrumentation enabled. (You can also enable it specifically for your build, although this is not recommended if your code has any dependencies, including the OCaml standard library). OCaml compiler variants with AFL enabled by default are available in
+afltag. All versions published starting with 4.05.0 are available, along with a backported 4.04.0.
$ opam switch 4.06.0+afl $ eval `opam config env` $ ./build_my_rad_test.sh # or your relevant build runes
Crowbar tests have two modes:
afl-fuzzwith OCaml's instrumentation enabled
Crowbar tests can be directly invoked with
--helpfor more documentation at runtime.
If you wish to use the quickcheck-like, fully random mode to run all tests distributed here, build the tests as above and then run the binary with no arguments.
$ ./my_rad_test.exe | head -5 the first test: PASS
the second test: PASS
To run the tests in AFL mode, you'll need to install American Fuzzy Lop (latest source tarball, although your distribution may also have a package available).
afl-fuzzis available on your system, create an
inputdirectory with a non-empty file in it (or use
test/input, conveniently provided in this repository), and an
afl-fuzzto store its findings. Then, invoke your test binary:
afl-fuzz -i test/input -o output ./my_rad_test.exe @@
This will launch AFL, which will generate new test cases and track the exploration of the state space. When inputs are discovered which cause a property not to hold, they will be reported as crashes (along with actual crashes, although in the OCaml standard library these are rare). See the afl-fuzz documentation for more on AFL's excellent interface.
An open issue has a list of issues discovered by testing with Crowbar. If you use Crowbar to improve your software, please let us know!