Need help with schema-guru?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

137 Stars 18 Forks Apache License 2.0 141 Commits 53 Opened issues


JSONs -> JSON Schema (and Avro coming soon)

Services available


Need anything else?

Contributors list

Schema Guru

Build Status Release License

Schema Guru is a tool (CLI, Spark job and web) allowing you to derive JSON Schemas from a set of JSON instances process and transform it into different data definition formats.

Current primary features include:

  • deriviation of JSON Schema from set of JSON instances (
  • generation of Redshift table DDL and JSONPaths file (

Unlike other tools for deriving JSON Schemas, Schema Guru allows you to derive schema from an unlimited set of instances (making schemas much more precise), and supports many more JSON Schema validation properties.

Schema Guru is used heavily in association with Snowplow's own Snowplow, Iglu and Schema DDL projects.

User Quickstart

Download the latest Schema Guru from Bintray:

$ wget
$ unzip

Assuming you have a recent JVM installed.


Schema derivation

You can use as input either single JSON file or directory with JSON instances (it will be processed recursively).

Following command will print JSON Schema to stdout:

$ ./schema-guru-0.6.2 schema {{input}}

Also you can specify output file for your schema:

$ ./schema-guru-0.6.2 schema --output {{json_schema_file}} {{input}}

You can also switch Schema Guru into NDJSON mode, where it will look for newline delimited JSONs:

$ ./schema-guru-0.6.2 schema --ndjson {{input}}

You can specify the enum cardinality tolerance for your fields. It means that all fields which are found to have less than the specified cardinality will be specified in the JSON Schema using the

$ ./schema-guru-0.6.2 schema --enum 5 {{input}}

If you know that some particular set of values can appear, but don't want to set big enum cardinality, you may want to specify predefined enum set with

multioption, like this:
$ ./schema-guru-0.6.2 schema --enum-sets iso_4217 --enum-sets iso_3166-1_aplha-3 /path/to/instances

Currently Schema Guru includes following built-in enum sets (written as they should appear in CLI):

If you need to include very specific enum set, you can define it by yourself in JSON file with array like this:

["Mozilla Firefox", "Google Chrome", "Netscape Navigator", "Internet Explorer"]

And pass path to this file instead of enum name:

$ ./schema-guru-0.6.2 schema --enum-sets all --enum-sets /path/to/browsers.json /path/to/instances

Schema Guru will derive

properties for strings based on shortest and longest strings. But this may be a problem if you process small amount of instances. To avoid this too strict Schema, you can use
$ ./schema-guru-0.6.2 schema --no-length /path/to/few-instances

DDL derivation

Like for Schema derivation, for DDL input may be also single file with JSON Schema or directory containing JSON Schemas.

Currently we support DDL only for Amazon Redshift, but in future releases you'll be able to specify another with


Following command will just save Redshift (default

value) DDL to current dir.
$ ./schema-guru-0.6.2 ddl {{input}}

If you specified as input a directory with several Self-describing JSON Schemas belonging to a single REVISION, Schema Guru will also generate a migrations. So, you can migratte any of previous tables to any of subsequent. For example, having following list of Self-describing JSON Schemas as input:

  • schemas/com.acme/click_event/1-0-0
  • schemas/com.acme/click_event/1-0-1
  • schemas/com.acme/click_event/1-0-2

You will have following migrations as output:

  • sql/com.acme/click_event/1-0-0/1-0-1 to alter table from 1-0-0 to 1-0-1
  • sql/com.acme/click_event/1-0-0/1-0-2 to alter table from 1-0-0 to 1-0-2
  • sql/com.acme/click_event/1-0-1/1-0-2 to alter table from 1-0-1 to 1-0-2

This migrations (and all subsequent table definitions) are aware of column order and it will never put a new column in the middle of table, so you can safely alter your tables while they belong to a single REVISION.

You also can specify directory for output:

$ ./schema-guru-0.6.2 ddl --output {{ddl_dir}} {{input}}

If you're not a Snowplow Platform user, don't use Self-describing Schema or just don't want anything specific to it you can produce raw schema:

$ ./schema-guru-0.6.2 ddl --raw {{input}}

But bear in mind that Self-describing Schemas bring many benefits. For example, raw Schemas will not preserve an order for your columns (it just impossible!) and also you will not have a migrations.

You may also want to get JSONPaths file for Redshift's COPY command. It will place

dir alongside with
$ ./schema-guru-0.6.2 ddl --with-json-paths {{input}}

The most embarrassing part of shifting from dynamic-typed world to static-typed is product types (or union types) like this in JSON Schema:

["integer", "string"]
. How to represent them in SQL DDL? It's a taught question and we think there's no ideal solution. Thus we provide you two options. By default product types will be transformed as most general
. But there's another way - you can split column with product types into separate ones with it's types as postfix, for example property
with type
["string", "integer"]
will be transformed into two columns
. This behavior can be achieved with

Another thing everyone need to consider is default VARCHAR size. If there's no clues about it (like

) 4096 will be used. You can also specify this default value:
$ ./schema-guru-0.6.2 ddl --varchar-size 32 {{input}}

You can also specify Redshift Schema for your table. For non-raw mode

used as default.
$ ./schema-guru-0.6.2 ddl --raw --schema business {{input}}

Some users do not full rely on Schema Guru JSON Schema derivation or DDL generation and edit their DDLs manually. By default, Schema Guru will not override your files (either DDLs and migrations) if user made any significant changes (comments and whitespaces are not significant). Instead Schema Guru will just warn user that file has been changed manually. To change this behavior you may specify

$ ./schema-guru-0.6.2 ddl --force {{input}}

Web UI

You can access our hosted demo of the Schema Guru web UI at To run it locally:

$ wget
$ unzip
$ ./schema-guru-webui-0.6.2

The above will run a Spray web server containing Schema Guru on Interface and port can be specified by


Apache Spark

Since version 0.4.0 Schema Guru shipping with Spark job for deriving JSON Schemas. To help users getting started with Schema Guru on Amazon Elastic MapReduce we provide pyinvoke

Recommended way to start is install all requirements and assembly fatjar as described in Developer Quickstart.

Before run you need:

  • An AWS CLI profile, e.g. my-profile
  • A EC2 keypair, e.g. my-ec2-keypair
  • At least one Amazon S3 bucket, e.g. my-bucket

To provision the cluster and start the job you need to use

$ cd sparkjob
$ inv run_emr my-profile my-bucket/input/ my-bucket/output/ my-bucket/errors/ my-bucket/logs my-ec2-keypair

If you need some specific options for Spark job, you can specify these in
. The Spark job accepts the same options as the CLI application, but note that
isn't optional and we have a new optional
. Also, instead of specifying some of predefined enum sets you can just enable it with
flag, so it has the same behaviour as
--enum-sets all

Developer Quickstart

Assuming git, Vagrant and VirtualBox installed:

 host$ git clone
 host$ cd schema-guru
 host$ vagrant up && vagrant ssh
guest$ cd /vagrant
guest$ sbt assembly

Also, optional:

guest$ sbt "project schema-guru-webui" assembly
guest$ sbt "project schema-guru-sparkjob" assembly

You can also deploy the Schema Guru web GUI onto Elastic Beanstalk:

guest$ cd beanstalk && zip *

Now just create a new Docker app in the Elastic Beanstalk Console and upload this zipfile.

User Manual


Schema derivation

  • Takes a directory as an argument and will print out the resulting JsonSchema:
    • Processes each JSON sequentially
    • Merges all results into one master Json Schema
  • Recognizes following JSON Schema formats:
    • uuid
    • date-time (according to ISO-8601)
    • IPv4 and IPv6 addresses
  • Recoginzed minLength and maxLength properties for strings
  • Recognizes base64 pattern for strings
  • Detects integer ranges according to Int16, Int32, Int64
  • Detects misspelt properties and produce warnings
  • Detects enum values with specified cardinality
  • Detects known enum sets built-in or specified by user
  • Allows to output Self-describing JSON Schema
  • Allows to produce JSON Schemas with different names based on given JSON Path
  • Supports Newline Delimited JSON

DDL derivation

  • Correctly transforms some of string formats
    • uuid becomes
    • ipv4 becomes
    • ipv6 becomes
    • date-time becomes
  • Handles properties with only enums
  • Property with
  • Can output JSONPaths file
  • Can split product types
  • Number with
    0.01 becomes
  • Handles Self-describing JSON and can produce raw DDL
  • Recognizes integer size by
  • Object without
    , but with


  • All JSONs in the directory are assumed to be of the same event type and will be merged together
  • All JSONs are assumed to start with either
    { ... }
    [ ... ]
    • If they do not they are discarded
  • Schema should be as strict as possible - e.g. no
    are allowed currently

Self-describing JSON

command allows you to produce Self-describing JSON Schema. To produce it you need to specify vendor, name (if segmentation isn't using, see below), and version (optional, default value is 1-0-0).
$ ./schema-guru-0.6.2 schema --vendor {{your_company}} --name {{schema_name}} --schemaver {{version}} {{input}}

Schema Segmentation

If you have set of mixed JSONs from one vendor, but with slightly different structure, like:

{ "version": 1,
  "type": "track",
  "userId": "019mr8mf4r",
  "event": "Purchased an Item",
  "properties": {
    "revenue": "39.95",
    "shippingMethod": "2-day" },
  "timestamp" : "2012-12-02T00:30:08.276Z" }


{ "version": 1,
  "type": "track",
  "userId": "019mr8mf4r",
  "event": "Posted a Comment",
  "properties": {
    "body": "This book is gorgeous!",
    "attachment": false },
  "timestamp" : "2012-12-02T00:28:02.273Z" }

You can run it as follows:

$ ./schema-guru-0.6.2 schema --output {{output_dir}} --schema-by $.event {{mixed_jsons_directory}}

It will put two (or may be more) JSON Schemas into output dir: PurchasedanItem.json and Postedacomment.json. They will be derived from JSONs only with corresponding event property, without any intersections. Assuming that provided JSON Path contain valid string. All schemas where this JSON Path is absent or contains not a string value will be merged into unmatched.json schema in the same output dir. Also, when Self-describing JSON Schema producing, it will take schema name in the same way and --name argument can be omitted (it will replace name specified with option).


Here's an example of some subtle points which a tool working with a single JSON instance would miss.

First instance:

{ "event": {
    "just_a_string": "Any string may be here",
    "sometimes_ip": "",
    "always_ipv4": "",
    "id": 43,
    "very_big_int": 9223372036854775102,
    "this_should_be_number": 2.1,
    "nested_object": {
        "title": "Just an nested object",
        "date": "2015-05-29T12:00:00+07:00" }}}

Second instance:

{ "event": {
    "just_a_string": "No particular format",
    "sometimes_ip": "This time it's not an IP",
    "always_ipv4": "",
    "id": 42,
    "very_big_int": 92102,
    "this_should_be_number": 201,
    "not_always_here": 32,
    "nested_object": {
        "title": "Still plain string without format",
        "date": "1961-07-03T12:00:00+07:00" }}}

The generated schema:

{ "type" : "object",
  "properties" : {
    "event" : {
      "type" : "object",
      "properties" : {
        "just_a_string" : { "type" : "string" },
        "sometimes_ip" : { "type" : "string" },
        "always_ipv4" : {
          "type" : "string",
          "format" : "ipv4" },
        "id" : {
          "type" : "integer",
          "minimum" : 0,
          "maximum" : 32767 },
        "very_big_int" : {
          "type" : "integer",
          "minimum" : 0,
          "maximum" : 9223372036854775807 },
        "this_should_be_number" : {
          "type" : "number",
          "minimum" : 0 },
        "nested_object" : {
          "type" : "object",
          "properties" : {
            "title" : { "type" : "string" },
            "date" : {
              "type" : "string",
              "format" : "date-time" } },
          "additionalProperties" : false },
        "not_always_here" : {
          "type" : "integer",
          "minimum" : 0,
          "maximum" : 32767 } },
      "additionalProperties" : false } },
  "additionalProperties" : false }

Copyright and License

Schema Guru is copyright 2014-2016 Snowplow Analytics Ltd.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this software except in compliance with the License.

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.