Tools to turn the schema into other formats
The OpenMapTiles toolbox for generating map vector tiles. It includes tools to prepare Imposm mappings and SQL files based on layers defined in OpenMapTiles or similar projects. It also includes map data downloading, parsing, debugging, and performance evaluation tools. We encourage other people to use this for their vector tile projects as well since this approach works well for us.
A collection of tools for downloading, parsing, and generating map tiles described below.
Multiple data sources packaged for import into PostgreSQL DB, includes data from Natural Earth, water polygons, and lake centerlines.
An image with PostgreSQL database, Postgis, and several other extensions, custom built for OpenMapTiles project.
The above
postgisimage pre-loaded with the
import-data. This image is mostly used for testing, and may not be appropriate for production. The image has hardcoded user
openmaptilesand password
openmaptiles.
Legacy Mapnik-based image that simplifies
tilelive-copytile generation. Eventually will be replaced with PostgreSQL-based ST_AsMVT approach.
You need either just Docker or Python 3 installed on your system. You also need Docker-compose for testing. If running without Docker, see below for the list of additional tools and libraries.
The easiest is to use docker directly to run this command. You do not need to clone
openmaptiles-toolslocally, just clone the openmaptiles repo and run from its root.
Note: container scripts can only access files from the given directory and below, e.g. in this example -
${PWD}- current dir.
bash docker run -it --rm -u $(id -u ${USER}):$(id -g ${USER}) \ -v "${PWD}:/tileset" \ openmaptiles/openmaptiles-tools \
Where the
could be any of the scripts in the bin/ directory, e.g.generate-imposm3 openmaptiles.yaml.
/importfolder
mapping.yamlinto the
/mappingfolder
/cachefolder for later reuse
# Some tool require these packages. On Debian/Ubuntu you can install them with sudo apt install graphviz sqlite3 aria2 osmctoolsinstall the package directly from git
python3 -m pip install git+https://github.com/openmaptiles/openmaptiles-tools
Run the script you want, e.g. from the openmaptiles dir:
generate-imposm3 openmaptiles.yaml
If the script doesn't run, make sure your PATH includes default PIP bin directory.
On Debian/Ubuntu that would be ~/.local/bin/ (under your home dir).
Otherwise just run it with ~/.local/bin/generate-imposm3 ...
Make sure you have all dependencies from the Usage section above. You should have the latest Python (3.6+) ```bash
git clone https://github.com/openmaptiles/openmaptiles.git
git clone https://github.com/openmaptiles/openmaptiles-tools.git
cd openmaptiles-tools
python3.6 -m pip install -r requirements.txt
PYTHONPATH=$PWD python3 bin/generate-imposm3 ../openmaptiles/openmaptiles.yaml ```
Use
make testto run all of the tests locally. The Makefile will build a docker image with all the code, run all tests, and compare the build result with the files in the testdata/expected dir.
Run
make rebuild-expectedafter you modify the output produced by the generation scripts. This will re-create the expected test results to match the actual ones, and make sure the changes are what you want.
You define a self contained Layer together with SQL files and layer and data source definitions (like an imposm3 mapping file) that you can then reference in a Tileset where you mix and match with other layers.
Take a look or copy a standard layer like building to get started with your own layer. A layer consists out of a Layer definition written in YAML format.
There you specify the
layerproperties like
id,
buffer_sizeand possible Markdown documentation (
descriptionand
fields). You can also reference SQL files in
schemafor writing the necessary queries for your layer or create generalized tables. We encourage you to have a function per layer which takes the bounding box and zoom level. This makes it easy to test and reuse.
If your data is based of OSM you can also directly reference a imposm3 mapping file to choose the OSM data you need.
layer: id: "building" description: Buildings from OpenStreetMap buffer_size: 4 datasource: query: (SELECT geometry, render_height, class FROM layer_building(!bbox!, z(!scale_denominator!))) AS t fields: render_height: An approximated height from levels and height of building. class: description: Defines a subclass of a building (one of the known values). # Values can be either a list of strings, or a dictionary # Dictionary defines mapping of OSM values to the OMT field value values: school: # subclass IN ('school','kindergarten') OR subclass LIKE 'uni%' subclass: ['school','kindergarten','uni%'] railway: # (subclass='station' AND mapping_key='railway') # OR subclass in ('halt','tram_stop','subway') - __AND__: subclass: 'station' mapping_key: 'railway' - subclass: ['halt', 'tram_stop', 'subway'] schema: - ./building.sql datasources: - type: imposm3 mapping_file: ./mapping.yaml
For the well known values (enums), the
fieldssection can also contain the mapping of the input (OSM) values.
If a layer SQL files contains
%%FIELD_MAPPING: class%%,
generate-sqlscript will replace it
SELECT CASE %%FIELD_MAPPING: class%% END, ...
into
sql SELECT CASE WHEN "subclass" IN ('school', 'kindergarten') OR "subclass" LIKE 'uni%' THEN 'school' WHEN ("subclass" = 'station' AND "mapping_key" = 'railway') OR "subclass" in ('halt','tram_stop','subway') THEN 'railway' END, ...
A Tileset defines which layer will be in your vector tile set (
layers) and metadata used for generating a TM2Source project to actually generate the vector tiles.
tileset: layers: - layers/building/building.yaml - layers/housenumber/housenumber.yaml - layers/poi/poi.yaml name: Street Level description: A tileset showing street level info like building, housenumbers and POIs. attribution: "OpenStreetMap contributors" maxzoom: 14 minzoom: 13 center: [-12.2168, 28.6135, 4] bounds: [-180.0,-85.0511,180.0,85.0511] pixel_scale: 256 defaults: srs: +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m [email protected] +wktext +no_defs +over datasource: srid: 900913
Use
test-perfto evaluate tile generation performance against a PostgreSQL database. This utility can test individual layers, several layers at once, as well as the whole tile. It has several pre-defined testing areas to provide cross-test consistency. Results are printed as histogram graphs, showing tile size distribution. The utility can compare current run with a previously saved one, highlighting large changes. If multiple zoom levels are tested, another histogram shows per-zoom size distribution. Run with
--helpto see all options.
test-perf ...
Just like
postservebelow,
test-perfrequires PostgreSQL connection.
Postserve is an OpenMapTiles map vector tile test server that dynamically generates metadata and tiles directly from PostgreSQL database based on the tileset file definition.
postserve ...
Use
postserveto start serving. Use
--helpto get the list of Postgres connection parameters. If you have a full planet database, you may want to use
MIN_ZOOM=6 postserve ...to avoid accidental slow low-zoom tile generation.
openmaptiles-toolsrepo is not needed with docker)
docker pull openmaptiles/openmaptiles-toolsto download the latest tools version
--net=hostto
--net=openmaptiles_postgres_connto match the openmaptiles quickstart, and also expose port 8090 to the host with
-p 8090:8090)
docker run -it --rm -u $(id -u ${USER}):$(id -g ${USER}) \ -v "${PWD}:/tileset" --net=host \ openmaptiles/openmaptiles-tools \ postserve openmaptiles.yaml
You can view tiles with any MVT-supporting viewer, such as: * Maputnik editor (online) -- change the data source to
http://localhost:8090* QGIS desktop -- add
Vector Tiles Readerplugin, and add a vector tile server connection with TileJSON URL set to
http://localhost:8090.
Use
debug-mvttool to examine tile content. The tool will query PostgreSQL server and show layers with each data row and geometry type/size. This tool can limit output to just a few layers, optionally show all localized names, and show geometries as text. This example queries shows entries in the "place" layer for the specified tile:
$ debug-mvt openmaptiles.yaml 4/7/6 -l place ======= Layer place (extra name columns are hidden by default) ======= capital class iso_a2 name name_de name_en rank osm_id is_valid_mvt mvtgeometry is_valid_geom geometry --------- ------- -------- -------------------------- -------------------------- -------------------------- ------ ----------- -------------- ------------- --------------- ---------- country ES Espaa Spanien Spain 1 1483323000 True POINT(32) True POINT(32) country PT Portugal Portugal Portugal 1 23770282470 True POINT(32) True POINT(32) 4 city Casablanca Casablanca Casablanca 2 257307240 True POINT(32) True POINT(32) country MA Maroc Marokko Morocco 2 4324250410 True POINT(32) True POINT(32) 2 city Madrid Madrid Madrid 2 210682950 True POINT(32) True POINT(32) 2 city Lisboa Lissabon Lisbon 3 2659584900 True POINT(32) True POINT(32) 4 city Oran Oran Oran 4 275651030 True POINT(32) True POINT(32) 4 city Valncia Valencia Valencia 4 341056070 True POINT(32) True POINT(32) 4 city Sevilla Sevilla Seville 4 2488206560 True POINT(32) True POINT(32) 4 city Fs Fs Fez 4 2890354320 True POINT(32) True POINT(32) 2 city Rabat Rabat Rabat 4 2991208620 True POINT(32) True POINT(32) ...
Use
profile-pg-functo compare PostgreSQL function execution speed. Each function is called thousands of times in several runs. The fastest and slowest runs are discarded.
profile-pg-funccan import SQL files before running the test, e.g. to add the latest developer versions of the function(s).
Use
layer-statsshow per zoom statistics for some column (field) in a single layer. Supports several metrics: *
frequency- Shows how often each unique value occurs in a layer's column or combination of columns. *
toplength- Shows the longest N values for a given layer's column. *
variance- Shows a few statistical metrics for a column's numeric value.
Various tools require these environment variables to be set
SQL_DIR/sql
PBF_DATA_DIR/import
IMPOSM_CACHE_DIR/cache
IMPOSM_MAPPING_FILE/mapping/mapping.yaml
IMPOSM_DIFF_DIR/import
EXPIRETILES_DIRused by
import-updatefor Imposm's
-expiretiles-dirparameter
EXPIRETILES_ZOOMdefaults to 14
download-osmtool can be used to download an area extract or the entire planet file, and validate file content. The entire planet file is downloaded from all available OSM mirrors at the same time to distribute the load across mirrors, and makes download faster. The tool will ensure you get the latest version, verifies that all mirrors contain the same version, and validates the download with md5 hash. By default the tool will not download from the primary OSM site to reduce its load. Downloader uses aria2c. Downloader can also get files from Geofabrik, BBBike, and openstreemap.fr, or an arbitrary URL.
# downloads the latest version to the current directory download-osm planetdownload to the target dir by passing -d ... param to aria2c
download-osm planet -- -d ./downloads
download New Zealand extract from Geofabrik, together with the state file
download-osm geofabrik new-zealand --state state.txt
List all extracts available from Geofabrik
download-osm list geofabrik
Uses tileset definition to create a PostgreSQL prepared or create function SQL code to generate an entire vector tile in the Mapbox Vector Tile format with a single
getTile(z,x,y)query using PostGIS MVT support.
Use
--helpto get all parameters.
NOTE: Current openmaptiles/postgis image (v2.9 and before) has incorrect support for the ST_AsMVT(). Until Postgis is updated, please use sophox/postgis docker image (based on the latest mdillon/postgis:11 base image). Another known bug is PostgreSQL JIT could make tile generation horribly slow in PG11+, and may need to be disabled.
generate-sqltomvt
Takes a tileset definition an generates an imposm3 mapping file for importing OSM data.
generate-imposm3
Assembles all SQL referenced in the layer definitions into an SQL script that can be executed with psql. If
--diroption is given, generates
.sqlfiles that can be executed in parallel.
generate-sql generate-sql --dir
Takes a tileset definition and generates Markdown documentation.
generate-doc
dependency: graphviz
Takes a source code from the imposm3 mapping file and the SQL postprocessing code, and parsing for the
etldoc:graphviz based comments, and generate an svg file. The
.dotand the
.svgfilename prefix is
etl_
generate-etlgraph generate-etlgraph layers/landcover/landcover.yaml ./build/devdoc generate-etlgraph layers/railway/railway.yaml ./build/etlgraph
example:
input command:
generate-etlgraph layers/landcover/landcover.yamloutput fies: -
layers/landcover/etl_landcover.dot-
layers/landcover/etl_landcover.svg
example:
generate-sqlquery layers/landcover/landcover.yaml 14
The
import-osm,
import-update, and
import-difftools will import and update PostgreSQL database by running imposm. The tools expect these env vars:
PGHOST,
PGDATABASE,
PGUSER,
PGPASSWORD, and optionally
PGPORTto connect to the PostgreSQL server, and a number of other vars for imposm configuration. See scripts.
The
import-wikidatatool searches for all wikidata tags in the database, and uses Wikidata Query Service to get the labels in all languages. This tool looks at all tables defined in layers' mapping files if they contain
tagshstore field, and if the table has no
_resolve_wikidata: falseflag. The localized names are written to the
wd_namestable, which will be created if missing.
import-wikidata openmaptiles.yaml
This command allows users to examine and manipulate mbtiles file: * generate metadata based on the layers definition * get, set, and delete individual metadata values * validate and print all metadata values and mbtiles file statistics * list all tile keys (hashes) that are used many times (usually indicates empty tiles) * copy zooms, e.g. copy all empty tiles z13 to z14, and create a list of all tiles that needs to be generated.
mbtiles-tools --help mbtiles-tools ./data/tiles.mbtiles meta-all
Takes a tileset definition and generates a TM2Source YAML project file. You need to provide PostgreSQL database connection settings before generating the project.
generate-tm2source --host="localhost" --port=5432 --database="osm" --user="osm" --password="osm"
The import-borders script will take the first PBF file from the
/importdir (by default), extract borders with osmborder tool, and import resulting CSV file into the database as osmborderlinestring table (by default). Example usages:
import-borders # Parse and import first PBF file in PBF_DATA_DIR import-borders [import] planet.pbf # Parse and import planet.pbf import-borders parse planet.pbf # Parse planet.pbf into a CSV file, but do not import import-borders load borders.csv # Load borders.csv into a table
This utility requires PostgreSQL's PG* environment variables, and optionally uses
PBF_DATA_DIR, BORDERS_PBF_FILE, BORDERS_CSV_FILE, BORDERS_TABLE_NAME.
The
import-sqlscript can execute a single SQL file in Postgres when the file is given as the first parameter.
If ran without any arguments,
import-sqlexecutes all of the following: * SQL files from
$SQL_TOOLS_DIR- contains all SQL files to be imported before the ones generated by OpenMapTiles project. By default, contains OMT fork of the Mapbox's postgis-vt-util.sql helper functions and contains the sql/language.sql script. (as
10_postgis-vt-util.sql- the file name has to be imported before files in
/sql) * SQL files from
$SQL_DIR- defaults to
/sql-- this directory should contain the results of the
generate-sql --dir ...script and possibly other user-defined sql scripts. If this directory contains
parallel/subdirectory,
import-sqlwill assume the parallel/*.sql files are safe to execute in parallel, up to
MAX_PARALLEL_PSQLat a time (defaults to 5). The script will also execute
run_first.sqlbefore, and
run_last.sqlafter the files in
parallel/dir (if they exist).
Generating and importing SQL could be done in a single step with
&&, e.g.
generate-sqltomvt openmaptiles.yaml > "$SQL_DIR/mvt.sql" && import-sql
Optionally you may pass extra arguments to
psqlby using
PSQL_OPTIONSenvironment variable. For example
PSQL_OPTIONS=-amakes psql echo all commands read from a file into stdout.
PSQL_OPTIONSallows multiple arguments as well, and understands quotes, e.g. you can pass a whole query as a single argument surrounded by quotes --
PSQL_OPTIONS="-a -c 'SELECT ...'"
Most PostgreSQL-related images support standard PostgreSQL environment variables like
PGUSER,
PGPASSWORD,
PGHOST,
PGDATABASE, and optionally
PGPORT.
postgisimage, which uses different variables (
POSTGRES_USER,
POSTGRES_PASSWORD,
POSTGRES_HOST,
POSTGRES_DB,
POSTGRES_PORT) during database creation due to how the official Docker postgres image has been set up.
Materialized views can be refreshed in parallel using
refresh-viewscommand. This could be especially useful if the
CREATE MATERIALIZED VIEWstatements had
WITH NO DATAclause.