OpenMapTiles Tools

The OpenMapTiles toolbox for generating map vector tiles. It includes tools to prepare Imposm mappings and SQL files based on layers defined in OpenMapTiles or similar projects. It also includes map data downloading, parsing, debugging, and performance evaluation tools. We encourage other people to use this for their vector tile projects as well since this approach works well for us.

Docker Images

openmaptiles-tools

A collection of tools for downloading, parsing, and generating map tiles described below.

import-data

Multiple data sources packaged for import into PostgreSQL DB, includes data from Natural Earth, water polygons, and lake centerlines.

postgis

An image with PostgreSQL database, Postgis, and several other extensions, custom built for OpenMapTiles project.

postgis-preloaded

The above

postgis

import-data

openmaptiles

openmaptiles

generate-vectortiles

Legacy Mapnik-based image that simplifies

tilelive-copy

Usage

You need either just Docker or Python 3 installed on your system. You also need Docker-compose for testing. If running without Docker, see below for the list of additional tools and libraries.

Usage with Docker

The easiest is to use docker directly to run this command. You do not need to clone

openmaptiles-tools

Note: container scripts can only access files from the given directory and below, e.g. in this example -

${PWD}

bash
docker run -it --rm -u $(id -u ${USER}):$(id -g ${USER}) \
           -v "${PWD}:/tileset" \
           openmaptiles/openmaptiles-tools \
            

Where the

generate-imposm3 openmaptiles.yaml

Docker Volumes

• Mount your PBFs into the

  /import

  folder
• Mount your

  mapping.yaml

  into the

  /mapping

  folder
• If you want to use diff mode mount a persistent location to the

  /cache

  folder for later reuse

Using without Docker

# Some tool require these packages. On Debian/Ubuntu you can install them with
sudo apt install graphviz sqlite3 aria2 osmctools

install the package directly from git
python3 -m pip install git+https://github.com/openmaptiles/openmaptiles-tools
Run the script you want, e.g. from the openmaptiles dir:
generate-imposm3 openmaptiles.yaml
If the script doesn't run, make sure your PATH includes default PIP bin directory.
On Debian/Ubuntu that would be  ~/.local/bin/  (under your home dir).
Otherwise just run it with  ~/.local/bin/generate-imposm3 ...

Running from source

Make sure you have all dependencies from the Usage section above. You should have the latest Python (3.6+) ```bash

Get OpenMapTiles layer data

git clone https://github.com/openmaptiles/openmaptiles.git

Get the tools repo

git clone https://github.com/openmaptiles/openmaptiles-tools.git

Run scripts from the root of the tools repository

cd openmaptiles-tools

Install required dependencies (might need to run with sudo

python3.6 -m pip install -r requirements.txt

The PYTHONPATH=$PWD allows script to find its modules located in the current dir.

PYTHONPATH=$PWD python3 bin/generate-imposm3 ../openmaptiles/openmaptiles.yaml ```

Tools Development

Use

make test

Run

make rebuild-expected

Data Concepts

You define a self contained Layer together with SQL files and layer and data source definitions (like an imposm3 mapping file) that you can then reference in a Tileset where you mix and match with other layers.

Define your own Layer

Take a look or copy a standard layer like building to get started with your own layer. A layer consists out of a Layer definition written in YAML format.

There you specify the

layer

id

buffer_size

description

fields

schema

If your data is based of OSM you can also directly reference a imposm3 mapping file to choose the OSM data you need.

layer:
  id: "building"
  description: Buildings from OpenStreetMap
  buffer_size: 4
  datasource:
    query: (SELECT geometry, render_height, class FROM layer_building(!bbox!, z(!scale_denominator!))) AS t
  fields:
    render_height: An approximated height from levels and height of building.
    class:
      description: Defines a subclass of a building (one of the known values).
      # Values can be either a list of strings, or a dictionary
      # Dictionary defines mapping of OSM values to the OMT field value
      values:
        school:
          # subclass IN ('school','kindergarten') OR subclass LIKE 'uni%'
          subclass: ['school','kindergarten','uni%']
        railway:
          # (subclass='station' AND mapping_key='railway')
          # OR subclass in ('halt','tram_stop','subway')
          - __AND__:
              subclass: 'station'
              mapping_key: 'railway'
          - subclass: ['halt', 'tram_stop', 'subway']
schema:
  - ./building.sql
datasources:
  - type: imposm3
    mapping_file: ./mapping.yaml

For the well known values (enums), the

fields

If a layer SQL files contains

%%FIELD_MAPPING: class%%

generate-sql

SELECT CASE
    %%FIELD_MAPPING: class%%
END, ...

into

sql
SELECT CASE
    WHEN "subclass" IN ('school', 'kindergarten')
        OR "subclass" LIKE 'uni%' THEN 'school'
    WHEN ("subclass" = 'station' AND "mapping_key" = 'railway')
        OR "subclass" in ('halt','tram_stop','subway') THEN 'railway'
END, ...

Define your own Tileset

A Tileset defines which layer will be in your vector tile set (

layers

tileset:
  layers:
    - layers/building/building.yaml
    - layers/housenumber/housenumber.yaml
    - layers/poi/poi.yaml
  name: Street Level
  description: A tileset showing street level info like building, housenumbers and POIs.
  attribution: "OpenStreetMap contributors"
  maxzoom: 14
  minzoom: 13
  center: [-12.2168, 28.6135, 4]
  bounds: [-180.0,-85.0511,180.0,85.0511]
  pixel_scale: 256
  defaults:
    srs: +proj=merc +a=6378137 +b=6378137 +lat_ts=0.0 +lon_0=0.0 +x_0=0.0 +y_0=0.0 +k=1.0 +units=m [email protected] +wktext +no_defs +over
    datasource:
      srid: 900913

Testing tiles

Tile size and PostgreSQL querying speed

Use

test-perf

--help

test-perf  ...

Just like

postserve

test-perf

Realtime tile server

Postserve is an OpenMapTiles map vector tile test server that dynamically generates metadata and tiles directly from PostgreSQL database based on the tileset file definition.

postserve  ...

Use

postserve 

--help

MIN_ZOOM=6 postserve ...

Postserve quickstart with docker

• clone openmaptiles repo (

  openmaptiles-tools

  repo is not needed with docker)
• get a PostgreSQL server running with the openmaptiles-imported OSM data, e.g. by following quickstart guide.
• run

  docker pull openmaptiles/openmaptiles-tools

  to download the latest tools version
• from inside the openmaptiles repo dir, run this command. (This assumes PostgreSQL is on the localhost:5432, but if it runs inside docker, you may want to change

  --net=host

  to

  --net=openmaptiles_postgres_conn

  to match the openmaptiles quickstart, and also expose port 8090 to the host with

  -p 8090:8090

  )

  docker run -it --rm -u $(id -u ${USER}):$(id -g ${USER}) \
  -v "${PWD}:/tileset" --net=host \
  openmaptiles/openmaptiles-tools \
  postserve openmaptiles.yaml
  

Viewing dynamic tiles

You can view tiles with any MVT-supporting viewer, such as: * Maputnik editor (online) -- change the data source to

http://localhost:8090

Vector Tiles Reader

http://localhost:8090

Examining realtime tile content

Use

debug-mvt

$ debug-mvt openmaptiles.yaml 4/7/6 -l place
======= Layer place (extra name columns are hidden by default) =======
  capital  class    iso_a2    name                        name_de                     name_en                       rank       osm_id  is_valid_mvt    mvtgeometry    is_valid_geom    geometry
---------  -------  --------  --------------------------  --------------------------  --------------------------  ------  -----------  --------------  -------------  ---------------  ----------
           country  ES        Espaa                       Spanien                     Spain                            1   1483323000  True            POINT(32)      True             POINT(32)
           country  PT        Portugal                    Portugal                    Portugal                         1  23770282470  True            POINT(32)      True             POINT(32)
        4  city               Casablanca                  Casablanca                  Casablanca                       2    257307240  True            POINT(32)      True             POINT(32)
           country  MA        Maroc                       Marokko                     Morocco                          2   4324250410  True            POINT(32)      True             POINT(32)
        2  city               Madrid                      Madrid                      Madrid                           2    210682950  True            POINT(32)      True             POINT(32)
        2  city               Lisboa                      Lissabon                    Lisbon                           3   2659584900  True            POINT(32)      True             POINT(32)
        4  city               Oran                        Oran                        Oran                             4    275651030  True            POINT(32)      True             POINT(32)
        4  city               Valncia                     Valencia                    Valencia                         4    341056070  True            POINT(32)      True             POINT(32)
        4  city               Sevilla                     Sevilla                     Seville                          4   2488206560  True            POINT(32)      True             POINT(32)
        4  city               Fs                          Fs                          Fez                              4   2890354320  True            POINT(32)      True             POINT(32)
        2  city               Rabat                       Rabat                       Rabat                            4   2991208620  True            POINT(32)      True             POINT(32)
        ...

Profile PostgreSQL functions

Use

profile-pg-func

profile-pg-func

Show layer statistics for a field

Use

layer-stats

frequency

toplength

variance

Tools

Environment variables

Various tools require these environment variables to be set

• SQL_DIR

  /sql

• PBF_DATA_DIR

  /import

• IMPOSM_CACHE_DIR

  /cache

• IMPOSM_MAPPING_FILE

  /mapping/mapping.yaml

• IMPOSM_DIFF_DIR

  /import

• EXPIRETILES_DIR

  used by

  import-update

  for Imposm's

  -expiretiles-dir

  parameter

• EXPIRETILES_ZOOM

  defaults to 14

Multi-streamed OSM Data Downloader

download-osm

# downloads the latest version to the current directory
download-osm planet

download to the target dir by passing -d ... param to aria2c
download-osm planet -- -d ./downloads
download New Zealand extract from Geofabrik, together with the state file
download-osm geofabrik new-zealand --state state.txt
List all extracts available from Geofabrik
download-osm list geofabrik

Generate SQL code to create MVT tiles directly by PostGIS

Uses tileset definition to create a PostgreSQL prepared or create function SQL code to generate an entire vector tile in the Mapbox Vector Tile format with a single

getTile(z,x,y)

Use

--help

NOTE: Current openmaptiles/postgis image (v2.9 and before) has incorrect support for the ST_AsMVT(). Until Postgis is updated, please use sophox/postgis docker image (based on the latest mdillon/postgis:11 base image). Another known bug is PostgreSQL JIT could make tile generation horribly slow in PG11+, and may need to be disabled.

generate-sqltomvt 

Generate Imposm3 Mapping File

Takes a tileset definition an generates an imposm3 mapping file for importing OSM data.

generate-imposm3 

Generate SQL scripts

Assembles all SQL referenced in the layer definitions into an SQL script that can be executed with psql. If

--dir

.sql

generate-sql 
generate-sql  --dir 

Generate Markdown Documentation

Takes a tileset definition and generates Markdown documentation.

generate-doc 

Generate ETL (Extract-Transform-Load) graph

dependency: graphviz

Takes a source code from the imposm3 mapping file and the SQL postprocessing code, and parsing for the

etldoc:

.dot

.svg

etl_

generate-etlgraph   
generate-etlgraph layers/landcover/landcover.yaml  ./build/devdoc
generate-etlgraph layers/railway/railway.yaml      ./build/etlgraph

example:

input command:

generate-etlgraph layers/landcover/landcover.yaml

layers/landcover/etl_landcover.dot

layers/landcover/etl_landcover.svg

Generate SQL query for a given layer and zoom level

example:

generate-sqlquery layers/landcover/landcover.yaml  14

Import and Update OSM data

The

import-osm

import-update

import-diff

PGHOST

PGDATABASE

PGUSER

PGPASSWORD

PGPORT

Import Wikidata localized names

The

import-wikidata

tags

_resolve_wikidata: false

wd_names

import-wikidata openmaptiles.yaml

Mbtiles file tools

This command allows users to examine and manipulate mbtiles file: * generate metadata based on the layers definition * get, set, and delete individual metadata values * validate and print all metadata values and mbtiles file statistics * list all tile keys (hashes) that are used many times (usually indicates empty tiles) * copy zooms, e.g. copy all empty tiles z13 to z14, and create a list of all tiles that needs to be generated.

mbtiles-tools --help
mbtiles-tools ./data/tiles.mbtiles meta-all

Generate TM2Source Projects for Mapbox Studio Classic

Takes a tileset definition and generates a TM2Source YAML project file. You need to provide PostgreSQL database connection settings before generating the project.

generate-tm2source  --host="localhost" --port=5432 --database="osm" --user="osm" --password="osm"

Import OSM Borders

The import-borders script will take the first PBF file from the

/import

import-borders                      # Parse and import first PBF file in PBF_DATA_DIR
import-borders [import] planet.pbf  # Parse and import planet.pbf
import-borders parse planet.pbf     # Parse planet.pbf into a CSV file, but do not import
import-borders load borders.csv     # Load borders.csv into a table

This utility requires PostgreSQL's PG* environment variables, and optionally uses

PBF_DATA_DIR, BORDERS_PBF_FILE, BORDERS_CSV_FILE, BORDERS_TABLE_NAME

Importing into Postgres

The

import-sql

If ran without any arguments,

import-sql

$SQL_TOOLS_DIR

10_postgis-vt-util.sql

/sql

$SQL_DIR

/sql

generate-sql --dir ...

parallel/

import-sql

MAX_PARALLEL_PSQL

run_first.sql

run_last.sql

parallel/

Generating and importing SQL could be done in a single step with

&&

generate-sqltomvt openmaptiles.yaml > "$SQL_DIR/mvt.sql" && import-sql

Optionally you may pass extra arguments to

psql

PSQL_OPTIONS

PSQL_OPTIONS=-a

PSQL_OPTIONS

PSQL_OPTIONS="-a -c 'SELECT ...'"

Environment variables

Most PostgreSQL-related images support standard PostgreSQL environment variables like

PGUSER

PGPASSWORD

PGHOST

PGDATABASE

PGPORT

The only exception is the

postgis

POSTGRES_USER

POSTGRES_PASSWORD

POSTGRES_HOST

POSTGRES_DB

POSTGRES_PORT

Performance optimizations

Materialized views can be refreshed in parallel using

refresh-views

CREATE MATERIALIZED VIEW

WITH NO DATA