A quick way to get started with Geoscape's open GNAF & Admin Boundaries
IMPORTANT: There have been significant changes in the August 2021 release. See the CHANGELOG for details
A quick way to load the complete Geocoded National Address File of Australia (GNAF) and Australian Administrative Boundaries into Postgres, simplified and ready to use as reference data for geocoding, analysis, visualisation and aggregation.
Running the Python script takes 30-120 minutes on a Postgres server configured to take advantage of the RAM available.
To get a good load time you'll need to configure your Postgres server for performance. There's a good guide here, noting it's a few years old and some of the memory parameters can be beefed up if you have the RAM.
CREATE EXTENSION postgis
-hargument (see command line examples below)
The behaviour of gnaf-loader can be controlled by specifying various command line options to the script. Supported arguments are:
--gnaf-tables-pathspecifies the path to the extracted source GNAF tables (eg *.psv files). This should match the extracted directory which contains the subfolders
Standard. This directory must be accessible by the Postgres server, and the corresponding local path for the server to this directory may need to be set via the
--local-server-dirspecifies the local path on the Postgres server corresponding to
gnaf-tables-path. If the server is running locally this argument can be omitted.
--admin-bdys-pathspecifies the path to the extracted source admin boundary files. This path should contain a subfolder named
Administrative Boundaries. Unlike
gnaf-tables-path, this path does not necessarily have to be accessible to the remote Postgres server.
--pghostthe host name for the Postgres server. This defaults to the
PGHOSTenvironment variable if set, otherwise defaults to
--pgportthe port number for the Postgres server. This defaults to the
PGPORTenvironment variable if set, otherwise
--pgdbthe database name for Postgres server. This defaults to the
PGDATABASEenvironment variable if set, otherwise
--pguserthe username for accessing the Postgres server. This defaults to the
PGUSERenvironment variable if set, otherwise
--pgpasswordpassword for accessing the Postgres server. This defaults to the
PGPASSWORDenvironment variable if set, otherwise
--geoscape-versionGeoscape version number in YYYYMM format. Defaults to current year and last release month. e.g.
--raw-gnaf-schemaschema name to store raw GNAF tables in. Defaults to
--raw-admin-schemaschema name to store raw admin boundary tables in. Defaults to
--gnaf-schemadestination schema name to store final GNAF tables in. Defaults to
--admin-schemadestination schema name to store final admin boundary tables in. Defaults to
--statesspace separated list of states to load, eg
--states VIC TAS. Defaults to loading all states.
--prevacuumforces the database to be vacuumed after dropping tables. Defaults to off, and specifying this option will slow the import process.
--raw-fkcreates both primary & foreign keys for the raw GNAF tables. Defaults to off, and will slow the import process if specified. Use this option if you intend to utilise the raw GNAF tables as anything more then a temporary import step. Note that the final processed tables will always have appropriate primary and foreign keys set.
--raw-unloggedcreates unlogged raw GNAF tables, speeding up the import. Defaults to off. Only specify this option if you don't care about the raw data tables after the import - they will be lost if the server crashes!
--max-processesspecifies the maximum number of parallel processes to use for the data load. Set this to the number of cores on the Postgres server minus 2, but limit to 12 if 16+ cores - there is minimal benefit beyond 12. Defaults to 4.
--no-boundary-tagDO NOT tag all addresses with some of the key admin boundary IDs for creating aggregates and choropleth maps.
python load-gnaf.py --gnaf-tables-path="C:\temp\geoscape_202108\G-NAF" --admin-bdys-path="C:\temp\geoscape_202108\Administrative Boundaries"Loads the GNAF tables to a Postgres server running locally. GNAF archives have been extracted to the folder
C:\temp\geoscape_202108\G-NAF, and admin boundaries have been extracted to the
python load-gnaf.py --gnaf-tables-path="\\svr\shared\gnaf" --local-server-dir="f:\shared\gnaf" --admin-bdys-path="c:\temp\unzipped\AdminBounds_ESRI"Loads the GNAF tables which have been extracted to the shared folder
\\svr\shared\gnaf. This shared folder corresponds to the local
f:\shared\gnaffolder on the Postgres server. Admin boundaries have been extracted to the
python load-gnaf.py --states VIC TAS NT ...Loads only the data for Victoria, Tasmania and Northern Territory
You can load the Admin Boundaries without GNAF. To do this: comment out steps 1, 3 and 4 in def main.
Note: you can't load GNAF without the Admin Bdys due to dependencies required to split Melbourne and to fix non-boundary locality_pids on addresses.
When using the resulting data from this process - you will need to adhere to the attribution requirements on the data.gov.au pages for GNAF and the Admin Bdys, as part of the open data licensing requirements.
GNAF and the Admin Boundaries are ready to use in Postgres in an image on Docker Hub.
docker pull minus34/gnafloader:latest
docker run --publish=5433:5432 minus34/gnafloader:latest
5433. Default login is - user:
Note: the compressed Docker image is 6Gb, uncompressed is 16Gb
WARNING: The default postgres superuser password is insecure and should be changed using:
ALTER USER postgres PASSWORD ''
Download Postgres dump files and restore them in your database.
Should take 15-60 minutes.
Incorporates or developed using G-NAF © Geoscape Australia licensed by the Commonwealth of Australia under the Open Geo-coded National Address File (G-NAF) End User Licence Agreement.
Incorporates or developed using Administrative Boundaries © Geoscape Australia licensed by the Commonwealth of Australia under Creative Commons Attribution 4.0 International licence (CC BY 4.0).
GNAF and the Admin Bdys have been customised to remove some of the known, minor limitations with the data. The most notable are: - All addresses link to a gazetted locality that has a boundary. Those small number of addresses that don't in raw GNAF have had their localitypid changed to a gazetted equivalent - Localities have had address and street counts added to them - Suburb-Locality bdys have been flattened into a single continuous layer of localities - South Australian Hundreds have been removed and ACT districts have been added where there are no gazetted localities - The Melbourne, VIC locality has been split into Melbourne, 3000 and Melbourne 3004 localities (the new locality PIDs are `loc9901d119afda1
&loc9901d119afda_2`). The split occurs at the Yarra River (based on the postcodes in the Melbourne addresses) - A postcode boundaries layer has been created using the postcodes in the address tables. Whilst this closely emulates the official Geoscape postcode boundaries, there are several hundred addresses that are in the wrong postcode bdy. Do not treat this data as authoritative