Vector Map Creation Pipeline
This document will guide you through using MapShop to create a vector map for use with the Altus Mapping Engine. MapShop has not been packaged for a public release yet and there are a few areas that require manual setup and configuration. In addition, it would help if you have some familiarity with PostgreSQL / PostGIS and are comfortable writing basic SQL queries.
If you work through this document you should be able to get up and running with MapShop, import some vector data, style it, and generate an extremely compact vector map for use on-device.
Altus Mapping Engine — Maps Overview
The Altus Mapping Engine supports several different types of map layers:Terrain
- Streaming (via TileProvider)
- Animated (via Tileprovider)
- METool-generated / MapShop-generated
- Streaming (via VectorTileProvider in MapShop)
- Clustered — METool-generated
- Clustered — Altus Mapping Engine-generated
METool-generated map types are created by having METool process data ‘offline’ and create a highly-optimized map for the mapping engine. The final product here is either a header file (.sqlite) and data file (.map) or, in the case of markers, just a .sqlite file. Dynamic map types are created at run time with data you feed into the engine via the API. These live only in memory. One map type can be created on-device and saved to a file just as if METool had generated it: Clustered Markers.
MapShop can open METool generated Terrain, Raster, and Vector maps (at present) and also serves as a tool for iterating on vector map design.
This document focuses on 3 of these types:
- Vector — Streaming (via VectorTileProvider in MapShop)
- Vector — METool-generated / MapShop-generated
- Marker — Clustered, METool-generated clustered
Altus MapShop Installation Overview
The Altus Mapping Engine vector map creation pipeline involves installing and configuring several pieces of software:
- PostgreSQL relational database server
- PostgreSQL client libraries
- PostGIS extensions for PostgreSQL
- BA3 Altus MapShop (and dependencies)
- BA3 METool (and dependencies)
- At least one SQLite database editor (i.e. Base)
NOTE: Presently, there is no installer or script to aid in the installation of these components; it must be done by hand. This document covers manually installing and configuring each component on Mac OSX Mountain Lion so you can get up and running designing your own vector map content. As the pipeline matures we’ll work to make things like this more automated.
Altus MapShop Pipeline Overview
The pipeline generally functions according to this series of steps:
- Import your source vector data into a PostGIS database such that your vector data is represented as a geometry column.
- ‘Point’ MapShop at the PostGIS database and a SQLite design session database. The design session database contains query strings and styling instructions. Styling instructions will later be ‘injected’ into the generated map by adding the styling table from the session database into the map header database.
- Add/Remove/Edit entries in the styling database and refresh the MapShop view. MapShop will construct queries into PostGIS and generate a non-optimized vector view of your data based on the styles you set. You iterate on this step until you are satisfied with the look/feel of your map.
- Generate an optimized version of your map down to a specific ‘Z’ level either using MapShop’s export feature, or METool. During this process, a data structures are generated for the bounds and ‘depth’ of your map based on your inputs and many queries are issued to the PostGIS database to intersect/simplify/LOD source data against our internal representations. The LOD’s data is then optimized and compressed into the internal representation used by the mapping engine. The result of this process are 2 files: a .sqlite header file, and a .map data file.
- Embed the styling table used during the design session into the .sqlite map header file.
- Use your newly created vector map on device and mix and match it with other map types (markers, raster, terrain) to create a novel product for your customers.
NOTE: This pipeline has several moving parts and there are areas that can be problematic: for example if you create an invalid PostGIS query, or if your PostGIS database hangs for some reason, you may need to close MapShop, fix the offending query, and re-open MapShop and reconnect.
Altus MapShop Installation
- Ensure you have administrative access to your computer.
- Download and install GEOS from: http://dev1.ba3.us/public_sdk/3rdParty/geos-3.3.5.mpkg.zip
Install MapShop binaries:
- Download http://dev1.ba3.us/mapshop/MapShop.zip
- Extract the zip file
- You will see a folder named MapShop with these contents:
- These next steps you can do in Finder or in a command line tool, if you using a command line you will need to be the super user. If using Finder you will be prompted to enter your password:
- Drag MapShop/Applications/MapShop to your Applications folder
- Drag the folder MapShop/Library/Frameworks/ME.framework to /Library/Frameworks
- Copy the file MapShop/usr/local/lib to /usr/lib
You’ve now installed enough of MapShop to give it a try. Find and open the MapShop program, the icon looks like this in your Applications:
Run the program and you should see something like this on your screen:
Tap on the MapQuest Aerial button and it should download and display some imagery. (These tiles will be downloaded to ~/AltusTileCache):
If MapShop does not start it is either because it cannot find a prerequisite library, or you missed a step above, or you don’t have enough memory, or your video card is too old, or some other thing is happening we did not foresee. You can try running it from a command prompt and seeing what stdout says by opening a terminal and executing this command:
If all is well, that should show output like this:
MapShop[1961:707] ME: Initialization: coreCacheSize=90000000
NOTE: Running MapShop from a command prompt like this is a good way to see debug spew and may help with diagnosing future issues. Keep that in mind. For example, if at this point, you try to tap on one of the vector map buttons, MapShop will immediately exit. If you had run it with the command line, you would see these errors:
ME: Can't open database file '/Users/you/Dump/METoolConfig.sqlite'. The file does not exist.
[SystemConfig] Could not open: /Users/you/Dump/METoolConfig.sqlite
This is because we haven’t set up all the files necessary for vector map processing yet.
Configuring for MapShop Vector Map Processing
You need several more components to support vector map processing:
- PostgreSQL + PostGIS extensions
- A SQLite configuration database named METoolConfig.sqlite located at ~/Dump on your machine.
- A SQLite vector session database named MapShopSession.sqlite located in ~/Dump on your machine.
- Proper entries in the previous 2 SQLite databases
- Your source vector data imported into a PostGIS database that MapShop can read from.
Installing PostgreSQL + PostGIS
If you are expert IT type, feel free to install PostgreSQL + PostGIS however you like. Otherwise, we highly recommend if this is your first time installing PostgreSQL and PostGIS that you use pre-built binaries from William Kyngesburye hosted here: http://www.kyngchaos.com/software/postgres
Follow these steps:
- First download and install GDAL Complete from here:http://www.kyngchaos.com/files/software/frameworks/GDAL_Complete-1.10.dmg
- Next install PostgreSQL from here:http://www.kyngchaos.com/files/software/postgresql/PostgreSQL-9.2.4-2.dmg
- Next install PostGIS extensions for PostgreSQL from here:http://www.kyngchaos.com/files/software/postgresql/PostGIS-2.0.3-4.dmg
- If you don’t already have a way of administering PostgreSQL, do install pgAdmin III from here:http://ftp.postgresql.org/pub/pgadmin3/release/v1.16.1/osx/pgadmin3-1.16.1.dmg
Ensure that you can connect to your PostgreSQL data base form pgAdmin3 which will look something like this:
If you cannot connect to your PostgreSQL database, you’ll need to start investigating what went wrong. Each package above comes with several README files and there are a variety of issues that could prevent you from installing and connecting to PostgreSQL which are beyond the scope of this document.
Assuming everything up to this point has gone smoothly you can continue.
Create METool PostGIS Database
- In pgAdmin III create a new database named metool and open it.
- Drill down into the Extensions area. It should already have the plpgsql extension. We need to add a another.
- Right click on Extensions and select Add New...
- From the list select postgis and select OK.
- Once successfully installed your extensions list should look like this:
Create METool Login Role
- In pgAdmin III, create a new Login role named METool.
- In Role privileges grant Can login, and Superuser rights (this is strictly so security is not a blocker, feel free to come back and lock down this role later):
Get a decent SQLite Database Editor
With our current pipeline, you are going to spend a fair amount of time editing .sqlite files. As our tools mature, we will begin integrating some of this into MapShop, but for now, we suggest installing a program like Base:http://menial.co.uk/baseIt is also available on the App store.
- Install a decent SQLite editor
Install MapShop Config Files
- Create a folder inside your home folder named Dump. (MapShop looks for config files here and may generate output files here as well).
- Download, unzip this file:http://dev1.ba3.us/MapShopConfigFiles.zip
- Place the 4 .sqlite files you find into ~/Dump
These config files are configured to assume that:
- You have PostgreSQL running locally
- That you have a database named METool with PostGIS extensions installed
- That you have a Login Role named METool with superuser access and it’s password is METool
Now run MapShop again.
Now when you tap on the Vector Map 1 button, it should not crash. If it does, something isn’t configured correctly.
Creating a Simple Coastline Vector Map
Assuming you have followed all steps up until this point, we are now ready to create a simple vector map.
Get and Show Some Vector Data
- Download and extract this small already-generated ESRI shape file here of the Earth’s coastlines:http://www.naturalearthdata.com/http//www.naturalearthdata.com/download/110m/physical/ne_110m_coastline.zip
- Start MapShop
- Click File/Open... and point it at the ne_110m_coastline.shp file.
- You should see something that looks like this:
What has just happened is:
- MapShop used shp2pgsql and converted the .shp file for the coastlines into a .sql file named ne_110m_coastline.sql in your Dump folder.
- MapShop then executes this .sql file against the METool database thus creating a new table in the METool PostGIS database named ne_110m_coastline.
- MapShop added a vector map layer that points at the data in this new table.
- The vector map layer issues queries to PostGIS and renders the data with no styling applied.
- You now see coastline data on the sphere.
If this did not happen, trying running MapShop from the command line and see what the debug spew says. It may be that a path is wrong to a conversion tool, or a database connection failed.
Styling the Vector Data
- Open the file ~/Dump/MapShopSession.sqlite in Base (or your SQLite editor)
- In the Features table, add a row and add this data for the feature:
Query: select geom from ne_110m_coastline
- Make sure you save the new row of data.
- In MapShop, click on the Vector Map 1 button, this opens the session database you just edited. If all goes well, you’ll see the same data you saw earlier (without the grid) Note, there is still no style applied, let’s define a style.
- Back in Base, open the Styles table, there is already a row of data there. Change the FeatureName to Coastlines and make sure you save the row of data (Pressing Enter is not good enough, switch to another table or click below the row...)
- Back in MapShop, click View / Refresh (or Command+R) to see the style applied. The coastlines should now be purple (ff00ffff):
- Try playing around with the styles for this feature a bit. Here we’ve changed Width, OutineColor, and OutlineWidth:
You could also try adding some more shape layers as a different layer and styling them differently. The workflow is thus:
- Get your vector data into the PostGIS METool database
- Define a row in the Features table of the session database that has a query that can access your data for that feature. The feature row determines the zoom level the data should appear (MinScale, MaxScale) and it’s ZOrder (Order) for drawing in the mapping engine.
- Define a row in the Styles table of the session database that has all the styling information for the data. NOTE: Some style attributes only apply to polygonal data (i.e. Texture).
- Save the Features / Style entries
- Go back to MapShop and Refresh your map view and pan and zoom around to see if you like the appearance.
Another exciting thing to try is viewing/styling OpenStreetMap data. See this post for instructions on getting started with OSM data on MapShop.
- If you use Base, make sure that you have actually saved a row in the Features / Styles tables by clicking away to another row, or to another table.
- If you create a query that is not valid, it is possible to either get PostGIS or MapShop into an unrecoverable state. You may have to close down MapShop and restart it. In some cases we’ve had to shutdown and restart PostgreSQL because of a run-away query.
- The performance of MapShop is not an indicator of performance on an iOS or Android device. Even though it is the same rendering engine, MapShop can render more data since it has access to a better GPU and more memory on your workstation. So its sheer rendering performance is faster. However, if you are iterating over a session-based vector map design, MapShop is having to issue hundreds (maybe thousands, maybe millions...) of queries to PostGIS, so in that case it can be slower, depending entirely on the complexity of your vector map queries, layers, etc. Therefore, always export your maps and try them on a device to see what performance will be like for end-users.
Exporting a Vector Map using METool
If you have been following along up this point you you have:
- Installed MapShop
- Created a local PostGIS database on your workstation
- Imported vector data into the database
- Created and styled features for your vector map in PostGIS using MapShop
You are now ready to create an optimized rendition of your vector map for use on via the Altus Mapping Engine SDK on iOS or Android.
To do this you will use METool, our command line program for map processing. METool is sort-of like a swiss-army knife creating maps for the Altus Mapping Engine. It is capable of creating terrain, raster, vector, and marker layers, and it is designed to be used in shell scripts on Linux and OSX as part of map data processing pipelines.
Install METool-required Libraries
Before running METool, you’ll need to download and install these packages from our development server:
These packages are created using Mac Ports and are the version of the libraries that METool has been tested against internally. You, of course, are free to use different versions of these libraries; your mileage may vary.
For now, we’ll use the OSX version to create this simple vector map.
- Download and extract http://dev1.ba3.us/metool.zip
- From a command prompt run ./metool and look at the available options.
- You should see something like this as output:
metool - BA3 Mapping Engine Tool Note:This program requires ImageMagick USAGE: metool -OPTION_1 argument -OPTION_n argument OPTIONS: -clusterdistance ARG When creating a marker map, specifies the distance in pixels as measured from the center of markers where marker weight is used to decide which to display. -complete Make a map the encompasses all source data. -converttopvr Converts an existing map generated from a geotif to one containing PVR-only textures. When used, you must specify -mappath. PVRTexTool must exist on your path. -donothing Check the command line, but don't generate the map. Useful for debugging command line options. -extracttiles Extracts all the tiles from the .map file. Useful for debugging. -generatewatertiles Creates SRTM3 water sample tiles in the same folder as your SRTM3 data. -geotif Process geo-tif file. For example FAA sectional, TAC. -gtopopath ARG Location of NASA GTOPO data. -h Show usage. -injectmarkers Inject marker data into an existing Sqlite database. -inputfile ARG Name of input file. Required for if -geotif or -markers option is specified. -inputpath ARG Input tile path where files begin with coordinates (i.e. W060.75xN47.25) -inputsqlitefile ARG Database into which marker tables should be added. Use with -injectmarkers -level ARG Output detail level. Higher means more detail. Level 11 works well for sectionals. -mapname ARG Filename for the .map and .sqlite files for the generated map. (i.e. Midwest) -mappath ARG Path (minus the .map extension) to an existing map file. Used for post-processing functions like -converttopvr -markers Create a marker map. Input file will be a tab delimited text file with these columns: 1) Marker ID (string), 2) Latitude (double), 3) Longitude (double), 4) Weight (double) 5) Minimum level. Columns 4 and 5 are optional and will default to 0. If you set this option you should also set the -clusterdistance option to a positive integer value. -markerselectsql ARG SQL statement that returns markerid, . Use with -injectmarkers -max_lat ARG North-most latitude (60.5). Ignored if -complete is set. -max_lon ARG East-most longitude (-30.5). Ignored if -complete is set. -maxlevel ARG When creating a marker map, specifies the maximum level of the marker tree. -min_lat ARG South-most latitude (59.5). Ignored if -complete is set. -min_lon ARG West-most longitude (-78.5). Ignored if -complete is set. -nocollar Applies if -usecutline is set. Prevents generation of a collar. -outputpath ARG Output path (i.e. /Users/JohnDoe/mapdata) -skipgeneration Do everything except generate tiles and map files. This is useful for seeing stats of the input data and the number of tiles that would have been generated. -srtm3path ARG Location of SRTM3-style height data. -swbdpath ARG Location of NASA SWBD data. -tablenameprefix ARG Prefix all generated tables with this prefix. Use with -injectmarkers -terrain Create a terrain map. -tilepostprocesscommand ARG Name of program or shell script to run to process a tile before packing .map file. The program will be called with a single argument, the name of the tile. When finished the same file must exist (.jpg or .png) but the internals of the file may be adjusted. -usecutline If set, looks for input.tif-cutline/cutline.shp file, then creates two output files, name.map and name_collr.map. Use -nocollar to disable collar generation. -zorder ARG Set z-order of generated map. If not specified, defaults to 2. metool 0.24.0 Copyright (C) 2013 BA3 LLC. ALL RIGHTS RESERVED You need to tell metool what to do: -geotif, -markers, -terrain, or -vector
If you don’t see that, you’ll see some form of error, likely because a required library is missing. All required libraries for METool should be available here for OSX: http://dev1.ba3.us/public_sdk/3rdParty/
Have METool Generate Vector Map
In a terminal issue this command to METool:
./metool -vector -complete -mapname Coastlines -postgisdatabasename metool
-outputpath "/Users/your_user_name/Dump" -level 5
(Replace your_user_name with your user name)
An explanation of the command line parameters:
- -vector Generate a vector map. Other map types are -marker, -terrain, -geotif
- -complete The map covers the entire planet. If you want a smaller map, you would add -min_lon, -min_lat, -max_lon, and -max_lat command line parameters and set them using degrees.
- -mapname The name of the map. The generates files will be use this name as their root.
- -postgisdatabasename The name of your local PostGIS database which contains your vector data, as well as a user named metool with password named metool.
- -sessionfilename The name of the MapShop session that has the feature queries and styling information that you’ve been working with in MapShop
- -outputpath Where metool will store intermediate files and where map files will be saved to.
- -level The ‘depth’ of the map.
If everything is configured properly, METool will generate a world-wide vector map consisting of 660 vector tiles from your source coastline data. In addition to the intermediate tile files which are placed in ~/Dump/Coastlines/, two other files will be produced:
These two files are what is needed by the Altus Mapping Engine, the intermediate files may be deleted.
METool will also embed the Styles and Features tables from the session database you used into the newly generated map (except for the queries to your PostGIS database, those are no longer needed).
You can open this newly generated map in MapShop. Just click File/Open... and navigate to Coastlines.map.
Understanding METool and Altus Mapping Engine Layers
This section is intended for those who want to take a deeper dive into METool.
METool generates map layers for the Altus Mapping Engine in several different formats. Map layers consists of two files, a .map file and a .sqlite file. The sqlite file serves as index and meta data for the .map file.
The .map file is a blob of concatenated map data. The data type varies depending on the map type. It could be raster data (.png, .jpg. .pvr), height data, or vector data. For raster and terrain data we internally store data in 256x256 sample chunks. Markers don’t used a tile storage technique. Vector maps are tile along geographic boundaries, but there is no fixed pixel size to what is contained in a vector tile.
At run-time, the mapping engine locates, loads, and caches necessary subsets of data data from the .map file, generally from many simultaneous read threads.
The meta data for the map is located in the mapinfo table in the .sqlite file. This includes data type, original name, and map z-order. It is important to note that for pre-generated maps, z-order is set ahead of time in the .sqlite file. For virtual layers, those generated at run-time, z-order is controlled by the virtual layer API of the engine.
Because our mapping engine renders a true 3D representation of Earth it is very different than flat models with regards to how map data is generated and stored. In 2D systems where maps are projected on a plane, a simple “power of 2” scheme can be used to create tiles at different zoom levels. This is how spherical mercator (google, OSM, etc.) tiles are organized. But, these tiles are meant to be rendered on a plane.
Because our approach is 3D and based on a sphere, the math is more complex than “power of 2” and strives for uniform spherical parameterization. Therefore, there is no simple scheme that can be used to subdivide projected raster maps for our engine. Therefore, METool, in essence, unprojects projected raster map data and puts pixels into a space that works for projecting textures onto a sphere in our engine. Our internal projection is variable depending where on the sphere the data is to be drawn.
In addition, our data organization is highly optimized for runtime loading and caching. So, to get the absolute best performance from the BA3 Altus Mapping Engine, it is always best create map layers with METool.
METool generates layers to a given ‘level’ which is set by the -level command line option. Level controls the depth of subdivision of the designated geographic area into which data is sampled. This is an important distinction and understanding this will help you set the level setting to the optimal value when generating map layers from raster data.
METool creates a subdivided data structure based on the level (depth) and target geographic area you specify and then samples from the source data (geotif, terrain, or PostGIS database) to fill that structure and generate the resulting map layer.
At a given level, for any point on the planet, our tiles approach a constant resolution in samples per meter. For fast loading and rendering performance we use square tiles which makes this not exact everywhere on the planet. Near the equator (and for most of the continental U.S.) our tiles use an equirectangular (or plate carrèe) projection. Therefore, to pick an effective level in the US you can go through a few steps:
- Find maximum resolution in arc-seconds per pixel for the source data
- Use the chart below (for 256x256 tiles, our current default) to find what level that falls between
- For pixel-perfect tiles:
- Pick the higher of the two levels
- Note: you will be oversampling your data unless it lies exactly on a level
- Otherwise, experiment with lower resolution levels to find an acceptable resolution
- Note: look in the source data for the minimum size feature that needs to be visible
|Level||x||y||Seconds per Pixel||Tiles||Tiles Accumulated||Meters per pixel (at equator)|
For example, if 1 pixel of your raster source data represents 0.01 x 0.01 arc hours of geographic data, then the resolution in arc-seconds per pixel is 36. This lies between level 13 and 14.
You will notice that there is both a geometric increase in the number of children AND a corresponding geometric decrease in the arc-hour area of a given node as level goes up (or down depending on how you look at it). Also note that the best maximum level is dependent on both the source data resolution in terms of pixels per arc hour area AND the geographic size of the map and what you are trying to achieve.
Experimentation for acceptable levels can be done in a small geographic region (min and max geographic coordinate) using the command line options -min_lon, -min_lat, -max_lon, -max_lat to specify a bounding box for the target map layer.
In practice, we’ve found level 14 works out well for FAA Terminal Area Charts, but level 13 works out well for FAA sectionals.
With regards to run-time functionality, there is no requirement that you generate the optimum (or higher than optimum) level. The mapping engine will handle any combination of detail level and geographic area. If you generate a lower level than the resolution of your source data, the map will simply appear to have lower detail when zoomed in (but will have a size advantage). If you generate a detail level that is too high, you will see every single pixel of the source data in our engine (and then some because it will have been over-sampled), but you will have generated a map that wastes a lot of space for no reason.
Mathematically speaking, the optimum child sample size will almost always fall between levels. In that case you need to go to the higher level if you want optimum map detail. If you want the best looking maps that waste no space in terms of un-needed leaf nodes, try to match the resolution of your source data to the arc hour size of the generated map.
Similarly, with vector data, you can determine the resolution of your source data in samples per meter to determine the optimum level for generating a vector map.
And finally, not matter what level you choose, make sure you actually look at the data on device. Sometimes it is easier to just create maps of a small geographic area and experiment until you arrive at the right trade-off between data size and map appearance.
If you have any questions about Altus MapShop, any feature requests or suggestions for improving Altus MapShop, please send them to [email protected].