Flight Contest maps server

Introduction

Setting up a Flight Contest maps server.

Descriptions and tools based on following projects

Printmaps (http://printmaps-osm.de, https://github.com/printmaps/printmaps)
OpenTopoMap (https://github.com/der-stefan/OpenTopoMap)

Virtual server requirements

RAM: 60GB (not dynamic)
SSD: 2TB
CPU: 16
OS: Ubuntu 18.04 LTS 64bit

such as: Strato V-Server Linux V80-60

Software installation

SSH login with user root

ssh -l root <server-dns-name>

If first login, change password with passwd

Install Ubuntu updates

apt update
apt upgrade

If something has been installed run

sync;sync;shutdown -r now

Install tools, libraries and fonts

apt-get install nano git-core cmake
apt-get install zip unzip tar wget bzip2 mc
apt-get install libboost-all-dev build-essential autoconf libtool
apt-get install libosmium2-dev libprotozero-dev rapidjson-dev libboost-program-options-dev
apt-get install zlib1g-dev libexpat1-dev pandoc
apt-get install libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev
apt-get install munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev
apt-get install libtiff5-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev libagg-dev
apt-get install devscripts libjson-perl libipc-sharelite-perl libgd-perl debhelper
apt-get install liblua5.2-dev lua5.1 liblua5.1-dev lua5.3 liblua5.3-dev
apt-get install ttf-unifont fonts-noto

Install Postgres database server

apt-get install postgresql postgresql-contrib postgis

nano /etc/postgresql/10/main/postgresql.conf

In nano you can search with Ctrl-w.
Check or edit the following values

shared_buffers = 128MB
work_mem = 256MB
maintenance_work_mem = 256MB
autovacuum = off

Save with Ctrl-x then y then Enter

nano /etc/sysctl.conf

At the top, after the other kernel-definitions, insert the following and save it

kernel.shmmax=268435456

Restart server

sync;sync;shutdown -r now

Check successfully restart

who -b

Install Mapnik renderer

apt-get install libmapnik3.0 libmapnik-dev mapnik-utils python-mapnik unifont

Compile and install osm2pgsql

mkdir ~/src
cd ~/src
git clone git://github.com/openstreetmap/osm2pgsql.git

cd osm2pgsql
mkdir build && cd build
cmake ..
make
make install

Install Osmium

apt-get install osmium-tool

Configure Python 3 as default

nano ~/.bashrc

Insert the following at the bottom and save it

alias python=python3

Then

source ~/.bashrc

Check Python version with

python --version

Install phyghtmap

apt-get install python3-setuptools python3-matplotlib python3-bs4 python3-numpy python3-gdal

cd ~/src
wget http://katze.tfiu.de/projects/phyghtmap/phyghtmap_2.10.orig.tar.gz
tar -xvzf phyghtmap_2.10.orig.tar.gz
cd phyghtmap-2.10
python3 setup.py install

Install gdal

apt-get install gdal-bin python-gdal

Configure users and roles

Create user gis with group gis and directory /home/gis.

adduser gis

Add role gis to Postgres database server

sudo -u postgres -i
createuser --createdb gis -s
exit

Create data

SSH login with user gis

ssh -l gis <server-dns-name>

Start screen

screen

You may close SSH console while command is running. After new SSH login continue with

screen -r

Close screen with

exit

Create data folder

cd ~
mkdir data

Download and install Flight Contest maps server

cd ~/data
wget http://download.flightcontest.de/fcmaps.zip
cd ~
unzip ~/data/fcmaps.zip

Download and install water polygons

cd ~/data
wget https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip
wget http://download.flightcontest.de/water-polygons-generalized-3857.zip

cd ~/fcmaps/otm/mapnik
mkdir data
cd ~/fcmaps/otm/mapnik/data
unzip ~/data/water-polygons-split-3857.zip
unzip ~/data/water-polygons-generalized-3857.zip

cd ~/fcmaps/otm-dev/mapnik
ln -s ~/fcmaps/otm/mapnik/data .

OSM data

Download data (Europe)

cd ~/data
mkdir osm && cd osm
wget http://download.geofabrik.de/europe-latest.osm.pbf

Create database gis2

dropdb gis2
createdb gis2
psql -d gis2 -c 'CREATE EXTENSION postgis;'

Load OSM data into OSM database (~/data/osm/*.pbf → gis2)
(Germany, Austria, Switzerland, Poland, Czech Republic, Slovakia, Hungary, parts of France, parts of Denmark)

osm2pgsql -d gis2 --slim --drop --flat-nodes=FILENAME --bbox 1.0,42.0,24.0,55.0 --style ~/fcmaps/osm2pgsql/fcmaps.style ~/data/osm/europe-latest.osm.pbf

Wait… (37h)

Preprocessing

cd ~/fcmaps/otm-dev/mapnik/tools/
psql gis2 < arealabel.sql
psql gis2 < stationdirection.sql

Wait… (10min)

psql gis2 < viewpointdirection.sql
psql gis2 < pitchicon.sql
psql -d gis2 -c 'ALTER TABLE planet_osm_point ADD COLUMN otm_isolation text;'
bash update_lowzoom.sh

Wait… (20h)

Activate new databases gis and lowzoom

dropdb gisold
dropdb lowzoomold
psql -d postgres -c 'ALTER DATABASE gis RENAME TO gisold;'
psql -d postgres -c 'ALTER DATABASE gis2 RENAME TO gis;'
psql -d postgres -c 'ALTER DATABASE lowzoom RENAME TO lowzoomold;'
psql -d postgres -c 'ALTER DATABASE lowzoom2 RENAME TO lowzoom;'

Change date

cd ~/fcmaps
nano printmaps_webservice_capabilities.json

Restart server

Check database size

psql -d postgres -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;"

Contour data

Download all necessary SRTM tiles, see http://www.viewfinderpanoramas.org/Coverage%20map%20viewfinderpanoramas_org3.htm

cd ~/data
mkdir srtm
cd ~/data/srtm
nano list.txt

Insert the following (for Germany)

http://viewfinderpanoramas.org/dem3/N32.zip
http://viewfinderpanoramas.org/dem3/N33.zip
http://viewfinderpanoramas.org/dem3/M32.zip
http://viewfinderpanoramas.org/dem3/M33.zip
http://viewfinderpanoramas.org/dem3/L32.zip
http://viewfinderpanoramas.org/dem3/L33.zip

Save it and continue with

wget -i list.txt

Move zip files to folder germany and unpack all

mkdir germany && mv N*.zip germany && mv M*.zip germany && mv L*.zip germany
for zipfile in germany/*.zip; do unzip -j -o "$zipfile" -d germany/unpacked; done

Fill all voids (*.hgt → *.hgt.tif)

for hgtfile in germany/unpacked/*.hgt; do gdal_fillnodata.py $hgtfile $hgtfile.tif; done

Merge into one huge tif (*.hgt.tif → raw.tif)

gdal_merge.py -n 32767 -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -o germany/unpacked/raw.tif germany/unpacked/*.hgt.tif

Convert into Mercator projection, interpolate and shrink (raw.tif → 90.tif)

gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 90 90 germany/unpacked/raw.tif germany/unpacked/90.tif

Create 10m contour data (90.tif → lon*.pbf) (high memory consumption)

phyghtmap --step=10 --start-node-id=1000000000 --start-way-id=1000000000 --max-nodes-per-tile=0 --no-zero-contour --pbf germany/unpacked/90.tif

tif: 14844 x 23256 points, Lon 6...18°, Lat 44...56°

Wait… (4h)

Do these steps for other countries (with different folders and ids). Because of memory consumption of phyghtmap only max. 6 zip files.

France (node-id=3000000000)

http://viewfinderpanoramas.org/dem3/M30.zip
http://viewfinderpanoramas.org/dem3/M31.zip
http://viewfinderpanoramas.org/dem3/L30.zip
http://viewfinderpanoramas.org/dem3/L31.zip
http://viewfinderpanoramas.org/dem3/K30.zip
http://viewfinderpanoramas.org/dem3/K31.zip

tif: 14844 x 21493 points, Lon -6...6°, Lat 40...52°

South Africa (node-id=2000000000)

http://viewfinderpanoramas.org/dem3/SG33.zip
http://viewfinderpanoramas.org/dem3/SG34.zip
http://viewfinderpanoramas.org/dem3/SG35.zip
http://viewfinderpanoramas.org/dem3/SG36.zip
http://viewfinderpanoramas.org/dem3/SH33.zip
http://viewfinderpanoramas.org/dem3/SH34.zip
http://viewfinderpanoramas.org/dem3/SH35.zip
http://viewfinderpanoramas.org/dem3/SH36.zip
http://viewfinderpanoramas.org/dem3/SI33.zip
http://viewfinderpanoramas.org/dem3/SI34.zip
http://viewfinderpanoramas.org/dem3/SI35.zip

Lon 14...36°, Lat -35...-24°

Old

http://viewfinderpanoramas.org/dem3/N31.zip
http://viewfinderpanoramas.org/dem3/N34.zip
http://viewfinderpanoramas.org/dem3/M34.zip
http://viewfinderpanoramas.org/dem3/L34.zip

Create database contours2

createdb contours2
psql -d contours2 -c 'CREATE EXTENSION postgis;'

Load contour data into contour database (all generated pbf files must be loaded together in one step)

osm2pgsql -d contours2 --slim --cache 12000 --number-processes 10 --style ~/fcmaps/osm2pgsql/contours.style ~/data/srtm/lon*.pbf

Wait… (4h)

Activate new database contours

dropdb contoursold
psql -d postgres -c 'ALTER DATABASE contours RENAME TO contoursold;'
psql -d postgres -c 'ALTER DATABASE contours2 RENAME TO contours;'

Check database size

psql -d postgres -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;"

Configure server

Check server ressources

Find out database size

psql -d postgres -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;"

gis: 107GB, contours: 56GB, lowzoom: 152MB

Display free disk space

df

Display free ram

free -m

SSH login with user root

ssh -l root <server-dns-name>

Restart server

sync;sync;shutdown -r now

Check successfully restart

who -b

Run services

SSH login with user gis

Configure services as executable

cd ~/fcmaps
chmod +x printmaps_buildservice
chmod +x printmaps_webservice

Start services manually

nohup ./printmaps_buildservice 1>./logs/printmaps_buildservice.out 2>&1 &
nohup ./printmaps_webservice 1>./logs/printmaps_webservice.out 2>&1 &

Test web service

http://<server-dns-name>:8181/api/beta2/maps/capabilities/service

Search running service processes

ps -Af | grep "printmaps_"

Terminate running service processes

kill <pid>

Show log files

cat ./logs/printmaps_buildservice.log
cat ./logs/printmaps_webservice.log

Install services autostart

SSH login with user root

cd /etc/systemd/system

Create service description of build service

nano fcmaps_build.service

[Unit]
Description=Flight Contest maps build service
Requires=network.target
After=network.target

[Service]
User=gis
WorkingDirectory=/home/gis/fcmaps
ExecStart=/home/gis/fcmaps/printmaps_buildservice ./printmaps_buildservice.yaml
Restart=always

[Install]
WantedBy=multi-user.target

Create service description of web service

nano fcmaps_web.service

[Unit]
Description=Flight Contest maps web service
Requires=network.target
After=network.target

[Service]
User=gis
WorkingDirectory=/home/gis/fcmaps
ExecStart=/home/gis/fcmaps/printmaps_webservice ./printmaps_webservice.yaml
Restart=always

[Install]
WantedBy=multi-user.target

Enable autostart

systemctl enable fcmaps_build.service
systemctl enable fcmaps_web.service

Restart server

sync;sync;shutdown -r now

Check successfully restart

who -b

Configure Flight Contest

Add server in → Extras → Settings → Config

flightcontest {
  maps {
    fcmap {
       server = "http://<server-dns-name>:8181/api/beta2/maps"
    }
  }
}

and restart Flight Contest.

Test map generation with → Routes → <Route> → OSM Contest Map.