Introduction
Setting up a Flight Contest maps server.
Descriptions and tools based on following projects
-
Printmaps (http://printmaps-osm.de, https://github.com/printmaps/printmaps)
-
OpenTopoMap (https://github.com/der-stefan/OpenTopoMap)
Virtual server requirements
-
RAM: 60GB (not dynamic)
-
SSD: 2TB
-
CPU: 16
-
OS: Ubuntu 18.04 LTS 64bit
such as: Strato V-Server Linux V80-60
Software installation
SSH login with user root
ssh -l root <server-dns-name>
If first login, change password with passwd
Install Ubuntu updates
apt update apt upgrade
If something has been installed run
sync;sync;shutdown -r now
Install tools, libraries and fonts
apt-get install nano git-core cmake apt-get install zip unzip tar wget bzip2 mc apt-get install libboost-all-dev build-essential autoconf libtool apt-get install libosmium2-dev libprotozero-dev rapidjson-dev libboost-program-options-dev apt-get install zlib1g-dev libexpat1-dev pandoc apt-get install libxml2-dev libgeos-dev libgeos++-dev libpq-dev libbz2-dev libproj-dev apt-get install munin-node munin libprotobuf-c0-dev protobuf-c-compiler libfreetype6-dev apt-get install libtiff5-dev libicu-dev libgdal-dev libcairo-dev libcairomm-1.0-dev libagg-dev apt-get install devscripts libjson-perl libipc-sharelite-perl libgd-perl debhelper apt-get install liblua5.2-dev lua5.1 liblua5.1-dev lua5.3 liblua5.3-dev apt-get install ttf-unifont fonts-noto
Install Postgres database server
apt-get install postgresql postgresql-contrib postgis
nano /etc/postgresql/10/main/postgresql.conf
In nano you can search with
.Ctrl-w
Check or edit the following values
shared_buffers = 128MB work_mem = 256MB maintenance_work_mem = 256MB autovacuum = off
Save with
then Ctrl-x
then y
Enter
nano /etc/sysctl.conf
At the top, after the other kernel-definitions, insert the following and save it
kernel.shmmax=268435456
Restart server
sync;sync;shutdown -r now
Check successfully restart
who -b
Install Mapnik renderer
apt-get install libmapnik3.0 libmapnik-dev mapnik-utils python-mapnik unifont
Compile and install osm2pgsql
mkdir ~/src cd ~/src git clone git://github.com/openstreetmap/osm2pgsql.git
cd osm2pgsql mkdir build && cd build cmake .. make make install
Install Osmium
apt-get install osmium-tool
Configure Python 3 as default
nano ~/.bashrc
Insert the following at the bottom and save it
alias python=python3
Then
source ~/.bashrc
Check Python version with
python --version
Install phyghtmap
apt-get install python3-setuptools python3-matplotlib python3-bs4 python3-numpy python3-gdal
cd ~/src wget http://katze.tfiu.de/projects/phyghtmap/phyghtmap_2.10.orig.tar.gz tar -xvzf phyghtmap_2.10.orig.tar.gz cd phyghtmap-2.10 python3 setup.py install
Install gdal
apt-get install gdal-bin python-gdal
Configure users and roles
Create user gis with group gis and directory /home/gis.
adduser gis
Add role gis to Postgres database server
sudo -u postgres -i createuser --createdb gis -s exit
Create data
SSH login with user gis
ssh -l gis <server-dns-name>
Start screen
screen
You may close SSH console while command is running. After new SSH login continue with
screen -r
Close
withscreen
exit
Create data folder
cd ~ mkdir data
Download and install Flight Contest maps server
cd ~/data wget http://download.flightcontest.de/fcmaps.zip cd ~ unzip ~/data/fcmaps.zip
Download and install water polygons
cd ~/data wget https://osmdata.openstreetmap.de/download/water-polygons-split-3857.zip wget http://download.flightcontest.de/water-polygons-generalized-3857.zip
cd ~/fcmaps/otm/mapnik mkdir data cd ~/fcmaps/otm/mapnik/data unzip ~/data/water-polygons-split-3857.zip unzip ~/data/water-polygons-generalized-3857.zip
cd ~/fcmaps/otm-dev/mapnik ln -s ~/fcmaps/otm/mapnik/data .
OSM data
Download data (Europe)
cd ~/data mkdir osm && cd osm wget http://download.geofabrik.de/europe-latest.osm.pbf
Create database gis2
dropdb gis2 createdb gis2 psql -d gis2 -c 'CREATE EXTENSION postgis;'
Load OSM data into OSM database (~/data/osm/*.pbf → gis2)
(Germany, Austria, Switzerland, Poland, Czech Republic, Slovakia, Hungary, parts of France, parts of Denmark)
osm2pgsql -d gis2 --slim --drop --flat-nodes=FILENAME --bbox 1.0,42.0,24.0,55.0 --style ~/fcmaps/osm2pgsql/fcmaps.style ~/data/osm/europe-latest.osm.pbf
Wait… (37h)
Preprocessing
cd ~/fcmaps/otm-dev/mapnik/tools/ psql gis2 < arealabel.sql psql gis2 < stationdirection.sql
Wait… (10min)
psql gis2 < viewpointdirection.sql psql gis2 < pitchicon.sql psql -d gis2 -c 'ALTER TABLE planet_osm_point ADD COLUMN otm_isolation text;' bash update_lowzoom.sh
Wait… (20h)
Activate new databases gis and lowzoom
dropdb gisold dropdb lowzoomold psql -d postgres -c 'ALTER DATABASE gis RENAME TO gisold;' psql -d postgres -c 'ALTER DATABASE gis2 RENAME TO gis;' psql -d postgres -c 'ALTER DATABASE lowzoom RENAME TO lowzoomold;' psql -d postgres -c 'ALTER DATABASE lowzoom2 RENAME TO lowzoom;'
Change date
cd ~/fcmaps nano printmaps_webservice_capabilities.json
Restart server
Check database size
psql -d postgres -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;"
Contour data
Download all necessary SRTM tiles, see http://www.viewfinderpanoramas.org/Coverage%20map%20viewfinderpanoramas_org3.htm
cd ~/data mkdir srtm cd ~/data/srtm nano list.txt
Insert the following (for Germany)
http://viewfinderpanoramas.org/dem3/N32.zip http://viewfinderpanoramas.org/dem3/N33.zip http://viewfinderpanoramas.org/dem3/M32.zip http://viewfinderpanoramas.org/dem3/M33.zip http://viewfinderpanoramas.org/dem3/L32.zip http://viewfinderpanoramas.org/dem3/L33.zip
Save it and continue with
wget -i list.txt
Move zip files to folder germany and unpack all
mkdir germany && mv N*.zip germany && mv M*.zip germany && mv L*.zip germany for zipfile in germany/*.zip; do unzip -j -o "$zipfile" -d germany/unpacked; done
Fill all voids (*.hgt → *.hgt.tif)
for hgtfile in germany/unpacked/*.hgt; do gdal_fillnodata.py $hgtfile $hgtfile.tif; done
Merge into one huge tif (*.hgt.tif → raw.tif)
gdal_merge.py -n 32767 -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -o germany/unpacked/raw.tif germany/unpacked/*.hgt.tif
Convert into Mercator projection, interpolate and shrink (raw.tif → 90.tif)
gdalwarp -co BIGTIFF=YES -co TILED=YES -co COMPRESS=LZW -co PREDICTOR=2 -t_srs "+proj=merc +ellps=sphere +R=6378137 +a=6378137 +units=m" -r bilinear -tr 90 90 germany/unpacked/raw.tif germany/unpacked/90.tif
Create 10m contour data (90.tif → lon*.pbf) (high memory consumption)
phyghtmap --step=10 --start-node-id=1000000000 --start-way-id=1000000000 --max-nodes-per-tile=0 --no-zero-contour --pbf germany/unpacked/90.tif
tif: 14844 x 23256 points, Lon 6...18°, Lat 44...56°
Wait… (4h)
Do these steps for other countries (with different folders and ids). Because of memory consumption of phyghtmap only max. 6 zip files.
France (node-id=3000000000)
http://viewfinderpanoramas.org/dem3/M30.zip http://viewfinderpanoramas.org/dem3/M31.zip http://viewfinderpanoramas.org/dem3/L30.zip http://viewfinderpanoramas.org/dem3/L31.zip http://viewfinderpanoramas.org/dem3/K30.zip http://viewfinderpanoramas.org/dem3/K31.zip
tif: 14844 x 21493 points, Lon -6...6°, Lat 40...52°
South Africa (node-id=2000000000)
http://viewfinderpanoramas.org/dem3/SG33.zip http://viewfinderpanoramas.org/dem3/SG34.zip http://viewfinderpanoramas.org/dem3/SG35.zip http://viewfinderpanoramas.org/dem3/SG36.zip http://viewfinderpanoramas.org/dem3/SH33.zip http://viewfinderpanoramas.org/dem3/SH34.zip http://viewfinderpanoramas.org/dem3/SH35.zip http://viewfinderpanoramas.org/dem3/SH36.zip http://viewfinderpanoramas.org/dem3/SI33.zip http://viewfinderpanoramas.org/dem3/SI34.zip http://viewfinderpanoramas.org/dem3/SI35.zip
Lon 14...36°, Lat -35...-24°
Old
http://viewfinderpanoramas.org/dem3/N31.zip http://viewfinderpanoramas.org/dem3/N34.zip http://viewfinderpanoramas.org/dem3/M34.zip http://viewfinderpanoramas.org/dem3/L34.zip
Create database contours2
createdb contours2 psql -d contours2 -c 'CREATE EXTENSION postgis;'
Load contour data into contour database (all generated pbf files must be loaded together in one step)
osm2pgsql -d contours2 --slim --cache 12000 --number-processes 10 --style ~/fcmaps/osm2pgsql/contours.style ~/data/srtm/lon*.pbf
Wait… (4h)
Activate new database contours
dropdb contoursold psql -d postgres -c 'ALTER DATABASE contours RENAME TO contoursold;' psql -d postgres -c 'ALTER DATABASE contours2 RENAME TO contours;'
Check database size
psql -d postgres -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;"
Configure server
Check server ressources
Find out database size
psql -d postgres -c "SELECT pg_database.datname, pg_size_pretty(pg_database_size(pg_database.datname)) AS size FROM pg_database;"
gis: 107GB, contours: 56GB, lowzoom: 152MB
Display free disk space
df
Display free ram
free -m
SSH login with user root
ssh -l root <server-dns-name>
Restart server
sync;sync;shutdown -r now
Check successfully restart
who -b
Run services
SSH login with user gis
Configure services as executable
cd ~/fcmaps chmod +x printmaps_buildservice chmod +x printmaps_webservice
Start services manually
nohup ./printmaps_buildservice 1>./logs/printmaps_buildservice.out 2>&1 & nohup ./printmaps_webservice 1>./logs/printmaps_webservice.out 2>&1 &
Test web service
http://<server-dns-name>:8181/api/beta2/maps/capabilities/service
Search running service processes
ps -Af | grep "printmaps_"
Terminate running service processes
kill <pid>
Show log files
cat ./logs/printmaps_buildservice.log cat ./logs/printmaps_webservice.log
Install services autostart
SSH login with user root
cd /etc/systemd/system
Create service description of build service
nano fcmaps_build.service
[Unit] Description=Flight Contest maps build service Requires=network.target After=network.target [Service] User=gis WorkingDirectory=/home/gis/fcmaps ExecStart=/home/gis/fcmaps/printmaps_buildservice ./printmaps_buildservice.yaml Restart=always [Install] WantedBy=multi-user.target
Create service description of web service
nano fcmaps_web.service
[Unit] Description=Flight Contest maps web service Requires=network.target After=network.target [Service] User=gis WorkingDirectory=/home/gis/fcmaps ExecStart=/home/gis/fcmaps/printmaps_webservice ./printmaps_webservice.yaml Restart=always [Install] WantedBy=multi-user.target
Enable autostart
systemctl enable fcmaps_build.service systemctl enable fcmaps_web.service
Restart server
sync;sync;shutdown -r now
Check successfully restart
who -b
Configure Flight Contest
Add
in printserverapi
→ Extras → Settings → Config
flightcontest { contestmap { printserverapi = "http://<server-dns-name>:8181/api/beta2/maps" } }
and restart Flight Contest.
Test map generation with
.→ Routes → <Route> → OSM Contest Map