Flatmap resources

Mapmaker

Mapmaker is a Python application that generates Mapbox compatible tilesets from a range of sources (currently including SVG drawings, MBF Biosciences segmented images, and PowerPoint slides). In conjunction with SCKAN, Mapmaker is used to render the flatmaps portraying the anatomy and nerve knowledge of species, as seen on the SPARC Portal Maps page.
Flatmaps can be generated and viewed by using the flatmap maker (mapmaker), server, and viewer. For more information on these flatmap resources see the SPARC Map Reuse documentation.

SPARC Connectivity Knowledge Base of the Autonomic Nervous System (SCKAN)

SCKAN is a key component of the SPARC Program, providing comprehensive knowledge of the autonomic nervous system (ANS) nerve to end organ connectivity. Derived from SPARC experts, literature and textbooks, SCKAN provides the resulting connectivity seen on the SPARC Portal flatmaps. For more information on SCKAN, refer to the documentation below.

• sparc.science | SCKAN
• SPARC SAWG | SCKAN
• Assessing SCKAN

ApiNATOMY

SCKAN knowledge is based on various ApiNATOMY models set up by SPARC investigators. For more information on ApiNATOMY, visit the SPARC SAWG site. The ApiNATOMY models can be accessed via the github repository.

Simply clone the repository.

$ git clone https://github.com/open-physiology/apinatomy-models

Navigate to the desired model in the models folder. Each model can be found in the docs folder, in the form of .svg and .pdf files.

Map-tools

Map-tools is a jupyter notebook package that provides information on individual neuron connections, according to the knowledge in SCKAN. It can be used to understand each connection and the respective anatomical features involved.

Installation

Map-tools is best run under a python environment and is reliant on Poetry. Map-tools can then be installed by cloning the GitHub repository.

$ git clone https://github.com/AnatomicMaps/map-tools

Having installed Poetry via the official installer instructions, clone and enter the map-tools directory. Install a Poetry environment with

$ poetry install

Usage

Once installed, map-tools can be run by opening Jupyter Notebook. In the map-tools directory, run jupyter notebook via poetry.

$ poetry run jupyter notebook

The command will provide access to the notebook with a URL, which should then be pasted into a browser window.

Once opened, navigate to and open the connectivity_graph.ipynb notebook in the connectivity-graph folder. A valid SCICRUNCH Key must be input in order to draw knowledge from SCKAN. This can be done by inserting the key under a scicrunch_key field as shown in Figure 2. Note that a SciCrunch key can be generated by setting up a SciCrunch account.

Neuron paths from ApiNATOMY Models can now be visualised, as seen in Figure 3. The desired neuron path can be visualised by changing the entity field. Table 1 represents the ilxtr attribute for each model. Note that the ID of each path is dependent on the visualisation of paths in each model.

Table 1: ilxtr attributes of each path in the respective ApiNATOMY models. These are used to visualise said paths in the connectivity_graphy.ipynb file.

Mapmaker Installation

It is recommended to install and run mapmaker under its own Python virtual environment. Mapmaker can be installed as a python application, or a development version can be set up. The Installing mapmaker in a development environment section explores how to set up the development version of mapmaker. Further instructions for each OS and/or environment can be found in the Installation for various environments section.

The mapmaker resources include flatmap-maker, flatmap-server, and flatmap-viewer. Refer to the SPARC Map Reuse documentation for more information. Each repository contains documentation regarding installation.

https://github.com/AnatomicMaps/flatmap-maker
https://github.com/AnatomicMaps/flatmap-server
https://github.com/AnatomicMaps/flatmap-viewer

Installing and using the mapmaker application

Simply install the mapmaker wheel and the flatmap server by following the respective instructions. For more information on how to install Python wheels, click here. Run mapmaker with the appropriate command, and start the flatmap server to view the map.

Refer to the documentation for more information:

• Mapmaker
• Flatmap server

📘

Note:

Be sure to use the latest release of mapmaker when installing the .whl file. Figure 4 highlights where in the flatmap maker repository to find the mapmaker releases.

Installing mapmaker in a development environment

Prerequisites

The following are required in order to successfully install and generate maps. Links to the packages have been provided, which capture installation instructions.

• Mapmaker:
  • Python 3.10
  • Tippecanoe
  • COIN-OR CBC Solver
  • Poetry
  • Miniforge (for ARM based MacOS machines)
• Flatmap viewer
  • Node.js
• Flatmap server
  • Python 3.8 (or higher)
  • Poetry
  • Libgl1-mesa-glx

Installation instructions

📘

Note:

Examples, including specific commands, for Windows and MacOS can be found in the Installation for various environments section.

The first step is to install the prerequisite packages.

1. 1. make
   2. g++
   3. sqlite libsqlite3-dev
   4. libz-dev
   5. coinor-cbc
2. Set up the Tippecanoe package, following the official instructions.

📘

Note:

Several warnings may appear but can be neglected. However, error messages in red may indicate failure.

3. In the root directory, clone the flatmap-maker, -server, and -viewer repositories.

📘

Note:

This will clone the head of the main branch of each repository. It is recommended to use the latest released version of mapmaker. Switch to the latest release while in the flatmap-maker directory (shown in Figure 4).

4. As aforementioned, the flatmap resources are reliant on Poetry. Following the link, install Poetry via the instructions provided.

📘

Note:

The ‘poetry’ command may not be recognised. This can be resolved by adding Poetry to the PATH, as per the official installation instructions.

5. Once this has been done, enter the flatmap-maker directory and set up a Poetry environment. Test the installation by opening a poetry environment.
6. Exit the poetry environment. In the flatmap-server directory, set up another Poetry environment. The mapserver can be started with

$ poetry run mapserver

📘

Note:

A warning may appear relating to an undefined SCRICRUNCH_API_KEY. This can be ignored as the SciCrunch key will be added via mapmaker.

7. Enter the flatmap-viewer directory and install npm. The output of npm install may mention several vulnerabilities, which can be resolved by following the suggested fix.

📘

Note:

As mentioned in the Prerequisites section, node.js must be installed for the flatmap viewer. Click the link provided in the Prerequisites section and follow the instructions.

The flatmap viewer can be started with

$ npm start

📘

Note:

The flatmap viewer may fail to start due to the updated version of Nodejs. This can be resolved by using Node Version Manager to install v12.22.9._

8. The flatmap resources have been installed and are ready for use. Flatmaps will require a source, including a set of JSON files and an SVG file (see Flatmap sources for more information). For an up-to-date version of the rat flatmap (to be used alongside the developer environment of mapmaker), the repository below should be cloned. The set of sources for the flatmaps of other species found on the SPARC Portal can be found within the AnatomicMaps organization.

https://github.com/AnatomicMaps/rat-flatmap

📘

Note:

This link is for the latest flatmap resources not an official release.

Flatmaps should now be ready for generation.

Using mapmaker

Once the flatmap sources have been appropriately created, mapmaker can now generate the flatmap. Mapmaker has various arguments that can be used to generate maps. These arguments, some optional, can allow for different visualizations of the flatmaps, and for debugging purposes.

Mapmaker involves various required and optional arguments (refer to the documentation for more information). Table 2 contains the required arguments to generate a map.

Table 2: The required arguments when running mapmaker. ‘--source’ refers to the path to the directory of the source files’ manifest.json. ‘--output’ should point to the directory to dump the knowledgebase.db file.

1. Enter the flatmap-maker directory and spawn a poetry shell.
2. The flatmaps utilise the SciCrunch knowledgebase, to use this service a valid SciCrunch API Key must be set. A SciCrunch key can be obtained from the SciCrunch website, following these instructions.

        $ export SCICRUNCH_API_KEY=*Enter a valid SciCrunch Key*

3. Run mapmaker using the following command (with the appropriate and any desired arguments).

$ python runmaker.py --source _Enter path to flatmap source_ --output _Enter path to output directory_

Given mapmaker runs with no crashes or issues, the flatmap has now been successfully generated. The flatmap can be viewed by setting up the server and viewer.

4. To set up the flatmap’s server, open a new terminal and navigate to the flatmap-server directory. The server depends on the knowledgebase.db output by mapmaker. Therefore, the directory that has been set up (via the --output argument in mapmaker) must be referenced.

In the flatmap-server directory, point to the knowledgebase.db file created by mapmaker.

$ export FLATMAP_ROOT=$HOME/\_Enter path to directory_  

Start the server.

$ poetry run mapserver

5. With the flatmap server running, the flatmap can be viewed by starting the viewer.

Open another terminal and navigate to the flatmap-viewer directory. The viewer can be started with

$ npm start

The flatmap can now be viewed on the local server output by the flatmap viewer. This is typically localhost:3000.

Congratulations you made a map!

Installation for various environments

Installing developer mapmaker via Ubuntu (WSL)

Mapmaker can be installed on machines operating WindowsOS via the activation of the Windows Subsystem for Linux (WSL). To set up WSL, open a Windows command prompt as administrator and run the following command:

$ wsl --install

Once installed, reboot and install Ubuntu 22.04 via the Microsoft Store. Open Ubuntu 22.04 and enter a desired username and password (Note that the password will be needed upon sudo installations). Upon reboot, an Ubuntu terminal may open to complete the installation. Ubuntu 22.04 must be the environment mapmaker is run in, as Python 3.10 is required. The installation should now complete successfully, and a chevron prompt will appear.

Note that the flatmap resources require Python 3.10. The version of python in use can be found via

$ python3 -V

The flatmap resources can now be installed. Below are the commands and supplemental information for the respective instructions found in the Installing mapmaker resources section.

📘

Note:

Ubuntu may need to be updated via

terminal

$ sudo apt-get update && sudo apt-get upgrade

$ sudo apt install make
$ sudo apt install g++
$ sudo apt install sqlite libsqlite3-dev
$ sudo apt install libz-dev
$ sudo apt install coinor-cbc

$ git clone <https://github.com/mapbox/tippecanoe.git>  
$ cd tippecanoe  
$ make -j  
$ sudo make install

📘

Note:

The repositories must be cloned in the root directory.

$ git clone https://github.com/AnatomicMaps/flatmap-maker  
$ git clone https://github.com/AnatomicMaps/flatmap-server  
$ git clone https://github.com/AnatomicMaps/flatmap-viewer

$ curl -sSL https://install.python-poetry.org | python3 -  
$ PATH=$PATH:$HOME/.local/bin

$ cd flatmap-maker  
$ poetry install

$ cd ../flatmap-server  
$ sudo apt-get install libgl1-mesa-glx  
$ poetry install
$ poetry run mapserver

$ cd ../flatmap-viewer  
$ npm install

The output of npm install may mention several vulnerabilities. Following the recommended instructions, these can be resolved via:

$ npm audit fix

The viewer can be started with

$ npm start

📘

Note:

npm start may fail to start the viewer. This may be due to the updated version of Nodejs. This can be resolved by utilizing the Node Version Manager (nvm). Following the GitHub installation instructions, install nvm and run the command to install v12.22.9 of Nodejs. The terminal may need to be restarted.

$ nvm install 12.22.9

This version of node can be run via

$ nvm run node --12.22.9

Using Mapmaker - Windows

$ cd flatmap-maker  
$ poetry shell

$ export SCICRUNCH_API_KEY= *Enter a valid SciCrunch Key*

$ python runmaker.py --source ../flatmap-sources/rat-flatmap/manifest.json --output ../flatmap-knowledgebase --verbose --background-tiles --authoring --debug

In a separate terminal,

$ cd flatmap-server  
$ export FLATMAP_ROOT=$HOME/../flatmap-knowledgebase  
$ poetry run mapserver

In another separate terminal,

$ cd flatmap-viewer  
$ npm start

The generated flatmap can now be viewed on the local host server, on port 3000.

Installing developer mapmaker on MacOS

Following the instructions of the official installer, install and update Homebrew.

% /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)"  
% brew update

📘

Note:

The Homebrew will need to be added to PATH, as recommended by the official installer. This can be done by running the commands provided in the output, as shown in Figure 16.

Install Python3.10, if not already installed.

% brew install [email protected]

% brew install tippecanoe

% curl -sSL <https://install.python-poetry.org> | python3 -

The poetry path needs to be added to the shell configuration. If using bash command:

% echo "export PATH=$PATH:$HOME/.local/bin" >> ~/.bashrc

If using zsh command:

% echo "export PATH=$PATH:$HOME/.local/bin" >> ~/.zshrc

% git clone https://github.com/AnatomicMaps/flatmap-maker
% git clone https://github.com/AnatomicMaps/flatmap-serve
% git clone https://github.com/AnatomicMaps/flatmap-viewer

Switch to the latest release of mapmaker. Note that the command below refers to the latest release in October 2022.

% cd flatmap-maker  
% git checkout v1.5.1

The next step is dependent on the processing chip of the MacOS machine. Note that this then affects how mapmaker is run (See the Using Mapmaker - MacOS section). In the root directory:
For Intel based machines, install coin-or cbc.

% brew tap coin-or-tools/coinor  
% brew install coin-or-tools/coinor/cbc

For ARM based machines, documentation can be found in mapmaker. To install conda, follow the official instructions.

% curl -L -O  "https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh"  
% sh Miniforge3-MacOSX-arm64.sh

Confirm yes when prompted. In a new shell:

% cd flatmap-maker
% conda env create -f conda/M1Mac.yaml

Install Poetry in the flatmap-maker directory.

% poetry install

% cd ../flatmap-server  
% poetry install

The mapserver can be started by running

% poetry run mapserver

And shut down by

% Command + .

% cd ../flatmap-viewer  
% npm install  
% npm audit fix

📘

Note:

npm start may fail to start the viewer. This may be due to the updated version of Nodejs. This can be resolved by utilizing the Node Version Manager (nvm). Following the GitHub installation instructions, install nvm and run the command to install v12.22.9 of Nodejs. The terminal may need to be restarted.

terminal

% nvm install 12.22.9

This version of node can be run via

terminal

% nvm run node --12.22.9

The flatmap viewer can then be started with

% npm start

And shut down with

% Command + .

Using Mapmaker - MacOS

As aforementioned, how mapmaker is run is dependent on the processing chip of the MacOS machine.

Activate the virtual environment.
For intel based machines:

% cd flatmap-maker  
% poetry shell  
% export SCICRUNCH_API_KEY= *Enter a valid SCICRUNCH Key*

For ARM based machines, with Conda:

% cd flatmap-maker
% conda activate mapmaker
% export SCICRUNCH_API_KEY= *Enter a valid SCICRUNCH Key*

Mapmaker can now be run. Note that the command below is specific to this example case. The paths for the source and output directories are dependent on the set up of the flatmap resources.

% python runmaker.py --source ../rat-flatmap/manifest.json --output ../flatmaps --verbose --background-tiles --authoring --debug

In a separate terminal,

% cd flatmap-server  
% export FLATMAP_ROOT=$HOME/flatmaps  
% poetry run mapserver

In another separate terminal,

% cd flatmap-viewer  
% npm start

The generated flatmap can now be viewed on the local host server, on port 3000.

Flatmap sources

The flatmap source files must follow a particular structure in order to successfully generate maps with no issues. A clone repository of the rat flatmap on the SPARC Portal has been set up and can be found here.

SVG File

The SVG file of the flatmap sources represents the base drawing of the flatmap. As can be seen in the rat_flatmap.svg file, each shape has a unique identifier and a closed shape. The nomenclature of the features in the SVG is important as it must be consistent with the JSON files. Below is a table of identifiers for types of features in the SVG.

Table 3: The respective identifiers for features in the SVG file. These IDs must be strictly followed as the JSON files utilize them.

JSON Files

The JSON files of the flatmap sources contain information on the visual aspects of the SVG file. Here, knowledge on what the shapes in the SVG represent and the manner of their connections are laid out in the form of annotations. The identifiers used must match those in the SVG, and the layout of the files are very specific. For more information see the docs section of mapmaker.

Anatomical_map.json

The anatomical_map.json contains classes of features and their relative ontological ID - typically UBERON or ILX, as per the SciCrunch database. The layout is typically as follows:

{  
  “CLASS_ID”: {  
      “term”: “ONT_TERM”,  
      “name”: “FEATURE_NAME”  
  }   
}

Properties.json

The properties.json file contains information on what features belong to which classes, as well as the connectivity knowledge between features. The structure can be split into two sections - the features definitions and the networks definitions.

Features in properties.json

The features section layout is typically as follows:

{  
  “features”: {  
      “FEATURE_ID”: {  
        “class”: “CLASS_ID”  
  }   
}

The feature_ID must be consistent with the ID used in the SVG file, for the relative feature(s). The class_ID must be consistent with the relative class that has been set up in anatomical_map.json.

Networks in properties.json

The networks section layout is typically as follows:

{  
  “networks”: [  
    {  
        “centrelines”: [  
            {  
                “id”: “CENTRELINE_ID”,  
                “models”: “NERVE_ONT_TERM”  
                “connects”: [  
                    “FEATURE_ID1”,  
                    “FEATURE_ID2”  
                ],  
                “contained-in”: [  
                    “FEATURE_ID3”  
                ]  
        }  
  }  
}

The fields in the networks section are explained in Table 4. As with the anatomical_map.json, the identifiers used must be consistent with those in the SVG file.

Table 4: The required and optional fields for the structure of centrelines in the properties.json file.

Manifest.json

The manifest.json ties together the other JSON files and the SVG file. The structure of the manifest is typically as follows:

{  
    “id”:”FLATMAP_NAME”,  
    “models”: “FLATMAP_ONT_ID”,  
    "description”: “description.json”,  
    “properties”: “properties.json”,  
    “sckan-version”: “production”,  
    “sources”: [  
        {  
            “id”: “FLATMAP_ID”,  
            “href”: “FLATMAP_SVG”,  
            “kind”: “base”  
        }  
    ],  
    “anatomicalMap”: “anatomical_map.json”,  
    “connectivityTerms”: “connectivity_terms.json”,  
    “neuronConnectivity”: [  
        “https://apinatomy.org/uris/models/ard-arm-cardiac”,  
        “https://apinatomy.org/uris/models/bolser-lewis”,
        “https://apinatomy.org/uris/models/bronchomotor”,
        “https://apinatomy.org/uris/models/keast-bladder”,
        “https://apinatomy.org/uris/models/pancreas”,
        “https://apinatomy.org/uris/models/sawg-distal-colon”,
        “https://apinatomy.org/uris/models/sawg-stomach”,
        “https://apinatomy.org/uris/models/spleen”
    ]  
}

Given these files are set up in the correct manner, the desired flatmap should be able to generate with no issues.

Examples

As can be seen in Figure 24, from the anatomical_map.json file of the rat_flatmap files, the class urinary_2 represents the urinary bladder of the body.

The features in the SVG that represent the urinary_2 class are defined in the properties.json.

Note that the use of “-1” in the feature ID allows multiple features to belong to the same class (i.e. in Figure 26, urinary_7-1 and urinary_7-2 both belong to the urinary_7 class).

Within the networks section of the properties.json, urinary_13-1 is referenced as the endpoint of the centreline n_58.