SPARC Portal Homepage

Documentation, Examples, and Software for SPARC Map Reuse

SPARC provides documentation and examples to ensure that published maps are accessible to biomedical users.

Suggest Edits

To support the goal of making our research products available for reuse we need to provide documentation and examples to ensure that the SPARC maps we publish to the SPARC Portal (and related software) are accessible to end biomedical users. This includes the flatmaps, organ scaffolds, and scaffold maps, as well as the software required for non-expert users to be able to access these maps and the underlying computational artifacts from which they are derived. A high-level overview of the SPARC data curation and mapping efforts is provided in The SPARC DRC: Building a resource for the autonomic nervous system community.

Flatmaps

📘

Creating and displaying your own flatmap

Detailed instructions on creating and displaying your own flatmaps are available on the Anatomical Flatmap Resources page.

Flatmaps published on the SPARC Portal provide two-dimensional graphical representations of the anatomy and topology of the connectivity of the peripheral nervous system, in a Google Maps-like web-interface. The flatmaps allow knowledge captured in the SPARC Connectivity Knowledge Base of the Autonomic Nervous System (SCKAN) to be visualized and interactively explored on the SPARC Portal, including presenting relevant SPARC data as a user navigates through the interface.

The flatmaps published on the SPARC Portal consist of several distinct components summarized here and expanded in detail below.

  • The base anatomical diagrams: the manually-drawn anatomical cartoons, which form the base of each species’ flatmap, annotated with anatomical identifiers corresponding to those used in SCKAN.
  • The connectivity knowledge: captured via ApiNATOMY knowledge models and other knowledge sources and encoded as consistent connectivity knowledge in SCKAN.
  • The map maker: a tool which combines the annotated base diagrams with the connectivity knowledge to automatically render the connectivity into map tiles to be used in presenting the flatmaps.
  • The flatmap viewer application: the JavaScript application to render flatmaps obtained from a server and allow a user to interactively explore them.
  • FlatmapVuer: the widget used on the SPARC Portal that wraps the flatmap viewer to provide SPARC specific functionality.
  • The flatmap server: a web server that contains generated flatmaps, both map tiles and metadata about the flatmaps.

The figure below shows the relationship between each of these components.

Anatomical diagrams

These are the manually-drawn cartoons which provide the base diagram for each species’ flatmap. They are archived in PMR as SVG images (e.g., rat flatmap sources). The SVG is annotated with the anatomical terminology used in SCKAN.

Connectivity knowledge

SCKAN is built using information from both ApiNATOMY and the Neuron Phenotype Ontology.

The ApiNATOMY workflow used to build part of SCKAN is shown below.

SCKAN has a structured semantic form and provides endpoints for accessing and searching its knowledge. The latest version of the form can be found on Zenodo. Instructions on how to get started can be found here.

Importantly, the consistent use of anatomical terminology across the SPARC program enables rich semantic linking between SCKAN and SPARC data. In the case of the flatmaps, SCKAN is queried for connectivity knowledge which is then used by the map maker to render the topological connections between, through, and around anatomical entities in the anatomical diagrams.

The map knowledge package provides a Python wrapper for flatmap-related SCKAN queries, and an additional package, map tools, allows SCKAN connectivity to be viewed and explored in a Jupyter notebook.

Map maker

Map maker is the tool which combines the anatomical diagrams with the connectivity knowledge to produce the rendered map tiles which form the actual flatmap visualization.

Map maker’s source code is hosted on GitHub, which is where issues and feature requests can be raised, and documentation can be found here.

Connectivity knowledge from SCKAN is used to identify which segments of a pre-drawn (but hidden) nerve network that the path of each neuron population group traverses. These paths are then ordered within each nerve segment to minimize the number of crossings between paths, and drawn using an offset from the segment’s centreline.

Releases are versioned in GitHub.

Flatmap viewer

This is a viewer for anatomical flatmaps generated by map maker. The viewer is intended to be a component of a larger JavaScript web application, although it may be used standalone for local flatmap development and testing. Flatmap content is obtained from a flatmap server. The viewer application is based on the MapLibre open-source mapping library and uses industry standard interchange formats for web-based geographical maps.

The flatmap viewer’s source code is hosted on Github, which is where issues and feature requests can be raised and documentation can be found here.

The viewer is integrated with the SPARC portal via the flatmapvuer Vue component (see below). It can be integrated into a map server to allow a server’s maps to be explored independently to the SPARC Portal, as described in the map server’s documentation.

Releases are versioned in Github and available from the NPM repository.

Flatmap server

The flatmap server is the infrastructure where the generated map tiles are archived and made accessible to a flatmap viewer for display of flatmaps.

The flatmap server’s source code is hosted on Github, which is where issues and feature requests can be raised and flatmap server documentation can be found here.

The flatmap server application includes mapmaker and provides a /make/map endpoint to generate flatmaps described by a source manifest on PMR.

  • This process can be started using PMR’s Exposure Creation Wizard (see Publishing a Flatmap as a SPARC Dataset).
  • Alternatively it can be started by issuing a HTTP POST request to the make/map endpoint, for instance from a shell prompt:
$ curl -H "Content-Type: application/json" -X POST  
       -H "Authorization: Bearer TOKEN"  
       -d '{"source":"’PMR_MANIFEST_URL'"}'  
       MAP_SERVER_URL/make/map

Generated flatmaps are assigned a unique identifier, being the SHA 256 hash of its manifest’s URL. Additionally the full URL of the manifest becomes part of the flatmap’s metadata (its source attribute), and this URL includes the PMR revision of the map.

The map-server provides both MapBox vector tiles and image tiles to the viewer, using industry standard geographical formats, along with flatmap metadata and annotation. Additionally, a local knowledge store, containing cached SCKAN information, can be queried via a SQL endpoint.

Organ scaffold maps

Beyond the two-dimensional flatmaps, we also look to register SPARC data to three-dimensional organ scaffolds to provide maps in a common coordinate framework enabling disparate data to be integrated in order to improve our understanding of the collective SPARC knowledge.

The process for mapping SPARC data to the organ scaffolds is described in some detail in The SPARC DRC: Building a resource for the autonomic nervous system community, so here we focus on enabling the reuse of the organ scaffolds themselves, as well as specific instances of the scaffolds in a given scaffold map.

Documentation for all the mapping tools is available here. The following are quick links to the specific tools used in the SPARC default organ scaffolds workflow:

Generic organ scaffolds

Quarterly releases of the generic organ scaffolds are published to the SPARC Portal. In the same manner as the source flatmap anatomical diagrams, the source of these generic scaffolds are archived in PMR and then published to the SPARC Portal in a manner aimed at supporting reuse.

When a generic scaffold is archived in PMR a provenance record is stored in the archive that explicitly states the version of the software used. The provenance record can be used to recreate the environment that the archived data was created with. With the software environment and inputs to the workflow we can reproduce the data published to the SPARC Portal locally.

Scaffold reuse

Scaffold Creator is a plugin to the MAP Client workflow tool that can be used to generate organ scaffolds. Scaffold Creator uses a small number of parameters to configure the generation of a specific instance of an organ scaffold. Common features between organs and/or species are abstracted into reusable templates and then Scaffold Maker, the library behind the plugin, uses the given configuration to generate the desired finite element model from the available templates. The authoritative source for any of the generic organ scaffolds consists of the configuration used to generate it and the specific version of Scaffold Maker used to generate it - this is the information that is archived in PMR, along with the generated finite element model in the internal Scaffold Maker format.

We also archive a visualization configuration of the organ scaffold in PMR which, when combined with the organ scaffold configuration and provenance information defines all the required information to re-create a graphical rendition of the organ scaffold as available on the SPARC Portal.

To make use of an existing organ scaffold first install the Scaffold Mapping Tools following the instructions on installation from the SPARC Portal. The installation will include MAP Client, Scaffold Creator, and Scaffold Maker. Download the SPARC generic scaffold workflow from GitHub- here we refer to that as the workflow directory. Now we can launch the MAP Client application and open the SPARC generic scaffold workflow.

To reuse an organ scaffold from an existing dataset, download the dataset from the SPARC Portal and take the following steps:

  • Copy the OrganScaffold-settings.json file from the target dataset to the workflow directory.
    • In some cases the settings file has a different name, for example; mouse_lungs_mesh_settings.json. In this case copy the mouse_lungs_mesh_settings.json to OrganScaffold-settings.json in the workflow directory.
  • Copy the ViewCreation-backup-document.json from the target dataset to the workflow directory.
  • Configure the output of the webGL and thumbnail Argon Scene Exporter steps to a suitable directory on the local disk. When creating a dataset for the SPARC Portal this is a location under the derivative directory for the dataset.
  • Save the workflow.
  • Execute the workflow.

When running the SPARC generic scaffold workflow:

There are two user interactive steps: Scaffold Creator (with identifier OrganScaffold), and Argon Viewer (with identifier ViewCreation) and there are also two non-interactive steps for exporting the view, one with the identifier WebGLExport, and one with the identifier ThumbnailExport.

The first interface encountered when executing the workflow is the Scaffold Creator interface where modifications to the scaffold are made. In the Scaffold Creator interface we can change species, variant, finite element density, etc. When first executing the workflow, the interface will be configured as per the scaffold from the target dataset. The Done button is used to confirm the organ scaffold configuration and move to the next step of the workflow.

The second interface encountered is the Argon Viewer interface where we can create or modify an existing view of the scaffold. The Argon Viewer documentation explains in detail the functionality of the interface, which will be configured as per the scaffold visualization from the target dataset on first execution. The Done button on this interface finishes the workflow and returns to the initial view of the workflow.

Scaffold publication to the SPARC Portal

As described above, Scaffold Creator is used to generate specific instances of the generic organ scaffolds. To aid the reuse, when publishing the generic organ scaffolds the datasets are enriched with additional data for visualization of the scaffold on the SPARC Portal and common export formats for the underlying finite element models.

In the SPARC generic scaffold workflow we finish the workflow exporting the view. The exported files can be configured to export to the derivative directory of a SPARC dataset. It is usual to export these files to a directory named scaffold in the derivative directory.

There are two types of export formats; webGL, and thumbnail. The webGL export is a format that ScaffoldVuer (see below) can consume and render the exported data in the browser. The thumbnail export is a JPEG image which is used to give visual feedback on the exported data without actually loading the data in ScaffoldVuer.

When the correct annotation is stored in the dataset and uploaded to the SPARC Portal the SPARC Portal is able to interpret the exported data and create an interactive view of the organ scaffold in the browser.

Scaffold maps

When the Scaffold Creator is combined with the Geometry Fitter, Data Embedder, and an importer the generic organ scaffolds can be extended to display mapped data. A complete description on how to map data onto an organ scaffold can be found on the SPARC Portal.

Scaffold and Flatmap visualization on the SPARC Portal

Approved flatmaps and published scaffolds can be viewed on the portal with viewers created using the Vue.Js framework and these viewers can be reused and deployed elsewhere. Here is a brief description of the viewers:

ScaffoldVuer

ScaffoldVuer is the javascript front end application written using Vue.JS framework to visualize scaffolds. It currently accepts an in house json format, OBJ, STL and GLTF format. It has the capability to export currently visualized objects with the GLTF format.

FlatmapVuer

MapVuer

MapVuer is the integration of multiple different modules to provide a streamline workflow, the modules include scaffoldvuer, flatmapvuer, plotvuer and simulationvuer and a search engine (sidebar).

Flatmapvuer is the primary viewing tool on the mapvuer and the markers are added to the flatmap to indicate locations of interest (e.g., locations of SPARC datasets), clicking on these will trigger a search on the sidebar, then displays all relevant datasets and the corresponding actions such as open scaffolds, plots, simulations, and etc.

Measuring reuse

Beyond the tracking of dataset reuse (downloads, citations, etc.) already handled by the SPARC Portal, we will look to track all routes of reuse of the maps and supporting software. At a minimum, this will include tracking download or reuse of the open-source codes via Github and addition of maps to SPARC data that is not completed by MAP-Core. Tracking statistics for map research product reuse will be aggregated and made available to NIH.

A preliminary page has been set up to present map reuse statistics pulled from the Github repositories used for the mapping tools and dependent codes. This page dynamically updates with current data on page load. We will continue to add software information and additional statistics as they become available and will ensure the scaffold mapping SOP is updated to ensure new scaffold maps are able to be tracked in a similar manner.

Updated about 1 month ago