This page outlines the various ways that files can be uploaded to the DAT-Core.

Introduction

SPARC investigators are required to upload their scientific files to the DAT-Core. There are multiple ways for users to upload files to the platform. Here, we describe the various options and provide links to tutorials describing step by step instructions to upload files.

Step 1: Creating a dataset

All files belong to a dataset on the DAT-Core platform. A dataset contains both files and metadata and can be selectively shared with other users of the platform. You can create datasets through the web application or using one of the client tools. Click here to see more detailed information on how to create datasets on the DAT-Core platform.

Step 2: Uploading files

Once you have created a dataset, you can upload files to this dataset. You can create folders to separate files in a logical structure. After files are uploaded, you can associate files with metadata records. There are number of ways to upload files:

Upload files using the browser

Within a dataset, you can navigate to the ‘Files’ tab. This is the main location where all files are organized.

file upload

In the ‘Files’ tab, you can create folders to organize your files on the platform. Click on the “New Folder” button to create a new folder. You can navigate into a folder by clicking on the folder in the file browser.

To upload files, you can drag files from your local machine directly into the file browser panel. A popup window will show the files that will be uploaded. Note that two of the files in the example below are combined as they belong to the same timeseries package.

file upload

Click ‘Start Upload’ to begin uploading the files. During the upload process, you can safely hide the popup window as files will continue to be uploaded in the background. Once files are uploaded to the platform, they will show up in the file browser. If you drag a folder into the upload window, all nested folders will also be uploaded and the folder hierarchy will be maintained on the platform.

Depending on the file format, the platform will process the data to enable the data to be viewed in the browser based data viewer. Processing of the data can take a while and you will be notified when this has completed.

For more information, see the File Upload Documentation for Blackfynn

Upload folders/files using the Blackfynn CLI.

Blackfynn provides a Command Line Interface that can be used for uploading both small and very large datasets. Detailed information can be found here:

Uploading files with the Blackfynn CLI

Upload files using Python

You can use the Open Source Blackfynn Python Library to upload files to the DAT-Core platform. In Python, you will first need to import the library and establish a connection to the platform. Then retrieve or create a dataset. The dataset object has a method to upload files to the dataset as the following example highlights. Note that uploading large files is currently not supported for Python, please use the Blackfynn Agent to programmatically upload large files. The Python client will rely on the Agent for uploading files in future releases.

 # import Blackfynn
 from blackfynn import Blackfynn

 # create a client instance
 bf = Blackfynn()

 # create a dataset
 ds = bf.create_dataset('Timeseries Dataset')

 # upload data (3 files)
 ds.upload('example_data/test.edf', 'example_data/testData.nev', 'example_data/testData.ns2')

More information can be found in the Blackfynn developers documentation for Python and the sourcecode for the Python Library here.

Upload files using MATLAB

Finally, you can also upload files using the MATLAB client. The MATLAB client relies on the Blackfynn Agent to upload the files so you will have to both install the MATLAB client as well as the Blackfynn Agent. The upload command is a method of the BFDataset class and can be used as follows:

>> bf = Blackfynn();
>> ds = bf.datasets(1)

BFDataset with properties:

    description: 'This is a dataset for the second tutorial.'
         models: [1×3 BFModel]
          items: [0×0 BFBaseDataNode]
           name: 'tutorial2'
           type: 'DataSet'

>> ds.upload('~/Downloads/example_data');

A detailed tutorial on how to upload data using MATLAB can be found in the Blackfynn developers documentation for MATLAB and the sourcecode for the MATLAB client can be found here.