Data Upload Best Practices

How do I upload large datasets?

Large datasets that are over 250GB and/or contain 1000 or more files or folders are becoming more and more common among SPARC researchers. The SPARC DRC is constantly improving our tools and resources to make data sharing more efficient and less painful for our users.

If you are uploading a large dataset, keep in mind how much it will cost users to download your files.

How long will it take me to upload my dataset?

Uploading large datasets can sometimes take a hours. Many factors influence how long it might take for your dataset to be uploaded, such as the size and number of your dataset's files, your internet speed, and your local computing hardware and connections to your data storage location. For example:

• Uploading a data folder of large imaging files consisting of 308GB with a 1000 Mbps hardwired internet connection took 1hr 20mins
• Uploading a dataset consisting of 55,000 files and 1.5TB stored on an external hard drive usb connected to a local desktop took about 18hrs to upload on 1000 Mbps hardwired internet connection.

Here are a few tips to keep upload time to a minimum:

• Be sure you are using the most up-to-date version of the SPARC upload software: SODA and the Pennsieve Agent
• Utilize your institution's hardwired internet connection to access transfer speeds of around 1000 Mbps (most wireless networks limit upload speed and total amount of data transferred within a given time)
• Your computer or server will need to be on and connected to the internet for the entirety of the upload. Try to use computers that will not need to be interrupted for hours at a time. New feature alert: uploads that are stopped or interrupted can now be restarted without staring over from scratch!

Have a single file over 220GBs?

The standard settings of the SPARC upload software may need to be changed to allow for files over 220GBs to be uploaded without interuption. Please email DAT-Core before you begin your upload with your specific files sizes so we can help you configure your settings for a smooth upload experience.

Need help or have questions?

Email DAT-Core Our team is here to help you.