Skip to main content

Data flows

The primary way to get data into and out of FarEarth is through cloud storage providers. You can also upload and download data through a web browser and download data with our APIs. All raw and processed data remains your property. Data is controlled and never shared with other users.

FarEarth supports all the major cloud storage providers, including Amazon's AWS S3 buckets, Azure Blob Storage, and Google GCS.

  • Data can be ingested into FarEarth from a storage provider using a Pickup
  • Processed data can be stored to a storage provider using an Archive

Image that shows how data flows through FarEarth

Storage accounts

We will provide you with a storage account dedicated to your data. Alternatively, you can configure FarEarth to use your own storage account. A typical configuration is to use your FarEarth storage account for data ingestion through a Pickup, and your own external storage account for processed data.

Data retention

  • You can configure FarEarth to remove raw, unprocessed data automatically after a period of time
  • FarEarth will automatically maintain a Rolling Archive when you use FarEarth's storage. Once a configured threshold is reached, the oldest data will be removed
caution

This is your responsibility to move the data from the FarEarth Archive storage account into your own storage.

Data structure

Most storage providers use a virtual filesystem that mimics typical filesystems usually referred to as "prefixes". For example, a prefix of "/data/input/" mimics a classic folder structure.

For a FarEarth Pickup, you configure a prefix to indicate which folder to monitor for new data

Products can be ingested for processing as a single binary file, or as a folder containing multiple files. FarEarth needs to know how to detect these products, how to group the files together, and when the upload to the cloud storage provider is completed. See configuring Pickups

For a FarEarth Archive, you configure a prefix to determine the folder structure for how data should be stored

Processed data is organised into a neat structure in the Archive. The folder structure makes it easy to find the product directly on the cloud storage providers' infrastructure. The folder structure can be configured. The default structure is spacecraft/product-type/year/month/day/product-id/uuid.

Alternative ways to ingest and retrieve data

You can upload data to be processed and/or stored in FarEarth through a web browser. Data can be downloaded directly from the Catalogue. These methods are useful during the commissioning phase to quickly exchange sample acquisitions and products.