Archives

FarEarth's archives manage datastores. It supports many cloud storage providers. Archives store input data and processed products securely.

The archive protocols supported by FarEarth

• filesystem: used to integrate local filesystems directly into FarEarth (most commonly used in a standalone self-hosted version of FarEarth)
• azure: interfaces with Microsoft's Azure blob storage technology
• amazon-s3: interfaces with Amazon's S3 buckets
• gdrive: interfaces with Google Drive cloud storage systems
• google-gcs: interfaces with Google Cloud Storage buckets
• pyxis: support for FarEarth's own blob storage systems

Archive details

The table lists fields in an archive configuration.

Field	Example Value	Details
id	farearth.azure-archive-101 farearth.fs-archive-101	Unique identifier of the archive. The ID starts with the subscription ID (example, farearth). This ID is used to provide access to the archive for specific users and subscriptions, and tracks activity related to the archive
protocol	azure amazon-s3 google-gcs gdrive filesystem pyxis	The datastore protocol to use
prefix	FarEarth-Testing/{year}/{doy}/{version}	The virtual folder structure used when storing data. For more details, see prefix documentation
shareable	true false	Default: false Controls if the archive can be shared with other subscriptions. NOTE: setting this to true does not yet share the archive
accessModes	PUSH PULL ["PUSH", "PULL"]	Determines how FarEarth will get the data from the archive. If set to PULL, the component will fetch the data from the archive. It is possible to specify both PUSH and PULL
dataStoreLimits	capacity, warnOnRemaining, errorOnRemaining	Sets a capacity limit on the archive. Controls the behavior of the archive when nearing capacity. For more information see Datastore limits
properties		Unique properties required by the specific protocol.

Datastore limits

See the options for the dataStoreLimits below:

Datastore limit mode	Details
capacity	Sets the maximum available storage for the datastore. See Rolling archive
warnOnRemaining	Enables a warning when the archive has reached the setting limit
errorOnRemaining	Enables an error when the archive has reached the setting. It will prevent any further data writing to the datastore

An example dataStoreLimits configuration is given below.

    ...
    "dataStoreLimits": {
        "capacity": "64GB",
        "warnOnRemaining": "4GB",
        "errorOnRemaining": "2GB"
    }
    ...

In this example, the capacity is configured to 64GB. When 4GB or less remains, any further writing to the archive will trigger a warning. When 2GB or less remains, the warning is replaced by an error message prompting user action

Rolling archive

When a capacity limit is configured in the dataStoreLimits field of the archive configuration, FarEarth will implement a rolling archive once capacity is reached. In this mode, the oldest product by processed data will be deleted first to free space on the archive.

The capacity limit can be specified using MB, GB or TB. For example: 1000GB (or 1TB).

NOTE: Only data in the Catalogue contribute to the storage limit of the archive. Any data on the datastore that is not part of the Catalogue will not count towards the limit, which may cause the archive to take up more space than what is allocated by FarEarth

Learn more

To learn more about our secure data storage, read our blog: FarEarth has secure data storage!