Pickups

FarEarths's datapoints manage various data sources. It supports most cloud storage providers.

Ingested data is typically processed with FarEarth, or stored directly in the Archive.

Pickup protocols supported by FarEarth:

filesystem: used to integrate local filesystems into FarEarth (usually used in a FarEarth self-hosted version)
azure: interfaces with Microsoft's Azure blob storage
amazon-s3: interfaces with Amazon's S3 buckets
gdrive: interfaces with Google Drive cloud storage systems
google-gcs: interfaces with Google GCS buckets
pyxis: support for FarEarth's own blob storage systems

Pickup details

The table lists fields in an Pickup configuration.

Field	Example Value	Details
`id`	`farearth.azure-pickup-101` `farearth.fs-pickup-101`	Unique identifier of the Pickup. The ID starts with the subscription ID (example, `farearth`). The ID provides access to the Pickup for specific users and subscriptions. It tracks any activity related to the Pickup
`protocol`	`azure` `amazon-s3` `google-gcs` `gdrive` `filesystem` `pyxis`	The data source protocol to use. This field determines the required properties
`dataRetentionWindow`	`12 hours`	Defines how long data is retained in the Pickup before being automatic removal
`shareable`	`true` `false`	Default: `false` Controls Pickup sharing with other subscriptions. NOTE: setting this to `true` does not share the Pickup
`accessModes`	`PUSH` `PULL` `["PUSH", "PULL"]`	Default: `["PUSH","PULL"]` Determines how FarEarth interacts with the Pickup. If set to `PULL`, FarEarth initiates the request to draw data from the Pickup. If set to `PUSH`, the Pickup actively initiates the transfer to deliver the product data to FarEarth. It is possible to specify both `PUSH` and `PULL`. It is important to match the `accessMode` with the target protocol's configuration.
`attachParameters`	`workflowId` `sendNotification`	Add optional parameters to control the automated Pickup. See additional parameters
`dataStoreLimits`	`capacity`, `warnOnRemaining`, `errorOnRemaining`	Sets a capacity limit on the Pickup. It controls the behavior of the Pickup when nearing capacity. See Datastore limits
`properties`		Unique properties required by the specific protocol
`triggers`	`READY_FILE` `NO_CHANGE` `FILE_MANIFEST` `MD5_FILE` etc.	Controls the trigger mechanism used by this Pickup to accept input data. See Triggers

Additional parameters

Additional parameters can be added to a Pickup configuration.

Parameter	Details
`workflowId`	Specifies a specific workflow to run on data ingestion
`sendNotifications`	Sends a notification once data in this Pickup completes processing

Datastore limits

The options for the dataStoreLimits are:

Datastore limit mode	Details
`capacity`	Sets the maximum available storage for the Pickup
`warnOnRemaining`	Enables a warning when the datastore has reached the the limit specified by this setting
`errorOnRemaining`	Enables an error for when the datastore has reached the limit specified by this setting

Triggers

Triggers are how FarEarth identifies data for the specific Pickup. It determines how FarEarth knows which files belong to a specific product, and when all of the data for that product has finished loaded to the Pickup location.

info

Input data ingested via a web browser interface, such as Google Drive, should be placed inside a folder.

Each Trigger method has additional parameters to set.

Trigger	Additional parameters	Details
`ACCEPT_ALL`	`"delay": "5m"`	All files after a specified delay (5 minutes in the example)
`NO_CHANGE`	`"delay": "60s", "minimumFileCount": 3`	All files after a specified delay (60 seconds in the example) if the minimum number of files are reached (3 in the example)
`FOLDER_SUFFIX`	`"busySuffix": "_busy", "doneSuffix": "_completed"`	A folder with a specific "done" suffix (`_completed` in the example). This is useful for cases where folder renames are atomic; data is copied to a folder (with a `_busy` suffix), once completed is renamed instantly with the correct "done" suffix
`FILE_MANIFEST`	`"suffix": "_manifest.txt"`	Triggers once a manifest file with the specified suffix containing a list of files is present. The manifest file should be the last file created to ensure that all files have finished copying into the Pickup location. See manifest files
`READY_FILE`		Triggers once a sidecar file with a `*.READY` extension is detected. The sidecar file must have the same name, excluding the ready-extension, as the file to be ingested by the Pickup. See READY files
`PRODUCT_FILE`	`"suffix: "_product.json"`	Triggers when a STAC product file, with the specified suffix, is detected. The product file must reside in the products' folder to be ingested by the Pickup. See product files
`MD5_FILE`	`"suffix": ".MD5"`	Triggers when a file containing the MD5 checksums of a list of files is detected. The MD5 file should be within the folder of the product to be ingested by the Pickup. See MD5 files

info

The suffixes provided in the table above are the default suffixes for each Trigger. They can be omitted from the configuration if the defaults will suffice.

Below, an example of the Triggers in JSON.

    "triggers": [
        {
            "id": "ACCEPT_ALL",
            "delay": "5m"
        },
        {
            "id": "NO_CHANGE",
            "delay": "60s",
            "minimumFileCount": 3
        },
        {
            "id": "FOLDER_SUFFIX",
            "busySuffix": "_busy",
            "doneSuffix": "_completed"
        },
        {
            "id": "FILE_MANIFEST",
            "suffix": "_manifest.txt"
        },
        {
            "id": "READY_FILE"
        },
        {
            "id": "PRODUCT_FILE",
            "suffix": "_product.json"
        },
        {
            "id": "MD5_FILE",
            "suffix": ".MD5"
        },
    ]

Trigger file examples

Manifest files

An example of a manifest file is provided below.

5815_5_20231019132435_TRUBIT-1.raw 13779941880
6839_5_20231019124755_TRUBIT-1.raw 15774919584

Each line in the text file contains the name of a file that is part of the product, followed by its size in bytes.

READY files

An example of a READY file is provided below. The file extension is by default *.READY.

{
  "workflowId" : "farearth.archive",
  "properties" : {
    "id" : "1-TRUBIT-1-20220120-KANSAS-5",
    "gsd" : [ 5.0, -5.0 ],
    "productType" : "imagery",
    "dataset" : "qa-data",
    "datetime" : "2022-01-20T21:15:30Z",
    "spacecraft" : "TRUBIT-1",
    "instruments" : [ "TB-100" ],
    "bbox" : [ ],
    "eoBands" : [ {
      "name" : "B01",
      "commonName" : "Blue",
      "centerWavelengthNanoMeter" : 490.0,
      "fullWidthHalfMaxNanoMeter" : 65.0
    }, {
      "name" : "B02",
      "commonName" : "Green",
      "centerWavelengthNanoMeter" : 560.0,
      "fullWidthHalfMaxNanoMeter" : 35.0
    }, {
      "name" : "B03",
      "commonName" : "Red",
      "centerWavelengthNanoMeter" : 665.0,
      "fullWidthHalfMaxNanoMeter" : 30.0
    }, {
      "name" : "B04",
      "commonName" : "NIR",
      "centerWavelengthNanoMeter" : 842.0,
      "fullWidthHalfMaxNanoMeter" : 115.0
    } ],
    "geometry" : {
      "type" : "Point",
      "coordinates" : [ -101.40862, 39.18863 ]
    }
  },
  "inputFiles" : {
    "1-TRUBIT-1-20220120-KANSAS-5.TIF" : {
      "id" : "IMAGE",
      "title" : "Unprojected image",
      "roles" : [ "data" ],
      "type" : "image/tiff"
    }
  }
}

The READY file must contain the parameters provided in the table below. The fields are part of the STAC specification, with additional fields that can control the FarEarth processing.

Key	Details
`workflowId`	FarEarth Workflow ID to process this data
`productType`	STAC product type of the input data
`datetime`	Date and time of the input data (format: `yyyy-mm-ddThh:mm:ssZ`)
`inputFiles`	One or more files to process (see required fields below)

The input files must contain the below.

Key	Details
`inputName`	File name on disk
`id`	STAC-compliant ID as part of a Collection
`title`	Descriptive title for the input file

Product files

An example of a product file is provided below.

{
  "type" : "Feature",
  "id" : "1694394734_small",
  "assets" : {
    "RAW" : {
      "title" : "RAW ses file",
      "href" : "sample_500MB.raw",
      "size" : 524288000
    }
  },
  "bbox" : null,
  "geometry" : null,
  "properties" : {
    "spacecraft" : "TRUBIT-1",
    "datetime" : "2023-05-18T07:33:56Z",
    "id" : "1694394734_small",
    "productType" : "RAW"
  },
  "links" : [ ],
  "stac_version" : "0.9.0"
}

The product file format is that of the STAC specification.

MD5 files

An example of an MD5 file is provided below.

9B68D20CD048E31017207ED91817DC7C sample_500MB.raw

Each line in the text file contains the name of a file that forms part of the product, preceded with the checksum of the file.

Pickup details​

Additional parameters​

Datastore limits​

Triggers​

Trigger file examples​

Manifest files​

READY files​

Product files​

MD5 files​