Skip to main content

Aurora Transactions

caution

Depot Aurora transactions support is still in active development, therefore please contact the Depot team if you find any issues

Aurora Depot Transactions use the same schema as standard Depot Transactions. This page exists currently to help list good to know information and “gotchas” about the format of Data and how Aurora Transactions work.

Operations​

UPSERT, INSERT_IGNORE, MERGE, DELETE​

These operations are supported normally, with a source that can be another entity within the same location, a source in a different location if that location supports cross-location transactions, or a file on S3 via a StageSchema entity.

PATCH​

This operation is supported normally. Structured object, map and array columns will be replaced wholesale; the PATCH semantics are only applied to scalar columns.

EXPORT​

This operation is supported normally for both CSV and JSONL. It will result in exactly one file being written; on the default executor, the volume to write must be consistent with AWS lambda execution timeout (15 minutes). If you need longer transactions, you must use the Long-Running executor which can have arbitrarily long execution times but incurs a start-up latency of several dozen seconds.

Data formats​

CSV​

To import data from CSV using Aurora-depot transactions, the import action must contain the columnOrder field which contains an ordered list of all columns in the file. If the columnOrder field is not present, or the header line doesn't match its contents, the transaction will fail.

For now, the list of columns must be specified in the CSV Import transaction (DPT-2616).

JSONL​

As of Postgresql 16, JSONL is not supported natively. Depot-Aurora uses a combination of text files and Postgres' own JSON parsing function to make JSONL imports and exports work.

However, it is not possible (as of Postgresql 16) to tell Postgres not to handle certain escape sequences text import. This should generally not have an effect on regular data but newlines or escaped backslashes (where the backslash is intended to be part of the content) might sometimes be dropped.

Creating Transactions​

All other Depot Transaction creation methods are valid for Aurora Transactions. For quick reference, here is how you would create a Transaction using the Depot CLI:

depot create-transaction --dataset 282d8404e895 --executor 871efk4421e6b --data "s3://example-ingest-data/demo-load-1/first.csv" --format CSV

To create a Transaction using the Depot Lambda Gateway directly, you would invoke the Lambda function with a payload like this:

{
  "schema": "Transaction",
  "operation": "create",
  "data": {
    "dataset": {
      "id": "dataset123"
    },
    "actions": [
      {
        "target": "pet.PetStore",
        "operation": "REFRESH"
      }
    ],
    "context": {
      "trace": "xx1"
    }
  }
}