Stage overrides

The current approach to ingesting data from a Stage in Depot is to use a Query to Select from the Stage using a FileFormat which can then be written into an Object schema.

Normally the Stage will be defined at the CDK level because it has to reference the S3 bucket location for the External Stage. This means that the Stage schema will not be defined at the Package level, but the Query Schema will be referencing the Stage.

To do package level testing we need to do the following things:

• Create a Stage Schema in the Test dir
• Merge the Package Schemas with the Test Schemas
• Define the Tests Stage Override with Input Data File

Below is an example Stage Schema which can live in the Test directory.

Example Stage Schema in Test dir:

staging.ExampleReferenceStage:
  type: stage

Within the test files we need to merge the Package and Test Schema’s together so that we have a namespace with the Stage schema included in it.

Below is an example of what is generally defined in the test classes currently and an example of some code which merged together additional test schemas with the package schemas

Example of current Package Namespace

example.test.ts

import * as pkg from '../index';
const namespace = JSON.stringify(pkg.mergedSchemas, null, 2);

Example of merging Package Namespace with Test Schemas

example.test.ts

import * as pkg from '../index';
const testPkg = dptPkg.Packages.depotPackage({
  name: "@stage-tech/my-schema",
  schemaDir: [path.join(__dirname, './staging.ExampleReferenceStage.yml')],
  imports: [],
});

const mergedPkg = dptPkg.Schemas.mergeUniqueSchemas(pkg.mergedSchemas, testPkg.mergedSchemas);
const namespace = JSON.stringify(mergedPkg, null, 2);

The final step is to define the TestStep Stage Override with input data file. This will allow an Internal Stage to be created and the file pushed up to the Stage.

As we can see in this step it is running a Query which writes into the FlatFile Object Schema. At the bottom we can see the Stage Override which references:

• The Stage Schema FQN
• The path to the data file to be put into the Stage
• The File Format to set the Stage with (Optional)

Example DepotTest upsert with Stage Override

const scenario = DepotTest.in(pkg.mergedSchemas)
    .withAdditionalSchemas(path.join(__dirname, './staging.ExampleReferenceStage.yml'))
    .upsert({
        source: 'staging.FlatFileQuery',
        arguments: {
            assetId: '01856eeb427eec93ac3d95b73f0697cb8aaf481b',
            referenceCopyKey: '.*/01856eeb427eec93ac3d95b73f0697cb8aaf481b.gz',
        },
        stage: {
            name: 'staging.ExampleReferenceStage',
            url: path.join(stagingTestDir, 'e2e', 'data', '01856eeb427eec93ac3d95b73f0697cb8aaf481b'),
            format: 'staging.FlatFileUtf8',
        },
        target: 'staging.FlatFile'
    })
    .check({
        expectedUri: path.join(stagingTestDir, 'e2e', 'expected'),
        actual: [`staging.FlatFile`],
    });

Example TestStep with Stage Override

const testStep: TestStep = {
    name: 'Stage data',
    operation: Operation.UPSERT,
    source: 'staging.FlatFileQuery',
    target: 'staging.FlatFile',
    arguments: {
        assetId: '01856eeb427eec93ac3d95b73f0697cb8aaf481b',
        referenceCopyKey: '.*/01856eeb427eec93ac3d95b73f0697cb8aaf481b.gz',
    },
    expectedUri: path.join(stagingTestDir, 'e2e', 'expected'),
    actual: [`staging.FlatFile`],
    stage: {
        name: 'staging.ExampleReferenceStage',
        url: path.join(stagingTestDir, 'e2e', 'data', '01856eeb427eec93ac3d95b73f0697cb8aaf481b'),
        format: 'staging.FlatFileUtf8',
    }
}