Skip to main content

Snowflake Export Testing

Depot supports Snowflake export to S3 via transactions.

How to write export actions/transactions can be found here.

Writing export tests​

In most cases you will need to use stage overrides to test export functionality

In order to invoke export in test context add export action on DepotTest builder with same parameters that you would call export action To assert what was exported add .checkExport action on DepotTest builder where:

  • exportPath matches export action exportOptions.path
  • sourceSchema matches export action source
  • actual is singleton array that matches export action target
  • expectedUri is folder path to expected files

Currently depot-test supports only JSON and simple comma-separated CSV with header.

info

For other types or custom CSV assertions will be user side responsibility. Until Depot-test has ability to return files ask Depot team to help with workaround.

DepotTest.in(TestNs)
    .export({
        exportOptions: {
            path: "/path/",
            formatSchema: "my.schema.Format"
        },
        target: "my.schema.Stage",
        source: "my.schema.Source",
    })
    .checkExport({
        exportPath: "/path/",
        sourceSchema: "my.schema.Source",
        actual: ["my.schema.Stage"],
        expectedUri: "/some/path/expected/"
    })

if createAssertionFiles flag is true(default) files be created in folder matching assertions under expectedUri

Permissions

To run export tests it's necessary to set up two permissions:

Snowflake integration​

How to create snowflake integration:

  • create IAM role with trust relationship
  • create snowflake integration matching pattern <DB_NAME>_S3 where db name is the same as in depot.properties
  • update IAM role trust relationship with credentials from integration
  • add access to S# bucket policy to the IAM role

full guide can be found here

S3 access local runs​

To provide access for depot-test add aws.profile to depot-test and make sure your user have access to the bucket that you use in stage.url

S3 access for aws-codeBuild​

To provide access for depot-test in code build add access policy with s3 access to the code build role.

Role can be located codeBuild run "build details" under "Environment" with field name "Service role"

example s3 policy

{
    "actions": [
        "s3:*"
    ],
    "resources": [
        "arn:aws:s3:::some-service-ci",
        "arn:aws:s3:::some-service-ci/*"
    ]
}

S3 access for github actions​

To provide access for depot-test in github action you will need:

OIDC role (sphinx pipeline factory create it for all services)

to setup OIDC connection for your test job:

In your workflow file: OIDC requires AWS_ROLE_TO_ASSUME secret to be present at job context.

Add permissions to read Github id tokens(necessary for OIDC action). On job level add

    permissions:
        id-token: write
        contents: read

and add OIDC to test step

-   name: Setup OIDC Credentials
    uses: aws-actions/configure-aws-credentials@v1
    with:
        aws-region: "eu-west-1"
        role-to-assume: "${{ secrets.AWS_ROLE_TO_ASSUME }}"
        role-duration-seconds: 7000

Make sure that your OIDC role has permissions for specific action for example if in case CI is run on pull request you will need to add pull request token to OIDC role trust relationship i.e. "token.actions.githubusercontent.com:sub": "repo:stage-tech/<repo name>:pull_request" in

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Principal": {
                "Federated": "<redacted>"
            },
            "Action": "sts:AssumeRoleWithWebIdentity",
            "Condition": {
                "StringEquals": {
                    "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
                },
                "StringLike": {
                    "token.actions.githubusercontent.com:sub": "repo:stage-tech/<repo name>:pull_request"
                }
            }
        }
    ]
}
warning

OIDC role in sphinx project is managed by pipeline factory therefore your changes could be overridden. Ticket to tackle this here

also OIDC role must have access policy to the bucket used in CI for export test i.e.

example s3 policy:

{
    "actions": [
        "s3:*"
    ],
    "resources": [
        "arn:aws:s3:::some-service-ci",
        "arn:aws:s3:::some-service-ci/*"
    ]
}