Advanced Options
When running Depot tests, all of the packages schemas, including any dependencies, are created in the same test database specified in your depot.properties configuration file. It is possible to customise test behaviour and configuration using some supported environment variable options.
Environment variable options
The Snowflake schemas are created with a randomly generated test prefix in order to isolate the tests, for example, TEST_1637230633_ADF1_TYPED_ABC_DEF_10. These schemas are then deleted at the end of the test.
SNOWFLAKE_SCHEMA_PREFIX
To create schemas in your own database that exactly match the naming when
deployed to an environment, use SNOWFLAKE_SCHEMA_PREFIX=
To aid testing and debugging, you can specify the SNOWFLAKE_SCHEMA_PREFIX environment variable to assign a designated
prefix to your test schemas. When specified, schemas are created with the prefix, for example, MY_TEST_ADF1_TYPED_ABC_DEF_10.
The schemas are not deleted at the end of the test so you can inspect the results of the test.
However, they are deleted at the start of the next test run to make sure we always start with a clean set of test data.
if you set the SNOWFLAKE_SCHEMA_PREFIX to an empty string, and choose the SQL names of the referenced schemas under the
using clause of a view or query schema to match the name under which Depot deploys them, you may be able to
connect your IDE to your database and live edit the sql watching the results as you go.
if you set the SNOWFLAKE_SCHEMA_PREFIX and run tests in parallel, tests will run sequentially as the
schema prefix is held exclusively by the first test to run.
However, until
KEEP_SNOWFLAKE_SCHEMA
Set KEEP_SNOWFLAKE_SCHEMA=true to prevent the test schemas from being deleted at the end of a test run. Be sure to manually clean up any remaining test data yourself when using this option.
DEPOT_TEST_MAX_SCHEMA_OPS
Integer, defaults to 7. This controls the maximum number of simultaneous schema definition operations that the backend will perform at once.
DEPOT_TEST_MAX_DATA_OPS
Integer, defaults to 16. This controls the maximum number of data movement operations that the backend will perform at once. It should not be much more than the capacity of the underlying Snowflake warehouse; additional queries would be queued anyway.
DEPOT_TEST_MAX_TOTAL_OPS
Integer, defaults to 65536.
When smaller than (DEPOT_TEST_MAX_SCHEMA_OPS + DEPOT_TEST_MAX_DATA_OPS), it can be used to
further restrict the volume of traffic generated at once by the backend.
When larger than (DEPOT_TEST_MAX_SCHEMA_OPS + DEPOT_TEST_MAX_DATA_OPS), it has no effect in practice.
These three limits apply to AURORA and SNOWFLAKE jobs. A FAKESNOW job is not bounded by them: it queues at
its emulator instead, and the lever for fakesnow throughput is the instance count,
DEPOT_TEST_FAKESNOW_INSTANCES.
DEPOT_TEST_IDLE_DURATION_SECONDS
Integer, defaults to 900 (15 minutes). This controls the maximum amount of time that the backend will wait for additional operations before it shuts down.
DEPOT_TEST_BACKEND_LOCKFILE
Text, must be a valid file path. This controls the location of the lockfile through which test runs find and share a backend.
It defaults to a file in the temporary directory, named after the platform version that depot-test bundles and the fakesnow image.
All the test runs that set the same lockfile share one backend, whatever their fakesnow image.
It can be overridden for debugging purposes.
DEPOT_TEST_BACKEND_JAVA_OPTS
Text, must be a string with valid options for your JVM.
Defaults to -Xmx512m.
It is possible to supply an empty string to let the JVM decide for itself.
DEPOT_TEST_FAKESNOW_INSTANCES
auto or a positive integer, defaults to auto.
This controls the number of fakesnow instances the backend runs. auto sizes the pool to the machine.
See Testing against fakesnow.
DEPOT_TEST_FAKESNOW_IDLE_DURATION_SECONDS
Integer, defaults to 30. This controls how long a fakesnow instance stays idle before the backend stops it.
DEPOT_FAKESNOW_MEMORY
Text, a whole number of bytes, or a whole number with a unit k, m or g (binary), such as 2g or 1536m.
This controls the memory limit of each fakesnow container.
It defaults to a budget between 1 GiB and 4 GiB that the backend picks from the memory available to Docker.
Depot Local reads the same variable, but sets no limit when it is unset.
DEPOT_TEST_LOG_LEVEL
Must be one of "fatal", "error", "warn", "info", "debug", "trace". Controls the minimum logging level of depot-test's own traces. Defaults to "info".
The shared backend
The test runs on a machine share one backend for each platform version and fakesnow image (DEPOT_FAKESNOW_IMAGE).
The first test run to start that backend decides all its other settings: the variables that the table below marks, and anything else that the backend reads from its environment when it starts.
Later test runs use the backend as it is, until it stops after DEPOT_TEST_IDLE_DURATION_SECONDS.
Each backend sizes its fakesnow pool as if it had Docker to itself.
When test runs with two different images run at the same time, their containers can need more memory than Docker has, and a container can stop with "OOMKilled": true.
To prevent this, set DEPOT_TEST_FAKESNOW_INSTANCES and DEPOT_FAKESNOW_MEMORY so that the containers of all the backends fit in the memory of Docker.
Environment variables & profile attributes correspondence table
| Environment variable | profile attribute | Notes |
|---|---|---|
SNOWFLAKE_SCHEMA_PREFIX | schemaPrefix | |
KEEP_SNOWFLAKE_SCHEMA | keepSchema | |
DEPOT_TEST_MAX_SCHEMA_OPS | ||
DEPOT_TEST_MAX_DATA_OPS | ||
DEPOT_TEST_MAX_TOTAL_OPS | ||
DEPOT_TEST_IDLE_DURATION_SECONDS | must be defined at the point of starting the backend | |
DEPOT_TEST_BACKEND_LOCKFILE | must be defined at the point of starting the backend | |
DEPOT_TEST_BACKEND_JAVA_OPTS | must be defined at the point of starting the backend | |
DEPOT_TEST_FAKESNOW_INSTANCES | must be defined at the point of starting the backend | |
DEPOT_TEST_FAKESNOW_IDLE_DURATION_SECONDS | must be defined at the point of starting the backend | |
DEPOT_FAKESNOW_MEMORY | must be defined at the point of starting the backend | |
DEPOT_TEST_LOG_LEVEL |
Environment variables take precedence over profile attributes.