Data transfer component: Delete source file after successful transfer

Detail:

Who this is for

Data Engineers building pipelines that rely only moving files from External Sources (such as sFTP, Windows Fileshare, Box etc…)

What will it unlock

The ability to write pipelines that process files from external systems, but the remove the file from the source system at a specified point:

1. Either after a successful transfer into Cloud Storage, or
2. After the file has been loaded and processed successfully

In essence, a Delete feature for external systems that the current Data Transfer component can access.

What is it

This will likely be delivered as a new component that will allow deletion from external sources that are accessible to the existing Data Transfer component (note: not all sources will be compatible, for example HTTPS does not support remote deletes).

Delivering as a separate component will provide the flexibility to create processes that delete the file immediately after transform, or later in the pipeline after the file has been successfully processed. Use of the Matillion variable system will allow for specifying filenames as variables to prevent over-complication.