S3 move between buckets/folders

Who this is for

Data engineers building pipelines that create pipelines that involve heavy file movement between S3 locations, and similar workflows with Azure and GCP buckets

What will it unlock

Currently when moving a file within an S3 or similar the file will actually be downloaded, and then reuploaded to the new location at the agent. This is obviously inefficient and can significantly slow down a pipeline. This will resolve this - allow files to be moved in-situ without having to resort to Python - and when combined with the upcoming “Delete File” functionality will allow for complex file operations.

What is it

Improvements to the Data Transfer functionality to allow for files to be moved within S3 in situ to reduce time and complexity