Ingest CSV / files without knowing schema in advance

What will this unlock?

The ability to create pipelines that can ingest data from files (such as CSV, JSON) without having to define a table structure in advance, unlocking

1. Simpler data ingestion without the need to download and open a file outside of Matillion Data Productivity Cloud to understand the structure and create a target table
2. The ability to ingest files where the schema can vary over time - without ingestion pipelines failing.

What is it?

A set of new component per file type that will ingest a file and create the table based on the file structure, rather than relying on a table with the correct schema existing in advance of load.

How does it work currently?

Currently in Data Productivity Cloud the process for loading a CSV file is

1. Analyse the file structure outside of the platform - perhaps using Notepad or Excel
2. Create a table using the Create Table component with the correct structure
3. Run this, then add an S3 Load / Azure Blob Load / GCS Load component and configure to load the file(s)
4. Run the pipeline

We believe this approach currently has limitation in terms of

1. Delaying data value by requiring the understanding of the file schema in advance
2. Not supporting schema drift - that is the file schema changing between executions (e.g. new columns being added to a CSV file after initial pipeline development).

If you are interested in this potential please, please contact us so we can learn more and you can help us shape the product development