Matillion Roadmap and Ideas

Welcome to the Matillion Roadmap

Last updated: 05/05/2025
Check out our recently delivered features here

The Roadmap has evolved - see here

Kafka Connector

We’re excited to announce the release of our Kafka Connector!

This connector enables users to batch load messages from Kafka topics and supports Basic (Username/Password) and OAuth 2.0 authentication methods with TLS encryption.The connector offers three modes of operation:

  1. Single Connector - Full Load:
    • Reads data from the beginning of the topic on each run.
  2. Multi-Connector - Full Load (Parallel Execution):
    • Users can run multiple connectors in parallel, enabling faster data loading. By running 5 connectors in parallel, we successfully loaded 69.3 million rows in just 38 minutes, compared to around 3.5 hours with a single connector!
  3. Kafka-Managed Incremental Load:
    • Users can configure the connector to use Kafka’s internal offset management, allowing it to resume from the last read position on each run.

The connector supports two data formats:

  1. Schemaless JSON (Unstructured): Loads data into a single VARIANT column in the DWH table.
  2. JSON Schema using Schema Registry (Structured): Loads data in a structured format, unpacking/flattening it into the DWH table based on the message structure defined in the Schema Registry.