Connectors#

Flowman supports a broad range of data sources and sinks. Some are available directly in Flowman while others are contained in plugins to decrease code bloat when not required.

Overview#

The following table gives an overview of all currently supported data sources and sinks:

Data Source Supports Read Supports Write Plugin Relation
AWS S3 yes yes AWS file
Avro files yes yes Avro file
Azure ABS yes yes Azure file
Azure SQL yes yes MS SQL Server sqlserver, jdbcQuery, jdbcView
CSV files yes yes N/A file
Delta Lake yes yes Delta deltaFile, deltaTable
HDFS yes yes N/A file
Hive yes yes N/A hiveTable, hiveView
Impala yes (via Hive) yes (via Hive) Impala hiveTable, hiveView
JSON files yes yes N/A file
Kafka yes yes Kafka kafka
Local files yes yes N/A local
MariaDB yes yes MariaDB jdbcTable, jdbcQuery, jdbcView
MySQL yes yes MySQL jdbcTable, jdbcQuery, jdbcView
Oracle DB yes yes Oracle jdbcTable, jdbcQuery, jdbcView
ORC files yes yes N/A file
Parquet files yes yes N/A file
PostgreSQL yes yes PostgreSQL jdbcTable, jdbcQuery, jdbcView
SQL Server yes yes MS SQL Server sqlserver, jdbcQuery, jdbcView
Sequence files yes yes N/A file
Text files yes yes N/A file
Trino yes yes Trino jdbcTable, jdbcQuery, jdbcView