Connectors#
Flowman supports a broad range of data sources and sinks. Some are available directly in Flowman while others are contained in plugins to decrease code bloat when not required.
Overview#
The following table gives an overview of all currently supported data sources and sinks:
Data Source | Supports Read | Supports Write | Plugin | Relation |
---|---|---|---|---|
AWS S3 | yes | yes | AWS | file |
Avro files | yes | yes | Avro | file |
Azure ABS | yes | yes | Azure | file |
Azure SQL | yes | yes | MS SQL Server | sqlserver , jdbcQuery , jdbcView |
CSV files | yes | yes | N/A | file |
Delta Lake | yes | yes | Delta | deltaFile , deltaTable |
HDFS | yes | yes | N/A | file |
Hive | yes | yes | N/A | hiveTable , hiveView |
Impala | yes (via Hive) | yes (via Hive) | Impala | hiveTable , hiveView |
JSON files | yes | yes | N/A | file |
Kafka | yes | yes | Kafka | kafka |
Local files | yes | yes | N/A | local |
MariaDB | yes | yes | MariaDB | jdbcTable , jdbcQuery , jdbcView |
MySQL | yes | yes | MySQL | jdbcTable , jdbcQuery , jdbcView |
Oracle DB | yes | yes | Oracle | jdbcTable , jdbcQuery , jdbcView |
ORC files | yes | yes | N/A | file |
Parquet files | yes | yes | N/A | file |
PostgreSQL | yes | yes | PostgreSQL | jdbcTable , jdbcQuery , jdbcView |
SQL Server | yes | yes | MS SQL Server | sqlserver , jdbcQuery , jdbcView |
Sequence files | yes | yes | N/A | file |
Text files | yes | yes | N/A | file |