Connectors#
Flowman supports a broad range of data sources and sinks. Some are available directly in Flowman while others are contained in plugins to decrease code bloat when not required.
Overview#
The following table gives an overview of all currently supported data sources and sinks:
| Data Source | Supports Read | Supports Write | Plugin | Relation |
|---|---|---|---|---|
| AWS S3 | yes | yes | AWS | file |
| Avro files | yes | yes | Avro | file |
| Azure ABS | yes | yes | Azure | file |
| Azure SQL | yes | yes | MS SQL Server | sqlserver, jdbcQuery, jdbcView |
| CSV files | yes | yes | N/A | file |
| Delta Lake | yes | yes | Delta | deltaFile, deltaTable |
| HDFS | yes | yes | N/A | file |
| Hive | yes | yes | N/A | hiveTable, hiveView |
| Impala | yes (via Hive) | yes (via Hive) | Impala | hiveTable, hiveView |
| JSON files | yes | yes | N/A | file |
| Kafka | yes | yes | Kafka | kafka |
| Local files | yes | yes | N/A | local |
| MariaDB | yes | yes | MariaDB | jdbcTable, jdbcQuery, jdbcView |
| MySQL | yes | yes | MySQL | jdbcTable, jdbcQuery, jdbcView |
| Oracle DB | yes | yes | Oracle | jdbcTable, jdbcQuery, jdbcView |
| ORC files | yes | yes | N/A | file |
| Parquet files | yes | yes | N/A | file |
| PostgreSQL | yes | yes | PostgreSQL | jdbcTable, jdbcQuery, jdbcView |
| SQL Server | yes | yes | MS SQL Server | sqlserver, jdbcQuery, jdbcView |
| Sequence files | yes | yes | N/A | file |
| Text files | yes | yes | N/A | file |