Connectors¶
Flowman supports a broad range of data sources and sinks. Some are available directly in Flowman while others are contained in plugins to decrease code bloat when not required.
Overview¶
The following table gives an overview of all currently supported data sources and sinks:
Data Source | Supports Read | Supports Write | Plugin | Relation |
---|---|---|---|---|
AWS S3 | yes | yes | AWS | file |
Avro files | yes | yes | N/A | file |
Azure ABS | yes | yes | Azure | file |
Azure SQL | yes | yes | MS SQL Server | sqlserver , jdbcQuery , jdbcView |
CSV files | yes | yes | N/A | file |
Delta Lake | yes | yes | Delta | deltaFile , deltaTable |
HDFS | yes | yes | N/A | file |
Hive | yes | yes | N/A | hiveTable , hiveView |
Impala | yes (via Hive) | yes (via Hive) | Impala | hiveTable , hiveView |
JSON files | yes | yes | N/A | file |
Kafka | yes | yes | Kafka | kafka |
Local files | yes | yes | N/A | local |
MariaDB | yes | yes | MariaDB | jdbcTable , jdbcQuery , jdbcView |
MySQL | yes | yes | MySQL | jdbcTable , jdbcQuery , jdbcView |
Oracle DB | yes | yes | Oracle | jdbcTable , jdbcQuery , jdbcView |
ORC files | yes | yes | N/A | file |
Parquet files | yes | yes | N/A | file |
PostgreSQL | yes | yes | PostgreSQL | jdbcTable , jdbcQuery , jdbcView |
SQL Server | yes | yes | MS SQL Server | sqlserver , jdbcQuery , jdbcView |
Sequence files | yes | yes | N/A | file |
Text files | yes | yes | N/A | file |