What is Flowman¶
Flowman is a Spark based data build tool that simplifies the act of writing data transformation application. Flowman can be seen as a ETL tool, with a strong focus on transformation and schema management.
The main idea is that developers define all input/output tables and the whole transformation logic in purely declarative YAML files instead of writing complex Spark jobs in Scala or Python. The main advantage of this approach is that many technical details of a correct and robust implementation are encapsulated and the user can concentrate on the data transformations themselves.
In addition to writing and executing data transformations, Flowman can also be used for managing physical data models, i.e. Hive tables but also JDBC tables. Flowman will create such tables from a specification with the correct schema, and Flowman also provides mechanisms to automatically migrate these tables when the schema changes due to updated transformation logic (i.e. new columns are added, data types are changed, etc).
This helps to keep all aspects (like transformations and schema information) in a single place managed by a single application.
Flowman suits well to the requirements of a modern Big Data stack serving multiple different purposes like reporting, analytics, ML and more. Building on Sparks ability to integrate different data sources, Flowman will serve as the central place in your value chain for data preparations for the next steps.
- Declarative syntax in YAML files
- Full lifecycle management of data models (create, migrate and destroy Hive tables, JDBC tables or file based storage)
- Flexible expression language
- Jobs for managing build targets (like copying files or uploading data via sftp)
- Automatic dependency analysis to build targets in the correct order
- Powerful yet simple command line tool for batch execution
- Powerful Command line tool for interactive data flow analysis
- History server that provides an overview of past jobs and targets including lineage
- Metric system with the ability to publish these to servers like Prometheus
- Extendable via Plugins
Where to go from here¶
Quickstart & Tutorial¶
A small quickstart guide will lead you through a simple example.
In order to successfully implement your projects with Flowman, you need to get a basic understanding of Flowmans core concepts and abstractions.
After you have finished the introduction, you may want to proceed with the Flowman tutorial to get more in-depth knowledge step by step.
Flowman provides a command line utility (CLI) for running flows. Details are described in the following sections:
- Flowman Executor: Documentation of the Flowman Executor CLI
- Flowman Shell: Documentation of the Flowman Shell CLI
- Flowman Server: Documentation of the Flowman Server CLI
So-called specifications describe the logical data flow, data sources and more. A full specification contains multiple entities like mappings, data models and jobs to be executed. More detail on all these items is described in the following sections:
- Specification Overview: An introduction for writing new flows
- Mappings: Documentation of available data transformations
- Relations: Documentation of available data sources and sinks
- Targets: Documentation of available build targets
- Schema: Documentation of available schema descriptions
- Jobs: Documentation of creating jobs and building targets
Testing & Documenting¶
- Testing How to implement tests in Flowman
- Documenting How to create detailed documentation of your Flowman project
Flowman provides simple but powerful command line tools for executing your projects. This provides a great flexibility for defining your perfect workflow from development to production deployment. But this flexibility makes it more difficult to get started. Luckily Flowman also provides some guidance for setting up your projects and for defining a streamlined development workflow.
- Development & Deployment Workflow: How to implement an efficient development workflow with Flowman
Installation & Configuration¶
- Flowman Installation: Installation guide for local installation
- Running in Docker: How to run Flowman in Docker
- Configuration: Configuration settings