Contents Menu Expand Light mode Dark mode Auto light/dark mode
Flowman documentation
Logo

Quick Start Guide

  • Flowman Quick Start Guide

Concepts

  • Core Concepts
  • Main Entities
  • Lifecycles and Phases
  • Execution and Data Metrics
  • Schema Migrations

Tutorial

  • Flowman Tutorial
  • Lesson 1 — Flowman Basics
  • Lesson 2 — Schema Definitions
  • Lesson 3 — Transformations
  • Lesson 4 — Parameters
  • Lesson 5 — Multiple Build Targets
  • Lesson 6 — Flowshell
  • Lesson 7 — Execution Phases and Lifecycles
  • Lesson 12 — Deployment

Command Line Tools

  • Flowman CLI Tools
  • Flowman Executor (flowexec)
    • Project Commands
    • Job Commands
    • Target Commands
    • Miscellaneous Commands
  • Flowman Interactive Shell (flowshell)
  • Flowman Kernel Server (flowman-kernel)
  • Flowman Interactive Remote Shell (flowrshell)
  • Flowman History Server (flowman-server)
  • Flowman Schema Generator

Workflow

  • Development & Deployment Workflow
  • Using Standard Maven Plugins
  • Using Flowman Maven Plugin

Reference Documentation

  • Reference Documentation
  • Projects
  • Modules
  • Mappings
    • Aggregate Mapping
    • Alias Mapping
    • Assemble Mapping
    • Case Mapping
    • Cast Mapping
    • Coalesce Mapping
    • Conform Mapping
    • Deduplicate Mapping
    • Distinct Mapping
    • Drop Mapping
    • Earliest Mapping
    • Empty Mapping
    • Explode Mapping
    • Extend Mapping
    • Filter Mapping
    • Flatten Mapping
    • Grouped Aggregate Mapping
    • Historize Mapping
    • Iterative SQL Mapping
    • Join Mapping
    • JSON Extract Mapping
    • JSON Unpack Mapping
    • Latest Mapping
    • Mock Mapping
    • Observe Mapping
    • Project Mapping
    • Read Hive Mapping
    • Rebalance Mapping
    • Recursive SQL Mapping
    • Relation Mapping
    • Repartition Mapping
    • Schema Mapping
    • Select Mapping
    • Sort Mapping
    • SQL Mapping
    • Stack Mapping
    • Stream Mapping
    • Template Mapping
    • Union Mapping
    • Unit Mapping
    • Upsert Mapping
    • Values Mapping
  • Relations
    • Delta File Relations
    • Delta Table Relations
    • Empty Relation
    • File Relations
    • Generic Relation
    • Hive Query Relations
    • Hive Table Relations
    • Hive Union Table
    • Hive View Relations
    • JDBC Query Relations
    • JDBC Table Relations
    • JDBC View Relations
    • Kafka Relations
    • Local Relations
    • Mock Relation
    • SQL Server Relations
    • Template Relation
    • Values Relation
  • Targets
    • Blackhole Target
    • Compare Target
    • Console Target
    • Copy Target
    • CopyFile Target
    • Count Target
    • Delete File Target
    • Delta Vacuum Target
    • Document Target
    • Drop Relation Target
    • Empty Target
    • File Target
    • GetFile Target
    • Hive Database Target
    • JDBC Command Target
    • Local Target
    • Measure Target
    • Merge Target
    • Merge Files Target
    • Put File Target
    • Relation Target
    • SFTP Upload Target
    • Stream Target
    • Template Target
    • Truncate Target
    • Validate Target
    • Verify Target
  • Jobs
    • Flowman Job Execution
  • Datasets
    • File Dataset
    • Mapping Dataset
    • Relation Dataset
    • Values Dataset
  • Schemas
    • Avro Schema
    • Inline Schema
    • JSON Schema
    • Mapping Schema
    • OpenAPI Schema
    • Spark Schema
    • Swagger Schema
  • Connections
    • JDBC Connection
    • SSH Connection
  • Tests
    • Flowman Test Execution
  • Assertions
    • Columns Assertion
    • Expression Assertion
    • Schema Assertion
    • SQL Assertion
    • Unique Key Assertion
  • Measures
    • SQL Measure
  • Templates
    • Assertion Template
    • Connection Template
    • Dataset Template
    • Mapping Template
    • Measure Template
    • Relation Template
    • Schema Template
    • Target Template
  • Execution Metrics
    • Console Metric Sink
    • JDBC Metric Sink
    • Prometheus Metric Sink
  • Execution Hooks
    • JDBC Hook
    • Report Hook
    • REST Hook
    • Web Hook
  • Namespace
  • Fields, Data Types & Values
  • Script Expressions
  • Profiles

Testing

  • Testing with Flowman

Documenting

  • Documenting your Project
  • Checking Model Properties
  • Configuring the Documentation
  • Documenting Mappings
  • Documenting Relations
  • Documenting Targets

Setup & Configuration

  • Setup & Configuration
  • Installation Guide
  • Running Flowman in Docker
  • Running Flowman on Windows
  • Deploying Flowman to AWS EMR
  • Deploying Flowman to Azure Synapse
  • Configuration Properties
  • Building Flowman from Sources

Cookbooks

  • Advanced JDBC Database Features
  • Data Quality Checks
  • Data Validation before Build
  • Execution Metrics
  • Execution Phases
  • Hadoop Dependencies Installation
  • Impala Metadata
  • Kerberos Authentication
  • Override jar versions
  • Parallel Execution
  • Performance Tuning
  • Securely providing Secrets
  • Sharing Entities between Projects
  • Syntax Highlighting
  • Target Execution Ordering

Connectors & Plugins

  • Connectors
  • Plugins
    • Avro Plugin
    • AWS Plugin
    • Azure Plugin
    • Delta Lake Plugin
    • Impala Plugin
    • JSON Plugin
    • Kafka Plugin
    • MariaDB Plugin
    • MS SQL Server Plugin
    • MySQL Plugin
    • OpenAPI Plugin
    • Oracle Plugin
    • PostgreSQL Plugin
    • SFTP Plugin
    • Swagger Plugin
    • Trino Plugin

Misc

  • FAQ
  • Releases
Back to top

Schemas#

Flowman uses schema definitions at various places, most commonly where relations (data sources and sinks) are defined. Flowman does not only support inline schema definitions, but also supports various external schema definitions like Avro, Swagger and JSON Schema.

Schema Types#

  • Avro Schema
  • Inline Schema
  • JSON Schema
  • Mapping Schema
  • OpenAPI Schema
  • Spark Schema
  • Swagger Schema
Next
Avro Schema
Previous
Values Dataset
Copyright © 2023, The Flowman Authors
Made with Sphinx and @pradyunsg's Furo
On this page
  • Schemas
    • Schema Types