Deploying Flowman to Azure Synapse

Since version 1.0.0, Flowman officially supports Azure Synapse as an execution environment. In order to provide a high degree of compatibility, Flowman provides a special build variant for Azure Synapse, which can be identified via the term “synapse” in their version.

For example, the Flowman version 1.0.0-synapse3.3-spark3.3-hadoop3.3 contains a build specifically crafted for Azure Synapse version 3.3, which contains Spark 3.3 and Hadoop 3.3. You should always use a Flowman version which matches your Synapse version (or the other way round) in order to avoid incompatibilities between libraries.

Executing Flowman in Azure Synapse Spark

Running Flowman in Azure Synapse does not support the traditional shell based approach of Flowman. Instead, the easiest way is to create a single jar (Java Archive) file containing all Flowman code, additional libraries and your project. Such jar files are called “fat jar”, since they tend to be very big.

1. Build a fat jar

Normally, you need to be a Maven expert to build such a fat jar. But not so with Flowman, since it provides a plugin for Maven, the so-called flowman-maven-plugin, which greatly simplifies building such fat jars. You can find a detailed description for using the plugin at Using Flowman Maven Plugin.

In this case, we need a small pom.xml file (this is the build descriptor for Maven), which looks as follows:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
         xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>my.company</groupId>
    <artifactId>my-project</artifactId>
    <version>1.0-SNAPSHOT</version>
    <packaging>pom</packaging>

    <name>quickstart</name>

    <properties>
        <encoding>UTF-8</encoding>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>

    <build>
        <plugins>
            <plugin>
                <groupId>com.dimajix.flowman.maven</groupId>
                <artifactId>flowman-maven-plugin</artifactId>
                <version>0.4.0</version>
                <extensions>true</extensions>
                <configuration>
                    <deploymentDescriptor>deployment.yml</deploymentDescriptor>
                </configuration>

                <!-- Additional plugin dependencies for specific deployment targets -->
                <dependencies>
                    <!-- Support for deploying to S3 storage -->
                    <dependency>
                        <groupId>com.dimajix.flowman.maven</groupId>
                        <artifactId>flowman-provider-azure</artifactId>
                        <version>0.1.0</version>
                    </dependency>
                </dependencies>
            </plugin>
        </plugins>
    </build>
</project>

You also need an additional Flowman specific file deployment.yml, which is referenced by the pom.xml file above. This deployment descriptor contains all required information

flowman:
  # Specify the Flowman version to use
  version: 1.2.0-synapse3.3-spark3.3-hadoop3.3
  plugins:
    # Specify the list of plugins to use
    - flowman-avro
    - flowman-aws
    - flowman-azure
    - flowman-delta
    - flowman-mssqlserver


# List of subdirectories containing Flowman projects
projects:
  - flow


# Specify possibly multiple redistributable packages to be built
packages:
  # This package is called "synapse"
  synapse:
    # The package is a "fatjar" package, i.e. a single jar file containing both Flowman and your project
    kind: fatjar


# Optional: List of deployments, which will copy packages to their destination
deployments:
  # This deployment is called "synapse"
  synapse:
    kind: copy
    package: synapse
    # This specifies the location where the fatjar will be uploaded to in the "flowman:deploy" step
    location: abfs://flowman@dimajixspark.dfs.core.windows.net/integration-tests

You can then build the package via

mvn clean install

This will create a fat jar my-project-1.0-SNAPSHOT-synapse in the subdirectory target/synapse. The general pattern for the artifact names is <artifactId>-<version>-<package>.

2. Deploy jar to Azure Blob Storage

Once you have built the fat jar, you need to upload it to Azure Blob Storage, so Azure can access it. You can either do this step manually, or you can also use the Flowman Maven plugin to help you:

mvn flowman:deploy

3. Execute in Azure Synapse

First you need to create an “Apache Spark Pool” in Azure Synapse with the correct Spark version, which would be 3.3 as the time of writing. Then you need to create an “Apache Spark job definition” in Azure Synapse Workspace with the following properties:

Setting	Description	Value
Language	The type of the Spark application, either Python or Scala	"Scala"
Main job file	Location of the fat jar in Azure Blob Storage	abfss://flowman@dimajixspark.dfs.core.windows.net/integration-tests/my-project-1.0-SNAPSHOT-synapse.jar
Main class	The Java/Scala class to execute	com.dimajix.flowman.tools.exec.Driver
Command line arguments	Command line parameters passed to Flowman	-B -f flow job build main --force
Apache Spark pool	Spark pool to use for this job. Needs to provide the correct version	name of your Spark pool

Once you submit the job, and all settings are correct, AWS should spin up a temporary Spark cluster and start Flowman.