SFTP Upload Target¶
The SFTP upload target is used for uploading data which resides either on the local machine or in a Hadoop compatible filesystem (HDFS, S3, …) to an external SFTP server.
Plugin¶
The sftpUpload
target is provided by the SFTP Plugin, which needs to be enabled in your
namespace.yml
file. See namespace documentation for more information for configuring plugins.
Example¶
connections:
my-sftp-connection:
kind: ssh
host: "sftp.server.dimajix.net"
port: "22"
username: "testuser"
password: "12345678"
keyFile: "/home/user/private_key"
jobs:
main:
upload_data:
kind: sftpUpload
description: "Upload some data to a SFTP server"
connection: my-sftp-connection
source: "$hdfs_export_basedir/export_table/${processing_date}"
target: "${sftp_target}/inbox/_${processing_date}.txt"
merge: true
overwrite: true
You can also directly embed the connection into the target as follows:
jobs:
main:
upload_data:
kind: sftpUpload
description: "Upload some data to a SFTP server"
connection:
kind: ssh
host: "sftp.server.dimajix.net"
port: "22"
username: "testuser"
password: "12345678"
keyFile: "/home/user/private_key"
source: "$hdfs_export_basedir/export_table/${processing_date}"
target: "${sftp_target}/inbox/_${processing_date}.txt"
merge: true
overwrite: true
Fields¶
kind
(mandatory) (type: string):sftpUpload
description
(optional) (type: string): A textual description of the build target.source
(mandatory) (type: string): Specifies the source location in the Hadoop compatible filesystem. This may be either a single file or a whole directory.target
(mandatory) (type: string): Specifies the target location on the SFTP server. This may be either a single file or a whole directory.connection
(mandatory) (type: string): Specifies the name of the connection to use when connecting to the SFTP server.merge
(optional) (type: boolean) (default: false): Flowman can merge multiple files in thesource
directory into a single output file. This is useful when working with Hadoop or Spark outputs which often consist of multiple files. Per default this is set tofalse
, meaning that multiple files will be transferred individually.delimiter
(optional) (type: string) (default: empty): Specifies an optional delimiter, which should be used between files when they are concatenated whenmerge
is set totrue
. Has no effect otherwise.overwrite
(optional) (type: boolean) (default: true): Set totrue
in order to overwrite existing files on the SFTP server. Otherwise an existing file will result in an error.
Description¶
Supported Execution Phases¶
BUILD
- This will upload the specified file via SFTP
Read more about execution phases.