Securely providing Secrets
Most Flowman projects need to provide some sort of secrets (database credentials, S3 secrets, …) to external systems for being able to access them. This opens the question about a best practice how to provide those secrets to a Flowman project in a secure manner.
In order to securely provide secrets to Flowman, there are two ways:
Provide secrets via system environment variables, which then can be accessed in Flowman
Provide secrets stored in an external file, which then can be read by Flowman
Both approaches will extract the secret and store it in a Flowman environment variable. To prevent Flowman from printing the secret onto the console, it is enough when the variable name ends with “credential”, “secret” or “password”. In this case, Flowman will redact the value when logging the environment onto the console.
Using environment variables
By using the $System.getenv
template function, you can access environment variables within Flowman. For example
for extracting database credentials from system environment variables, you could proceed as follows:
environment:
- frontend_db_driver=$System.getenv('MYSQL_DRIVER', 'com.mysql.cj.jdbc.Driver')
- frontend_db_url=$System.getenv('MYSQL_URL')
- frontend_db_username=$System.getenv('MYSQL_USERNAME')
- frontend_db_password=$System.getenv('MYSQL_PASSWORD')
connections:
frontend:
kind: jdbc
driver: "$frontend_db_driver"
url: "$frontend_db_url"
username: "$frontend_db_username"
password: "$frontend_db_password"
relations:
advertiser_setting:
kind: jdbcTable
connection: frontend
table: "advertiser_setting"
schema:
kind: inline
fields:
- name: id
type: Integer
- name: business_rule_id
type: Integer
- name: rtb_advertiser_id
type: Integer
Then you only need to populate the system environment variables MYSQL_DRIVER
, MYSQL_URL
, MYSQL_USERNAME
and
MYSQL_PASSWORD
before starting Flowman. This approach fits well to deployments where Flowman runs inside Kubernetes,
but can also be used in many other scenarios.
Using external files
Another viable approach is to provide secrets via files. Flowman can read any file via the template function
$File.read
, which will simply return the files content. Moreover by using the template function $JSON.path
you
can parse any JSON content. An example might look as follows:
environment:
- sql_host=jdbc:sqlserver://$JSON.path($File.read($System.getenv('MSSQLSERVER_SECRETS')), '$.secret[?(@.key=="sql-host-name")].value')
- sql_username=$JSON.path($File.read($System.getenv('MSSQLSERVER_SECRETS')), '$.secret[?(@.key=="sql-db-username")].value')
- sql_password=$JSON.path($File.read($System.getenv('MSSQLSERVER_SECRETS')), '$.secret[?(@.key=="sql-db-password")].value')
- sql_database=$JSON.path($File.read($System.getenv('MSSQLSERVER_SECRETS')), '$.secret[?(@.key=="sql-db-name")].value')
connections:
datahub:
kind: jdbc
driver: "com.microsoft.sqlserver.jdbc.SQLServerDriver"
url: jdbc:sqlserver://$sql_host
username: $sql_username
password: $sql_password
properties:
databaseName: $sql_database
Then you need to store the secrets in a JSON file stored on the local file system where Flowman is executed. Using the JSON paths as defined above, the file needs to look as follows:
{
"secret": [
{
"key": "sql-db-username",
"value": "my-sql-username"
},
{
"key": "sql-db-password",
"value": "my-secret-password"
},
{
"key": "sql-db-name",
"value": "MY_DATABASE"
},
{
"key": "sql-host-name",
"value": "sql-database.windows.net:1433"
}
]
}
Finally, you need to set the system environment variable MSSQLSERVER_SECRETS
to point to the corresponding file before
starting Flowman. This additional indirection is not strictly required, but helps to decouple Flowman configuration
from deployment logic.