Delta Vacuum Target#
deltaVacuum target is used to truncate the history stored in Delta Lake.
relations: crm_customer: kind: deltaTable database: crm table: customer targets: vacuum_crm_customer: kind: deltaVacuum relation: crm_customer compaction: true minFiles: 10 maxFiles: 20
Or alternatively with an embedded relation:
targets: vacuum_crm_customer: kind: deltaVacuum retentionTime: P10D relation: kind: deltaTable database: crm table: customer
kind(mandatory) (type: string):
description(optional) (type: string): Optional descriptive text of the build target
relation(mandatory) (type: string or relation): Either the name of a
deltaFilerelation or alternatively an embedded delta relation
retentionTime(optional) (type: duration): Explicitly specify the retention period, i.e. how much history should be kept. If this field is not specified, the property
deletedFileRetentionDurationspecified in the delta relation will be used instead. Note that this time is actually measured in hours, i.e. it doesn’t make any sense to specify a time period with a finer granularity.
compaction(optional) (type: boolean) (default: false): Perform compaction before vacuum operation. A compaction will reduce the number of files per table and/or partition
minFiles(optional) (type: integer) (default: 16): Number of files per partition, which will be created during a compaction operation.
maxFiles(optional) (type: integer) (default: 64): Number of files per partition, below which no compaction will be performed.
Supported Execution Phases#
BUILD- This will execute the vacuum operation
Read more about execution phases.