Manage Data Quality

Control duplicate data during source file ingestion by configuring deduplication options for Bronze layer entities.

Deduplication Modes

Set the DQ_Deduplication_Mode parameter to define the deduplication strategy. Supported modes include:

• None
  All rows from the source file are loaded without deduplication.

• Row
  Only unique rows are retained based on the entire row content.

• Key
  Deduplication is performed using primary key column(s). Only the first occurrence of each key is kept.

Tip

Choose the deduplication mode that best matches your data quality requirements and the structure of your source data.

Deduplication Mode Examples

None

Before	name	age	height	After	name	age	height
	Alice	5	80		Alice	5	80
	Alice	5	80		Alice	5	80
	Alice	10	80		Alice	10	80

All rows are loaded, including duplicates.

Row

Before	name	age	height	After	name	age	height
	Alice	5	80		Alice	5	80
	Alice	5	80		Alice	10	80
	Alice	10	80

Only unique rows are retained.

Key (Name is the key column)

Before	name	age	height	After	name	age	height
	Alice	5	80		Alice	5	80
	Alice	5	80
	Alice	10	80

Key Deduplication

When enabling the Key option, make sure that your primary keys are correct. Otherwise, you may lose data extracted from the source system.

Only the first row for each key value is kept.