¶ Description
Remove dupicates rows from input table.
¶ Parameters
Parameters:
- Primary keys
- Number of lines for each distinct primary key
- Column name for duplicates counter
- Check if the input table is sorted on the key(s) column(s)
- Duplicate rows on the secondary output and unique rows on primary output
- Always allow multithread execution (change only if you know what you are doing)
¶ About
Two rows are considered duplicates when all their primary key(s) match. The input table for this action must be sorted by its primary key(s).
Only the primary key(s) are used to identify duplicates. This means that two rows with:
Different values in non-primary columns
The same primary key(s)
...will still be de-duplicated, keeping only one of them.
NOTE :
The input table of this action must be sorted on its primary key(s).
The NaïveDeDuplicate Action is also very often used to investigate data quality issues initially detected with the MultiJoin action or the FilterOnKey action.
These 2 actions requires no duplicates values inside the "key" column of the SLAVE tables. When a key-duplication is detected, you can see the duplicates in the following way:
