Data Deduplication

Problem

You want the duplicated data to appear only once during data processing, as shown in the following figures.

          Initial Data                                             Deduplicated Data

Solution

1. Deduplicate data with SQL statements.

2. Deduplicate data through Data Setting in the designer. For details, see Data Display Setting.

Note:

Considering the performance impact, you are advised to handle problems with SQL statements if possible.

Data Deduplication with the DISTINCT Keyword in SQL

Data Deduplication with the GROUP BY Clause in SQL

Data Deduplication with Analytic Functions in SQL

Some databases do not support analytic functions, and neither do older versions of certain databases.

Data Deduplication with the Group Attribute

You can deduplicate data through Data Setting in the designer. For details, see Data Display Setting.

Data Deduplication with the Filter Condition

You can add the filter condition SEQ($$$)==1 for the cell, as shown in the following figure.

For details about how to use the SEQ function, see Sequence Number.