Missing Value Handling

A data warehouse receives massive amounts of data from diverse sources, where discrepancies in formats, statistical calibers, and value representation methods may exist.

Therefore, data with missing values is inevitable. This document introduces how to handle such data.

You can click to download the example data: Internet.xls.

If feasible, simply deleting a small amount of sample data is the most effective way to handle data with missing values.

For example, the Event_Date field in the Internet table has null values. If invalid, data with missing values can be directly deleted.

Add a Data Transformation node, and drag in a DB Table Input operator to extract data from the Internet table, as shown in the following figure.

Use a Data Filtering operator to remove data records whose Event_Date are null, as shown in the following figure.

The effect of the Data Filtering operator is equivalent to that of the SQL statement: Event_Date is not null.

When excessive missing values make deletion impractical, imputation with specified data becomes necessary.

For example, in the following table, the Session_Duration field has many null values, and you want to replace these null values with 0.

Use a New Calculation Column operator to add a new column of data. Enter the formula IF(ISNULL(#{Session_Duration}),0.0,#{Session_Duration}). This means that if Session_Duration is empty, the function returns 0.0; otherwise, it returns the specific value of Session_Duration, as shown in the following figure.

Note:

The last two parameters of the IF function must be of the same type. For example, if the values of Session_Duration in the database are of double type, 0 replacing the null values also needs to be written in the 0.0 format.

Click Data Preview. The null values have been replaced with 0, as shown in the following figure.