MongoDB Output

Version

Application Scenario

Your business system data is stored in a MongoDB database, and you want to output data from data warehouses or other business systems to it through scheduled tasks.

Function Description

Scheduled Task supports data writing into the MongoDB database.

Usage Instruction

1. Before using this function, ensure that the NoSQL function point has been registered in FineDataLink. For details, see Registration Introduction.

2. You have created a MongoDB Data Connection.

3. You can use the to_json function to build the JSON field to be written into the MongoDB collection.

Scenario Description

Example: Synchronize data from the local table collection to a MongoDB collection that contains array-type data.

You can download the example data (source data): jihe.xlsx, as shown in the following figure.

Data in the MongoDB collection is shown in the following figure. (The info value is of the array type.)

You need to use the Spark SQL operator to concatenate fields from a 2D table into a JSON array.

Source Data Reading

1. Create a scheduled task, drag a Data Transformation node ointo the page, and enter the Data Transformation editing page.

2. Drag a DB Table Input operator onto the page and configure the source data, as shown in the following figure.

Data Processing

1. Drag a Spark SQL operator onto the page and concatenate the date and time fields in the source data into a JSON array to facilitate subsequent data synchronization to the MongoDB collection that contains array-type data, as shown in the following figure.

Note:

When writing the statement, specify the source table by clicking DB Table Input and specify the table field by entering the field name and selecting the one marked with DB Table Input from the drop-down list. For details about to_json and struct functions, see Common Syntax of SQL.

SELECT username, concat ("[," concat_ws ("," collect_list (to_json (struct (date, time)))),"]") as kaoqin
FROM DB Table Input GROUP BY username

Click Data Preview, as shown in the following figure.

Data Output

1. Drag a MongoDB Output operator onto the page to output the processed data to the MongoDB collection. Select Write Data to Target Collection Directly as the Write Method, as shown in the following figure.

Note:

To synchronize data to the field with array-type data in a MongoDB collection, field values in the source table must be enclosed with square brackets ([]).

The following table describes the setting items.

2. Click the Save button in the upper right corner.

Effect Display

After running, data in the MongoDB collection is shown in the following figure.