FineDataLink Version
Functional Change
4.1.6.4
Scheduled Task supported data writing into the MongoDB database.
Your business system data is stored in a MongoDB database, and you want to output data from data warehouses or other business systems to it through scheduled tasks.
Scheduled Task supports data writing into the MongoDB database.
1. Before using this function, ensure that the NoSQL function point has been registered in FineDataLink. For details, see Registration Introduction.
2. You have created a MongoDB Data Connection.
3. You can use the to_json function to build the JSON field to be written into the MongoDB collection.
Example: Synchronize data from the local table collection to a MongoDB collection that contains array-type data.
You can download the example data (source data): jihe.xlsx, as shown in the following figure.
Data in the MongoDB collection is shown in the following figure. (The info value is of the array type.)
You need to use the Spark SQL operator to concatenate fields from a 2D table into a JSON array.
1. Create a scheduled task, drag a Data Transformation node ointo the page, and enter the Data Transformation editing page.
2. Drag a DB Table Input operator onto the page and configure the source data, as shown in the following figure.
1. Drag a Spark SQL operator onto the page and concatenate the date and time fields in the source data into a JSON array to facilitate subsequent data synchronization to the MongoDB collection that contains array-type data, as shown in the following figure.
SELECT username, concat ("[," concat_ws ("," collect_list (to_json (struct (date, time)))),"]") as kaoqinFROM DB Table Input GROUP BY username
Click Data Preview, as shown in the following figure.
1. Drag a MongoDB Output operator onto the page to output the processed data to the MongoDB collection. Select Write Data to Target Collection Directly as the Write Method, as shown in the following figure.
To synchronize data to the field with array-type data in a MongoDB collection, field values in the source table must be enclosed with square brackets ([]).
The following table describes the setting items.
Setting Item
Description
Data Connection
The drop-down list displays all existing MongoDB data connections on which you have permission.
Database
The drop-down list displays all the databases connected via the selected data connection.
Target Collection
Existing Collection:
The drop-down list displays all the collections in the selected database.
Auto Created Collection:
1. You can configure the name and field type of the target collection.
2. Collection names in versions prior to 4.1.6.1 can contain English letters, numbers, and underscores. Chinese characters are additionally supported in FineDataLink 4.1.6.1 and later versions.
3. You can enable Not Null for fields.
4. The primary key field of the MongoDB collection defaults to _id and cannot be configured.
5. The Manual Table Creation function is not supported.
Field Mapping
1. The primary key field (_id) is not displayed.
2. You can configure the mapping of the _id field of an existing collection, as shown in the following figure.
Write Method
1. You can select Write Data into Target Collection After Emptying It as Write Method.
2. You can select Write Data into Target Collection Directly as Write Method.
Strategy for Primary Key Conflict is configurable.
Primary Key Mapping is configurable. If you have configured the mapping for the MongoDB _id field, the primary key mapping defaults to the mapping of _id. In other cases, you can select other fields as logical primary keys.
3. You can select Append/Update/Delete Data Based on Identifier Field as Write Method.
Logical Deletion: Logical deletion is performed only on data rows with null identifier values.
Identification fields and values are configurable.
2. Click the Save button in the upper right corner.
After running, data in the MongoDB collection is shown in the following figure.
滑鼠選中內容,快速回饋問題
滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy