MongoDB Output

  • Last update: December 25, 2024
  • Overview

    Version

    FineDataLink Version

    Functional Change

    4.1.6.4

    Scheduled Task supported data writing into the MongoDB database.

    Application Scenario

    Your business system data is stored in a MongoDB database, and you want to output data from data warehouses or other business systems to it through scheduled tasks.

    Function Description

    Scheduled Task supports data writing into the MongoDB database.

    Usage Instruction

    1. Before using this function, ensure that the NoSQL function point has been registered in FineDataLink. For details, see Registration Introduction.

    2. You have created a MongoDB Data Connection.

    3. You can use the to_json function to build the JSON field to be written into the MongoDB collection.

    Procedure

    Scenario Description

    Example: Synchronize data from the local table collection to a MongoDB collection that contains array-type data.

    You can download the example data (source data): jihe.xlsx, as shown in the following figure.

    1.png

    Data in the MongoDB collection is shown in the following figure. (The info value is of the array type.)

    2.png

    You need to use the Spark SQL operator to concatenate fields from a 2D table into a JSON array.

    Source Data Reading

    1. Create a scheduled task, drag a Data Transformation node ointo the page, and enter the Data Transformation editing page.

    2. Drag a DB Table Input operator onto the page and configure the source data, as shown in the following figure.

    3.png

    Data Processing

    1. Drag a Spark SQL operator onto the page and concatenate the date and time fields in the source data into a JSON array to facilitate subsequent data synchronization to the MongoDB collection that contains array-type data, as shown in the following figure.

    iconNote:
    When writing the statement, specify the source table by clicking DB Table Input and specify the table field by entering the field name and selecting the one marked with DB Table Input from the drop-down list. For details about to_json and struct functions, see Common Syntax of SQL.
    SELECT username, concat ("[," concat_ws ("," collect_list (to_json (struct (date, time)))),"]") as kaoqin
    FROM DB Table Input GROUP BY username

    4.png

    Click Data Preview, as shown in the following figure.

    bu(1).png

    Data Output

    1. Drag a MongoDB Output operator onto the page to output the processed data to the MongoDB collection. Select Write Data to Target Collection Directly as the Write Method, as shown in the following figure.

    iconNote:

    To synchronize data to the field with array-type data in a MongoDB collection, field values in the source table must be enclosed with square brackets ([]).

    5.png

    The following table describes the setting items.

    Setting Item

    Description

    Data Connection

    The drop-down list displays all existing MongoDB data connections on which you have permission.

    Database

    The drop-down list displays all the   databases connected via the selected data connection.

    Target Collection

    Existing Collection:

    The drop-down list displays all the collections in the selected database.

    Auto Created Collection:

    1. You can configure the name and field type of the target collection.

    2. Collection names in versions prior to 4.1.6.1 can contain English letters, numbers, and underscores. Chinese characters are additionally supported in FineDataLink 4.1.6.1 and later versions.

    3. You can enable Not Null for fields.

    4. The primary key field of the MongoDB collection defaults to _id and cannot be configured.

    5. The Manual Table Creation function is not supported.

    Field Mapping

    1. The primary key field (_id) is not displayed.

    2. You can configure the mapping of the _id field of an existing collection,   as shown in the following figure.

    6.png

    Write Method

    1. You can select Write Data into Target Collection After Emptying It as Write Method.

    2. You can select Write Data into Target Collection Directly as Write Method.

    • Strategy for Primary Key Conflict is configurable.


    • Primary Key Mapping is configurable. If you have configured the mapping for the MongoDB _id field, the primary key mapping defaults to the mapping of _id. In other cases, you can select other fields as logical primary keys.

    3. You can select Append/Update/Delete Data Based on Identifier Field as Write Method.

    8.png

    • Logical Deletion: Logical deletion is performed only on data rows with null identifier values.

    • Identification fields and values are configurable.

    • Primary Key Mapping is configurable. If you have configured the mapping for the  MongoDB _id field, the  primary key mapping defaults to the mapping of _id. In other cases, you can select other fields as logical primary keys.

    2. Click the Save button in the upper right corner.

    Effect Display

    After running, data in the MongoDB collection is shown in the following figure.

    9.png

    附件列表


    主题: Data Development - Scheduled Task
    Previous
    Next
    • Helpful
    • Not helpful
    • Only read

    滑鼠選中內容,快速回饋問題

    滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。

    不再提示

    10s後關閉

    Get
    Help
    Online Support
    Professional technical support is provided to quickly help you solve problems.
    Online support is available from 9:00-12:00 and 13:30-17:30 on weekdays.
    Page Feedback
    You can provide suggestions and feedback for the current web page.
    Pre-Sales Consultation
    Business Consultation
    Business: international@fanruan.com
    Support: support@fanruan.com
    Page Feedback
    *Problem Type
    Cannot be empty
    Problem Description
    0/1000
    Cannot be empty

    Submitted successfully

    Network busy