Elasticsearch Output

  • Last update: December 18, 2024
  • Overview

    Version

    FineDataLink Version
    Functional Change
    4.1.11.3

    /

    Application Scenario

    You want to write the data to be queried (which is fetched from the relational database) to Elasticsearch, which can be used as an acceleration layer to facilitate subsequent queries or as an ADS layer of the data warehouse.

    Function Description

    The Elasticsearch Output operator allows you to output the processed data to the specified Elasticsearch database.

    icon

    Note:


    If complex data processing is unnecessary, you can use the Data Synchronization node to output the data to Elasticsearch.

    Prerequisite

    To output data to Elasticsearch, you must have configured the data connection to Elasticsearch, and have permission to use the data connection. For details, see Elasticsearch Data Connection.

    You have registered the NoSQL function point.

    Function Description

    The Data Destination and Mapping tab page is shown in the following figure.

    2.png

    Setting Item
    Description
    Data Connection

    Select a configured Elasticsearch data connection.

    Target Index

    • Auto Created Index

    • Existing Index

    Definition: An index is the basic data storage unit in Elasticsearch, similar to a database in a relational database. Each index contains many documents with the same defined data structure (mapping).

    Usage: Indexes are used to store specific types of datasets. As each index supports only one data type starting from V 7.0, creating one index for each data type is a common practice.

    Field Mapping

    Choose between Map Fields with Same Name and Map Fields in Same Row to generate the default mapping.

    • The _id field is not configurable in the mapping and is masked on the front end.

    • You cannot configure remarks.

    • You cannot configure the primary key.

    • You cannot enable Not Null for fields.

    • Manual Table Creation and Update Target Table are not supported.

    • The current version defaults to static mapping.

    • The logical primary key cannot be set to a field with binary-type values or null values.

    The Write Method tab page is shown in the following figure.

    3.png

    Setting Item

    Description

    Write   Data into Target Index Directly

    Primary Key Mapping:

    You can select the field as the logical primary key and configure the mapping to ensure data uniqueness.

    Strategy for Primary Key Conflict:

    • Overwrite Data in Target Index If Same Primary Key Value Exists

    • Ignore Source Data If Same Primary Key Value Exists

    • Record as Dirty Data If Same Primary Key Value Exists

    Write Data into Target Index After Emptying it

    It works the same way as the one in the Data Synchronization node.

    Add/Modify/Delete Data Based on Identifier Field

    The configuration of Primary Key Mapping is consistent with that when Write Data into Target Index Directly is used.

    Procedure

    You can fetch the data to be queried from the relational database and write it into Elasticsearch, which can be used as an acceleration layer to facilitate queries.

    Task Configuration

    Create a scheduled task, drag a Data Transformation node onto the page, click the Data Transformation node to enter the editing page, and drag a DB Input operator onto the page, as shown in the following figure.

    4.png

    Process data, as shown in the following figure.

    5.png

    Add an Elasticsearch Output operator. Run the task to write data into Elasticsearch to facilitate subsequent queries, as shown in the following figure.

    6.png

    Effect Display

    You can use the RESTful API of Elasticsearch to query the data written to the database through HTTP requests, as shown in the following figure.

    7.png

    附件列表


    主题: Data Development - Scheduled Task
    • Helpful
    • Not helpful
    • Only read

    滑鼠選中內容,快速回饋問題

    滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。

    不再提示

    10s後關閉

    Get
    Help
    Online Support
    Professional technical support is provided to quickly help you solve problems.
    Online support is available from 9:00-12:00 and 13:30-17:30 on weekdays.
    Page Feedback
    You can provide suggestions and feedback for the current web page.
    Pre-Sales Consultation
    Business Consultation
    Business: international@fanruan.com
    Support: support@fanruan.com
    Page Feedback
    *Problem Type
    Cannot be empty
    Problem Description
    0/1000
    Cannot be empty

    Submitted successfully

    Network busy