SeaboxMPP Data Connection

  • Last update: September 19, 2024
  • Overview 

    Version 

    FineDataLink VersionFunctional Change
    4.0.28Pipeline Task supported data writing into the SeaboxMPP database.
    4.1.5.5Scheduled Task supported the reading from and writing into the SeaboxMPP database.

    Function Description 

    FineDataLink supports connection to Seabox MPP for data writing using pipeline tasks and data reading and writing using scheduled tasks.

    Preparation 

    Version and Driver 

    Download the driver package and upload it to FineDataLink. For the specific steps of uploading the driver package, see the section "Prerequisite" of Driver Management.

    Supported Database Version

    Driver Package Download

    SeaboxMPP 21.0.3.398

    postgresql-42.1.4.jar

    Connection Information Collection 

    Collect the following information before connecting FineDataLink to the database.

    • IP address and port number of the database server   

    • Username and password of the database

    Procedure 

    1. Log in to FineDataLink as the admin, choose System Management > Data Connection > Data Connection Management, and click New Data Connection, as shown in the following figure.

    iconNote:

    If you are not the admin, you can configure data connections only after the admin assigns you permission on Data Connection under Permission Management > System Management. For details, see Data Connection Management Permission


    2. Click the SeaboxMPP icon.

    3. Fill in the connection information.

    iconNote:

    You can modify the connection pool settings. For details, see Connection Pool Setting.


    4. Click Test Connection. If the connection is successful, click Save to save the configuration.

    Data Source Usage 

    Pipeline Task 

    For FineDataLink 4.0.28 and later releases, Pipeline Task supports data writing into the SeaboxMPP database. For details, see Overview of Data Pipeline.

    Scheduled Task 

    For FineDataLink 4.1.5.5 and later releases, Scheduled Task supports the reading from and writing into the SeaboxMPP database.

    When writing data into a SeaboxMPP database, you can set Load Method to COPY Loading or Common Loading. As shown in the following figure.

    Load Method

    Description

    Common Loading

    It is selected by default for JDBC-based parallel loading.

    COPY Loading

    Prerequisites:

    To use COPY Loading, ensure the target database and users meet the following requirements: 

    • Create a fdl_temp schema in the target database to store temporary tables.

    • Have permission to create tables and specified schemas. (If the schema has been created, on which the database administrator has assigned the permission, database users do not need the schema creation permission).

    Working principles:

    The loading effect varies with the write method when you select COPY Loading.

    Write Method

    Description

    Write data to the target table, which has no primary key or Primary Key Mapping configured.

    Use COPY Loading.

    Write data to the target table, which has a primary key or has Primary Key Mapping configured.Three strategies for primary key conflicts include:
    • Ignore Source Data If Same Primary Key Exists    

    • Record as Dirty Data If Same Primary Key Exists

    • Overwrite Data in Target Table If Same Primary Key Exists

    Use COPY Loading and Common Loading:

    • Each batch of data is loaded using the COPY method first.

    • If the COPY loading method fails, the batch of data will be written using the common loading method. Any data that fails to be written will be recorded as dirty data. Once the write of this batch is completed, the next batch will again prioritize the COPY method for loading.


    Write data into the target table after emptying the table.Use COPY Loading and Common Loading:
    • Each batch of data is loaded using the COPY method first.

    • If the COPY loading method fails, the batch of data will be written using the common loading method. Any data that fails to be written will be recorded as dirty data. Once the write of this batch is completed, the next batch will again prioritize the COPY method for loading.

    Add/Modify/Delete data based on the identifier field.



    附件列表


    主题: Data Source Configuration
    • Helpful
    • Not helpful
    • Only read

    滑鼠選中內容,快速回饋問題

    滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。

    不再提示

    9s后關閉

    Get
    Help
    Online Support
    Professional technical support is provided to quickly help you solve problems.
    Online support is available from 9:00-12:00 and 13:30-17:30 on weekdays.
    Page Feedback
    You can provide suggestions and feedback for the current web page.
    Pre-Sales Consultation
    Business Consultation
    Business: international@fanruan.com
    Support: support@fanruan.com
    Page Feedback
    *Problem Type
    Cannot be empty
    Problem Description
    0/1000
    Cannot be empty

    Submitted successfully

    Network busy