MaxCompute Data Connection

  • Last update: November 25, 2024
  • Overview 

    Version 

    FineDataLink VersionFunctional Change
    4.0.26/
    4.1.11.4Scheduled Task supported data writing into MaxCompute.

    Function Description 

    MaxCompute, formerly referred to as ODPS, is a fast and fully hosted data warehousing solution. It can process gigabytes, terabytes, or petabytes of data. MaxCompute provides complete data import schemes and various classic distributed computing models. It supports fast computing on a large amount of data, effectively reduces costs for enterprises, and ensures data security.

    FineDataLink supports connection to MaxCompute for data reading and writing using scheduled tasks.

    Limitation 

    • Scheduled Task supports data reading from and writing into MaxCompute, and Data Service supports the publication of MaxCompute data.

    • If the interactiveMode parameter is used and the number of data records exceeds 1 million, add autoSelectLimit=1000000000 to the URL.

     Preparation 

    Prerequisite 

    For details, see Overview of Data Connection.

    Version and Driver 

    Download the driver package and place it in the /webroot/web-inf/lib path in the FineDataLink installation directory.

    iconNote:
    Uploading the driver package through Driver Management is not supported.

    Driver NameURLDriver Package
    com.aliyun.odps.jd bc.OdpsDriverjdbc:odps:MaxCompute endpoint? project=MaxCompute project name

    odps-jdbc-3.2.26-jar-with-dependencies.jar

    iconNote:
    You must use the driver provided in this document. The driver downloaded from the official website is not supported.

    The URL is explained as follows:

    • MaxCompute endpoint: It refers to the endpoint of the region where the MaxCompute service is located For example, the public endpoint of UK (London) is https://service.eu-west-1.maxcompute.aliyun.com/api.

    • MaxCompute project name: It refers to the name of the MaxCompute project. 

    For details, see Usage Notes.

    Connection Information Collection 

    Collect the following information before connecting FineDataLink to the database.

    • Endpoint of the region where the MaxCompute service is located  

    • MaxCompute project name

    • Username and password of the database (The password cannot contain a percent sign (%), as it will cause a connection error.)

    • Schema of the database to be connected

    Procedure 

    1. Log in to FineDataLink as the admin, choose System Management > Data Connection > Data Connection Management, select a folder, and create a data connection, as shown in the following figure.

    2. Set the data connection name. You can also modify the directory of the data connection.

    3. Find the data source by searching the data source name or filtering the data source by Data Source TypeSupported Form, and Compatible Module, as shown in the following figure.

    4. Enter the connection information collected in the "Connection Information Connection" section.

    iconNote:

    1. To write data into MaxCompute, you need to remove &interactiveMode=true from the data connection URL. Queries are accelerated with this setting, but only SELECT query statements are allowed.

    2. You are advised to disable Check Before Connection and Regular Availability Check of Idle Connection to speed up the acquisition of field mapping relationships and preparation for data writing.

    Click Click to Connect Database. If the connection is successful, you can select the schema, as shown in the following figure.

    iconNote:
    You can optionally modify other advanced settings. For details, see Data Source Creation and Management.

    5. Configure Tunnel Setting if you need to write data into MaxCompute using scheduled tasks. Fill in the Tunnel endpoint of the region (in the corresponding network connection mode) where the MaxCompute service is located. For example, https://dt.eu-west-1.maxcompute.aliyun.com.

    For details about the Tunnel endpoint, see Endpoints.     

    For details about the data source usage, see MaxCompute Instruction

    6. Click Test Connection. If the connection is successful, click Save, as shown in the following figure. 

    Data Source Usage 

    After configuring the data source, you can use it for data reading and writing in Scheduled Task. For details, see Overview of Scheduled Task

    After configuring the data source, you can use it in Data Service. For details, see Overview of Data Service.

    For details about the data source usage in FineDataLink, see MaxCompute Instruction.

    附件列表


    主题: Data Source Configuration
    • Helpful
    • Not helpful
    • Only read

    滑鼠選中內容,快速回饋問題

    滑鼠選中存在疑惑的內容,即可快速回饋問題,我們將會跟進處理。

    不再提示

    8s后關閉

    Get
    Help
    Online Support
    Professional technical support is provided to quickly help you solve problems.
    Online support is available from 9:00-12:00 and 13:30-17:30 on weekdays.
    Page Feedback
    You can provide suggestions and feedback for the current web page.
    Pre-Sales Consultation
    Business Consultation
    Business: international@fanruan.com
    Support: support@fanruan.com
    Page Feedback
    *Problem Type
    Cannot be empty
    Problem Description
    0/1000
    Cannot be empty

    Submitted successfully

    Network busy