Kafka Deployment - ZooKepper Mode

FineDataLink uses Kafka as a middleware for Data Synchronization to temporarily store the data in the source database, facilitating the writing of data to the target database and realizing real-time data synchronization.

This document introduces how to deploy Kafka in the ZooKeeper mode. For details about subsequent pipeline task configuration, see Pipeline Task Configuration.

Note:

1.This document takes deploying Kafka on a Linux system (recommended) as an example. You can also install Kafka on Windows systems, which is not recommended for production environments as Kafka on Windows systems has limited performance and can be used for demonstration purposes only.

2. For details about deploying Kafka in the KRaft mode, see Deploying Kafka in the KRaft Mode.

3. Kafka clusters are not supported by FineDataLink currently.

For details, see Deploying Kafka in the KRaft Mode                                                                   

Kafka requires Java to run. The server must have usable Java (specified by the JAVA_HOME environment variable) installed. 

Note: Your JDK version cannot be earlier than 1.8.

Note: If Kafka and FineDataLink are deployed on the same server, they can use the same Java environment. 

 

Configure the Java environment in the etc/profile file by adding the following content.

export JAVA_HOME=FineDataLink installation path/jdk/jre 
export PATH=$JAVA_HOME/bin:$PATH 
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar

Execute source /etc/profile to apply the changes.

For details about Kafka performance, see the official Kafka document Apache Kafka.

Download the Kafka installation package:

Decompressing the Installation Package

Upload the Kafka installation package.

Note: You can install Kafka in any Linux environment. If Kafka and FineDataLink are deployed on different servers, you need to apply specific configuration to enable cross-server access to Kafka. For details, see Adding Configuration.

Use the following command to decompress the installation package.

tar zxvf Name of Kafka installation package.tgz

Note: Do not decompress the package in the Windows environment and then upload the extracted file. This will modify the file and trigger an error, saying "/bin/bash^M: bad interpreter: No such file or directory."

Modifying the Data Storage Directory of Kafka

Note: You can also ignore this section and use the default data storage location.

Create a folder for Kafka to store data temporarily at runtime.

For example, you can create a kafka-logs folder in /data, as shown in the following figure.

Note: The folder location is not specified. You are advised to create the folder at a location with relatively large disk space on the server.

Open the server.properties file in the config folder in the Kafka installation directory, find log.dirs=, replace the value after the equal sign with the pre-created directory for Kafka data storage, and save the change.

Note: The following figure is an example. You can set the data storage path according to your actual conditions.

Modifying the listeners Parameter

If Kafka and FineDataLink are not deployed on the same server, you need to modify the value of the listeners parameter in the server.properties file. For details, see Adding Configuration

You can skip this section if Kafka and FineDataLink are deployed on the same server.

Note: You do not need to specify topics in the Kafka configuration file. FineDataLink will automatically create a separate topic for each table.

Modifying the Data Storage Directory of ZooKeeper

For example, you can create a zookeeper folder in /data, as shown in the following figure.

Note: You can set the folder location.

Open the zookeeper.properties file in the config folder in the Kafka installation directory, find dataDir=, replace the value after the equal sign with the pre-created directory for Kafka data storage, and save the change, as shown in the following figure.

1. Start ZooKeeper.

Since Kafka depends on the ZooKeeper service, you need to start ZooKeeper first. Enter to the Kafka installation directory using the command

cd /Kafka installation directory/kafka_2.13-2.8.1

 and use the following command to start ZooKeeper that comes with Kafka.

nohup ./bin/zookeeper-server-start.sh ./config/zookeeper.properties > ./zk.log 2>&1 &

2. Check the ZooKeeper running status.

Execute the following command to check if ZooKeeper has been started successfully. If the output shows a process with an ID similar to 18300, ZooKeeper has been started successfully.

ps -ef | grep zookeeper

3. Start Kafka.

Execute the following command to start Kafka. 

nohup ./bin/kafka-server-start.sh ./config/server.properties >./kafka.log 2>&1 & =

4. Check the Kafka running status.

Execute the following command to check if Kafka has been started successfully. If the output shows a process with an ID similar to 18300,  Kafka has been started successfully.

ps -ef|grep kafka 

Note: If you want to shut down Kafka, you need to first shut down Kafka and then ZooKeeper. For details, see the "O&M Commands" section.

Subsequent Operation

After installing and deploying Kafka, you can configure Transmission Queue in FinedataLink. For details, see Transmission Queue Configuration.

Note:

1. If you want to shut down Kafka, you need to first shut down Kafka and then ZooKeeper.

2. Before restarting Kafka, you need to manually pause the pipeline task. After restarting Kafka, you need to manually restart the pipeline task. Otherwise, exceptions will occur in the pipeline task. 

Shutting Down Kafka

Enter the Kafka installation directory and execute the following command. 

bin/kafka-server-stop.sh

Shutting Down ZooKeeper

Enter the Kafka installation directory and execute the following command. 

bin/zookeeper-server-stop.sh