Kafka Deployment - KRaft Mode

FineDataLink uses Kafka as a middleware to buffer data extracted from source databases, facilitating writing to target databases, thus enabling real-time data synchronization.

This document introduces how to deploy Kafka in the KRaft mode to facilitate the subsequent configuration of pipeline tasks and real-time tasks.

Note:

1. This document takes deploying Kafka on a Linux system (recommended) as an example. You can also install Kafka on Windows systems, which is not recommended for production environments, as Kafka on Windows systems has limited performance and can be used for demonstration purposes only.

2. Kafka clusters are not supported by FineDataLink currently.

Kafka of versions before V2.8 heavily relied on ZooKeeper clusters to provide consensus services such as metadata management and controller election. This tight reliance means that any performance fluctuation in ZooKeeper clusters will greatly affect the performance of Kafka as well, as shown in the following figure.

Note:

1. For details about deploying Kafka in the ZooKeeper mode, see Kafka Deployment - ZooKeeper Mode.

2. You can search online for more details about deploying Kafka in the KRaft and ZooKeeper modes.

Following Kafka 2.8, the Raft protocol-based KRaft mode was introduced, eliminating the dependency on ZooKeeper. After the October 2022 release of Kafka 3.3.1, this new metadata management solution named KRaft was designated as production-ready.

In this mode, some Kafka brokers are designated as controllers, while others remain as brokers. These controllers assume the role previously served by ZooKeeper in providing consensus services, with all metadata now stored in Kafka topics and managed internally.

The main advantages of KRaft over ZooKeeper are as follows:

• Simplified O&M: Only Kafka needs to be deployed, with no dependency on ZooKeeper.

• Improved horizontal scalability: The number of partitions a Kafka cluster can support is a key indicator of its horizontal scalability. Previously, this value was limited to one hundred thousand due to the constraints of metadata transmission between ZooKeeper and the controllers. In contrast, the KRaft mode eliminates the need for such transmission, enabling this value to scale up to one million.

• Efficient metadata transmission: Metadata is managed via Kafka topics and transmitted via the native producer-consumer mechanism, improving integration and enhancing underlying performance.

Kafka requires a Java environment to run. Ensure a usable Java environment has been configured on the server (specified by the JAVA_HOME environment variable). 

Note:

1. Your JDK version cannot be earlier than V1.8.

2. If Kafka and FineDataLink are deployed on the same server, they can use the same Java environment. The steps are as follows.

Configure the Java environment in the etc/profile file by adding the following content.

Note:

You need to replace %FineDataLink% in the following code with the actual installation path of your FineDataLink.

export JAVA_HOME=%FineDataLink%/jdk/jre
export PATH=$JAVA_HOME/bin:$PATH
export CLASSPATH=.:$JAVA_HOME/lib/tools.jar

Execute source /etc/profile to apply the changes.

For details about Kafka performance, see the official Kafka document Apache Kafka.

Note:

The Kafka installation package version provided in this example is V3.5.1. For Kafka of V3.9.0 or earlier versions, non-critical vulnerabilities numbered CVE-2025-27817, CVE-2025-27818, and CVE-2025-27819 have been detected. The solutions are as follows.

1. FineDataLink standalone deployment:

• Solution One: Upgrade the FineDataLink to V4.2.7.2 or later versions. New users are advised to directly deploy the FineDataLink project of the latest version.

• Solution Two: If you do not want to upgrade the FineDataLink, enter FineDataLink installation directory/bin and add the following to the catalina.sh file:

JAVA_OPTS='-Dorg.apache.kafka.disallowed.login.modules
=com.sun.security.auth.module.JndiLoginModule,com.ibm.
security.auth.module.LdapLoginModule,org.apache.kafka.
common.security.oauthbearer.OAuthBearerLoginModule'

Restart the FineDataLink project.

2. If FineDataLink is deployed using FineOps, upgrade its images to V4.2.7.2 or later versions.

Downloading the Installation Package

You can click to download the Kafka installation package: Kafka

Note:

Production use of KRaft mode requires Kafka of V3.3.1 or later versions.

Decompressing the Installation Package

1. Upload the Kafka installation package. In this example, the installation package is uploaded to /opt/kafka, as shown in the following figure.

2. Use the command tar zxvf Name of Kafka installation package.tgz to decompress the installation package, as shown in the following figure.

Note:

Do not decompress the package in the Windows environment and then upload the extracted file. This will modify the file and trigger an error, saying "/bin/bash^M: bad interpreter: No such file or directory."

Modifying the Data Storage Directory of Kafka

Create a folder for Kafka to store data temporarily at runtime.

For example, you can create a kafka-logs folder in /data, as shown in the following figure.

Note:

The folder location is not specified. You are advised to create the folder at a location with relatively large disk space on the server.

Open the server.properties file in the config folder in the Kafka installation directory, find log.dirs=, replace the value after the equal sign with the pre-created directory for Kafka data storage, and save the change, as shown in the following figure.

Note:

The path here is an example. You can set the data directory location according to your actual situation.

Modifying the listeners Parameter

If Kafka and FineDataLink are not deployed on the same server, you need to modify the value of the listeners parameter in the server.properties file. For details, see Adding Configuration.

You can skip this section if Kafka and FineDataLink are deployed on the same server.

Note:

You do not need to specify topics in the Kafka configuration file. FineDataLink will automatically create a separate topic for each table.

Formatting the Data Storage Directory of Kafka

1. Generate a unique ID for the storage directory.

Navigate to the bin folder in the Kafka installation directory and execute the following command to generate a unique ID for the storage directory.

./kafka-storage.sh  random-uuid

2. Format the storage directory.

In this example, QRgd035ZQEqI9RPltqK7cg is the unique ID for the storage directory generated in the previous step, and /opt/kafka/kafka_2.13-3.5.1 is the installation directory of Kafka. Adjust the command based on the actual condition.

./kafka-storage.sh  format -t QRgd035ZQEqI9RPltqK7cg   -c  /opt/kafka/kafka_2.13-3.5.1/config/kraft/server.properties

Starting Kafka

1. Enter the bin folder in the Kafka installation directory and use the following command to start Kafka.

./kafka-server-start.sh -daemon ../config/kraft/server.properties

2. Execute the following command to check if Kafka has been started successfully. 

ps -ef|grep kafka

After installing and deploying Kafka, you can configure Transmission Queue in FineDataLink. For details, see Transmission Queue Configuration.

Before restarting Kafka, you need to manually pause pipeline tasks. After restarting Kafka, you need to manually restart the pipeline tasks. Otherwise, exceptions will occur in pipeline tasks.

In this example, the installation directory of Kafka is /opt/kafka/kafka_2.13-3.5.1. Adjust the command based on the actual condition.

1. Use the following command to view the Kafka service log, which records information such as the startup, shutdown, and status changes of Kafka service components. 

cat /opt/kafka/kafka_2.13-3.5.1/logs/server.log

2. Use the following command to view the Kafka controller log, which records the operation status and transaction processing information of the Kafka controllers, such as electing a new controller and assigning replicas. 

cat /opt/kafka/kafka_2.13-3.5.1/logs/controller.log