Understanding FineBI Deployment Principles

Function:

The load balancer is positioned between the client and the bi-web component.

It receives and distributes client requests across bi-web components. Therefore, the load balancer must have access to all BI nodes.

Advantage:

• Performance Optimization: It intelligently distributes requests across various BI nodes to prevent server overload and improve efficiency.

• Fault Isolation: It automatically detects and isolates faulty BI nodes to avoid impacting the overall service.

• Enhanced Security: It functions as the first line of defense to filter and monitor data flow, preventing malicious attacks.

• Session Persistence: You can configure session persistence strategies to ensure session continuity.

• Scalability: You can add BI nodes to meet business needs without affecting the client’s access, as it requires the load balancer address only. 

Type:

• nginx: It is a customized internal gateway for FanRuan projects to balance user requests and improve performance. The self-provisioned component is not supported.

• Others: To use other types of load balancers such as F5, SLB, and ELB, you can configure forwarding, allowing client requests to be forwarded to the self-provisioned gateway and then to the FanRuan internal gateway, and finally distribute them across BI nodes.

A BI node is divided into two layers, namely, the network forwarding layer and the business processing layer. 

Network Forwarding Layer:

A user's requests must be continuously handled by the same BI node. A unique session ID is generated when the client is connected to the BI node.

Subsequent requests from the client, regardless of which BI node they are forwarded to, can be identified by session ID and routed to the initially connected BI node to maintain session continuity.

Business Processing Layer:

This layer handles client requests, including platform functioning and front-end page rendering.

Tasks involving data updates or calculations are intelligently assigned to backend worker engines during request processing.

The master engine is used to store and provide worker information and store metadata.

Storing and Providing Worker Information:

Data updates and calculations involved in the request that is assigned to a BI node are distributed across worker engines intelligently for execution.

However, worker engines are stateless services that operate without storing any persistent data, so BI nodes cannot identify or contact any worker engine.

In this case, the BI node first communicates with the master engine, which dynamically provides a usable worker address to the BI node. This address is randomly assigned and used to complete the current task specifically.

The BI node then distributes the task to the worker engine of that address for it to conduct data processing and calculations.

Storing Metadata:

Metadata refers to the information of each BI data table (not the data within the BI data table), such as the table name, field names, and the location where it is stored in the data storage component.

This metadata is stored in the mounting directory of the bi-engine-master component, that is, on the server disk.

To fetch data from the data storage component for calculation or updates, the worker engine first obtains the metadata of these data tables from the master engine to accurately locate the required tables and then executes the task.

A worker engine consists of the monitoring probe and the worker engine.

Worker Engine:

Worker engines execute data update and calculation tasks distributed by BI nodes.

By default, an engine worker can handle both data updates and data queries.

Read-Write Separation and Its Application Scenario:

In formal business systems, data queries are typically performed during daytime working hours, while data updates are done during nighttime off-hours.

Data updates that occur during the day cannot consume resources for data queries. Similarly, data calculations that occur at night cannot consume resources for data updates.

In such cases, you can configure worker engines to let some nodes focus on specific tasks (to achieve read-write separation) during designated periods.

Monitoring Probe:                                     

The worker engine periodically (every 3 seconds) sends a heartbeat to the master engine, informing that it is still alive. If the master engine does not receive the heartbeat, it will assume the worker engine is faulty and needs to be restarted.       

The master engine will instruct the monitoring probe to execute the kill command, forcibly shutting down the worker engine and then the probe itself. The worker engine is triggered to restart automatically.

A restarted master engine cannot determine the status of the worker engine. Therefore, after restarting a master engine successfully, you must restart all worker engines.

It stores the basic tables and self-service datasets extracted in FineBI.

Note that the following data is not stored in the data storage component: 

• Excel dataset: It is stored in the /WEB-INF/assets/temp_attach path on the file server.

• Direct-connected data: It refers to cache data related to direct connection and is stored on the server and the disk.

The configuration database (which is finedb in older versions) is used to store configuration information of the project.

For example, it stores information about project users, directories, user permission, and data update schedules.   This configuration information is stored in a dedicated database that is always connected to the project, ensuring the long-term stable operation of the project.

A configuration database can be used for one project only. Otherwise, configuration conflict may occur.

For example, when accessing a cluster project with multiple BI nodes, you will always see the same platform style and directory whether you enter the project through the load balancer or a single node.

This is because each node reads the same information from the configuration database for displays.

The state server acts as a monitor, overseeing the running status of all nodes and components in the entire BI project. It records logs and error information, coordinates communication and task allocation between nodes, and more. 

Assume that you log into Node A and process business normally. When Node A crashes, the business is transferred to Node B for processing. Node B determines your login status by relying on the state server redis that stores the information.

The file server is used to store and share the files and data resources required in the cluster, ensuring that each BI node can access and use them.

The following lists some examples of files for you to understand the role of the file server.

• FineReport template files 

• FineReport template backups

• original Excel files in FineBI

• driver packages uploaded through Driver Management

• project resource files (such as maps and images)

• snapshot files generated by Task Schedule

• data packages generated by Cloud O&M

• project backups

The log service is used to record every operation performed by each user in the system, including login, data access, modification, and more. For FineBI 6.1, the elasticsearch component has replaced the original Swift engine to provide log services.

Necessity of Logs

Traceability: Logs record and help trace operations in the system, which is crucial for compliance audits.

Performance Analysis: Logs record the system's operation and performance data, helping analyze and evaluate system performance issues.

Behavior Analysis: Logs help understand user behavior patterns and usage habits, based on which you can improve system functions and user experience.

The engine service and the BI business are separated.

Engine failure barely affects business operations as the engine runs independently. The engine can automatically restart within minutes.

You can achieve high availability with multiple engine nodes.

5.x:

FineBI 5.x requires a plugin to implement a master-slave cluster model for data extraction, failing to provide high availability.  

It supports local disk storage only, failing to meet users' personalized storage needs. 

6.0.x:

FineBI 6.0.x has limited cluster scalability and supports up to five business nodes.

It supports local disk storage only, failing to meet users' personalized storage needs.

By introducing a unified data access layer that decouples data calculation and data storage, FineBI 6.1 addresses the challenges of data extraction and synchronization found in traditional multi-node clusters, such as inter-node synchronization, local data extraction, and persistent synchronization.

The computing engine now supports an unlimited number of nodes and horizontal scaling, with linear improvements in data query concurrency, meeting users' needs for high query concurrency.

The extracted data can be stored in the cloud object storage service (OSS) with persistence instead of being stored locally, meeting enterprises' data security requirements.

Complex Daily O&M

• Operations on projects deployed in a non-containerized manner, such as project start/stop, log acquisition, and system memory/CPU usage inspection, are performed in the background using command lines.

• There are no effective methods for monitoring the resource usage of the entire project and alerting. (You can only use the plugin to check memory and CPU usage on the front end.)

High Disk Usage

• 5.1.x: The master node continuously synchronizes extracted data to the slave node server, requiring large disk space on both master and slave servers to store the extracted data.

• 6.0.x: Nodes synchronize extracted data with each other, requiring large disk space on each synchronization node server to store the extracted data.

Simpler Daily O&M 

Basic FineBI O&M can be completed visually on the FineOps front end, such as starting and stopping projects, downloading logs, and generating stack dumps, effectively reducing O&M costs.

Less Disk Usage

Inter-node synchronization of extracted data is no longer required. You only need to store one persistent copy of the data in the minio component, which can be invoked by multiple engine nodes.

• Environment Consistency: Containers package applications and their dependencies, ensuring that applications maintain consistency across development, testing, and production environments, avoiding various issues caused by environment differences.

• Component Isolation: Each container operates in an independent environment, with processes isolated from each other, enhancing security and preventing interference between different applications.

• Resource Allocation: You can configure resources (such as the number of CPU cores and memory size) allocated to each container, ensuring efficient resource distribution.

• Simplified O&M Management: FineOps enables automated container O&M, including container start, stop, and management, simplifying O&M tasks.

• Convenient Monitoring: FineOps provides logging and monitoring solutions, making it easy to observe and analyze the running status of containers and identify performance risks in advance.

• Fault Tolerance: The worker engine processes can automatically restart, and you can quickly recover other services from failure through FineOps.