Overview
Version
Report Server Version | System Operation Plug-in Version | Function Changes |
V11.0 | / | / |
V11.0.8 | V1.4.5 | 1. Automatically export memory stack supported selecting export content: Stacks, histo, dump 2. Automatically export memory stack is enabled by default, and all options are selected by default. |
V11.0.14 | V1.4.6 | 1. Optimized Downtime Reason and Recommended solution in Downtime self-help wizard. 2. Automatic handling of downtime was compatible with FineBI. 3. The Process auto recovery function was split from the Auto restart when the machine is down function to be configured to take effect at a specific time. 4. The Auto restart when the machine is down function was no longer strongly associated with the automatic export of configuration dump. 5. The Automatically export memory stack function was enabled by default. |
Application Scenario
The report server downtime may occur due to various reasons after it is deployed.
If a server is restarted without capturing the DUMP file after downtime, it can be difficult to determine the root reason, making it difficult to quickly resolve the issue and take preventive measures to avoid it from happening again. This can result in significant time and energy loss, and it is also detrimental to server maintenance.
FineReport has the built-in Intelligent Operations plugin that provides the Downtime Handling function.
After downtime occurs, analyzing the generated DUMP file can quickly determine the reason, allowing for timely resolution of the issue and effective preventive measures to be taken, which provides an extra level of protection for you.
Function Introduction
The Downtime Handling function is essentially an automatic downtime handling tool, with two entry points for both the platform and the client.
1. Platform side: The built-in part of the platform mainly includes tool operation environment monitoring and some settings, including Downtime self-help wizard, Running check, Downtime Handling, Memory stack export record, and Server restart record.
2. Client side: It mainly monitors the operation status of the server.
Note: 1. The Downtime Handling function is not supported for servers deployed on non-Tomcat containers or servers built-in to the designer.
2: In case of downtime in servers integrated with both FineReport and FineBI, the Downtime Handling function of FineBI is given priority and that of FineReport is disabled, in order to ensure the normal operation of FineBI servers.
Platform Side
As the admin, you can log in to the decision-making platform, click Manage > Intelligent Operations > Downtime Handling, and access the corresponding functional modules on the platform side.
The functions under the module include five parts: Downtime self-help wizard, Running check, Downtime Handling, Memory stack export record, and Server restart record.
Note: The Downtime Handling page may take some time to load when opened on the platform side, so please be patient.
Downtime Self-help Wizard
It records the time and reason of the server downtime, and provides corresponding solutions.
The common reasons of downtime and recommended solutions are shown in the following table:
Reasons | Recommended Solutions |
Memory overflow, you can view the specific template that caused the problem, including: 1. Suspended template 2. Template calculation consumes too much memory. 3. The template calculation time is too long. 4. The SQL execution time is too long. 5. The number of rows in the template dataset 6. exceeds the recommended limit. 7. The number of template cells exceeds the recommended limit. | 1. It is recommended to perform performance troubleshooting on the templates in the details, while enabling template limitations and configuring reasonable limitation ranges to avoid template performance issues that may cause downtime. 2. If the current system heap memory configuration is less than the recommended configuration, it is recommended to use the system diagnostic function to check the application and modify the heap memory to the recommended value. |
There is a BUG in the version of JDK used. | It is recommended to upgrade to JDK8 version 1.8.0_181 or above. |
There is unreasonable memory configuration in the system. | Use the system diagnostic function to check the application and modify the memory configuration according to the recommended values. |
Insufficient disk space | Check the disk space and clean up unnecessary files. |
The number of memory-mapped files is set too low. | Use the system diagnostic function to check the application and modify the configuration of the number of memory-mapped files according to the recommended values. |
There is a bug in the current version that causes downtime. | It is recommended to update to the latest minor version. For FineReport update and upgrade, see FineReport Version Info. For FineBI update and upgrade,see FineBI Version. |
Unknown reasons | It is recommended to upload this situation to the cloud-based operation and maintenance logs or contact technical support group for feedback. |
SSH exit results in application exit. | If an application is started through SSH, it will also exit when SSH is exited. It is recommended to use other command line remote tools such as secureCRT or configure other application servers to start automatically. |
The data source takes too long to extract the data, causing thread blocking. | It is recommended to use methods such as extracted data cache, SQL optimization, and data preprocessing to improve data extraction performance. |
Thread blocking due to log output | Increase the log output level to reduce the log output or check if there is insufficient disk space. For details, see Log Introduction. |
The system takes too long to release memory. | If the heap memory used is equal to or greater than 64 GB, adjust the heap memory to a value less than 64 GB. If the heap memory used is less than 64 GB, it is recommended to use a higher performance CPU. |
Running Check
To use the functions referred to in section "Downtime Handling", it is necessary to ensure that each item of Running check meets the requirements.
The system needs to meet certain conditions to ensure the normal operation of the automatic downtime handling tool. Therefore, after the server is started, it will first check the port status, JDK environment, free memory, and deployment status.
If any issues are found, you will be alerted to make the corresponding modifications or disable the downtime function. If no issues are found, the system will continue to run stably.
The check contents for each item in Running check are shown in the table below:
Note: If the check result is normal, it will prompt This configuration is good, no need to adjust.
Serial Number | Items | Criteria | Suggestions |
1 | Operating System | The current system is not a Windows/Linux system. | It is recommended to use a Linux system to ensure the stable operation of the automatic downtime handling tool. In this situation, the following four items will not be checked. |
2 | Port | The status of port 12100 is abnormal (the port is not open or is occupied). | It is recommended to open port 12100 or set other ports to ensure the normal operation of the system. For details about port settings, see section "Downtime Notification". |
3 | JDK | 1. The server contains tools.jar that is not from Oracle JRE, and the server is not configured with a JDK (there is an issue with the system configuration of the JDK). 2. The server does not contain tools.jar, and the server is not configured with a JDK (there is an issue with the system configuration of the JDK). | It is recommended to add system JDK configuration or add tools.jar to ensure the normal operation of the system. |
4 | Free memory | Insufficient free memory | It is strongly recommended that the total physical memory of the host machine, minus the memory used by the container where the system is running, should not be less than 10 GB. |
5 | Deployment method | Not Tomcat container deployment | It is recommended to use Tomcat container for deployment. |
When it is detected that the operating system, port, JDK, or free memory affect the function usage, you will be notified through platform messages and pop-up messages in the lower right corner of the platform.
Click Process, and you will go to the configuration page of Downtime Handling, where you can handle different situations accordingly.
Downtime Handling
If you want to use the Downtime Handling function, you must ensure that each item in section "Running check" meets the requirements.
The items you can set at Downtime Handling include: Automatic handling of downtime, Automatically export memory stack, Auto restart when the machine is down, Process auto recovery, Downtime notification, and Port setting.
You should click the Save button to make all the following settings effective after modification.
1. Automatic Handling of Downtime
Introduction:
To configure the functions below, it is necessary to enable this option. Otherwise, all functions will be unavailable.
This option is enabled by default.
Prerequisites:
During working hours (6:00-23:00), when the main process of the application is closed for 5 minutes, Automatic handling of downtime will also be turned off.
During non-working hours (0:00-6:00, 23:00-24:00), the main process of the application is closed, but Automatic handling of downtime will not be turned off.
2. Automatically Export Memory Stack
Introduction:
After the option is enabled, downtime logs will be exported automatically when there is downtime.
The supported downtime logs include: stack, histo, and dump.
The exported logs will be located in the directory %Tomcat%\logs\FineLog\date folder on the server where the downtime occurs.
Note: After enabling this function, downtime logs will be automatically exported. If you are concerned about the long export time of dump affecting server performance, you can choose to only export Stacks and histo.
Prerequisites:
If Automatic handling of downtime is not enabled, this option will be grayed out and can not be edited or modified.
If Automatically handling of downtime is enabled, this option will be enabled by default.
3. Auto Restart When the Machine is Down
Introduction:
After it is enabled, the server will automatically restart when downtime occurs due to high load.
Prerequisites:
a. Automatic handling of downtime is enabled.
If Automatic handling of downtime is not enabled, this option will be grayed out and can not be edited or modified.
If Automatic handling of downtime is enabled, this option is disabled by default.
b. System status
When the option is enabled, the current system status will be checked. Check if it is a Windows operating system and whether it is in the form of a Windows service.
If the system is a Windows operating system and in the form of a Windows service, a pop-up window will appear indicating that this function does not support the current system. Click Complete or the close button to close the pop-up window, and the option will not be enabled.
If the system is a Windows operating system but not in the form of a Windows service, a pop-up window will appear indicating that the system may fail to restart. Click Complete or the close button to close the pop-up window, and the option will be enabled.
4. Process Auto Recovery
Introduction:
After this function is enabled, you can set the effective time period for the function. During a specified time period, the server will automatically restart when the application process disappears.
Prerequisites:
a. Automatic handling of downtime is enabled.
If Automatic handling of downtime is not enabled, this option will be grayed out and can not be edited or modified.
If Automatic handling of downtime is enabled, this option is disabled by default.
b. System status
When the option is enabled, the current system status will be checked. Check if it is a Windows operating system and whether it is in the form of a Windows service.
If the system is a Windows operating system and in the form of a Windows service, a pop-up window will appear indicating that this function does not support the current system. Click Complete or the close button to close the pop-up window, and the option will not be enabled.
If the system is a Windows operating system but not in the form of a Windows service, a pop-up window will appear indicating that the system may fail to restart. Click Complete or the close button to close the pop-up window, and the option will be enabled.
5. Downtime notification
If Automatic handling of downtime is not enabled, this option will be grayed out and can not be edited or modified. If Automatic handling of downtime is enabled, this option is disabled by default. After this function is enabled, you can configure SMS Reminder, Platform Message, and Email Reminder to be notified in the configured way when there is downtime.
6. Port Setting
Introduction:
You are supported to set the port used by the downtime handling tool. The default port is 12100.
The port number range must be between 1024 and 65535, otherwise the downtime handling tool can not be started, and the Downtime Handling page can not be opened.
Prerequisites:
If Automatic handling of downtime is not enabled, this option will be grayed out and can not be edited or modified.
If Automatic handling of downtime is enabled, the default port is 12100.
After entering the new port number, click Test. If the new port number is invalid, a pop-up window will appear saying: The port is abnormal, please reset.
If the port is normal, a pop-up window will appear indicating: The port is available. After saving, the automatic downtime processing tool will restart on the new port.
Memory Stack Export Record
Display the record of the automatically generated DUMP file.
It includes Export start time, Export content, Export duration, Is it successful, and Reason of failure.
Note: Only the downtime handling records from the past month will be read.
Server Restart Record
It includes Restart start time, Restart duration, Is it successful and Reason of failure.
Note: Only the downtime handling records from the past month will be read.
Client Side
After starting the server with admin privileges, you can go to the downtime tool client interface by entering in the browser: http://IP:port (the default port for Automatic Downtime Handling Tool is 12100. If the port has been modified, enter the modified port) / operation / tool
You can log in to the client interface using the account and password. If the login fails five times, the account will be locked for 60 minutes, during which you can not attempt anymore. The login session is valid for 15 minutes, and you will be automatically logged out if there is no activity within this time.
Note: If you are locked out due to downtime, you need to manually terminate the process.
The user interface mainly includes two functions: monitoring the running status of the server and outputting the stack. There is a Logout button in the upper right corner, and you can log out by clicking it.
Running Check
If the server is running normally, the page will display: Normal running
If there is downtime, the page will display: Has been down, handling
After downtime occurs, when the downtime handling tool is working automatically, you can manually terminate the process. Once terminated, the page will display: The downtime automatic handling has been stopped, please manually restart the system in time.
After downtime occurs, if the tool fails to restart the project, the page will display: Failed to restart automatically, please manually restart the system in time
Output Stack
You can export thread stack and memory stack files at Output Stack by clicking the corresponding content.
If the export is successful, a message will be displayed indicating that it is successful. Otherwise, a message will be displayed indicating that is has failed. The exported stack file is located at %Tomcat%\logs\FineLog\date.
