Overview
Version
FineOps Version | Functional Change |
|---|---|
V1.5.5 | / |
V1.5.7 | 1. Split and optimized the log management function of FineOps. 2. Supported the downloading of multi-node logs for multi-node cluster projects with a single operation. 3. Allowed downloading the required log by service type. |
V1.8.0 | Added Triggering Type to the analysis record list of the analysis log. |
V1.12.0 | 1. Allowed manually uploading local performance stack to FineOps for analysis. 2. Allowed displaying historical performance stack generation records. 3. Optimized the performance stack usage by changing the usage order to Analysis, Download, and Generation. |
V1.14.0 | Optimized the triggering condition of automatic log generation in case of a crash detected by FineOps. 1. Changed the application unresponsive time that triggered automatic log generation from three minutes to two minutes 2. Added a FullGC triggering condition: The total number of full GCs in the past five minutes is greater than three, with the total time exceeding 50 seconds. |
V1.15.0 | Allowed downloading logs by log type as needed, eliminating the need to download all logs, thereby reducing the costs. |
V2.0.0 | Optimized performance analysis results and added Memory Allocation. |
V2.1.0 | 1. Supported performance stack generation, analysis, and downloading for unusable projects. 2. Optimized analysis reports by splitting memory and CPU information, and improving the GC status display. |
Application Scenario
If a project crash or exception occurs:
1. In case of project crashes, the admin needs to quickly generate the stack log for troubleshooting and review afterward.
2. The admin needs to analyze the stack log to trace the root cause of the problem, which requires high technical competence.
3. The admin needs to download the stack log for analysis or provide it to FanRuan technical support staff to help confirm the issues in the project. However, log retrieval is laborious for many companies.
FineOps provides the Performance Stack function that enables rapid generation, intelligent analysis, and easy downloading of stack logs.
Log Introduction
Performance Stack supports the generations, analysis, and downloading of the following stack logs:
Log | Description |
|---|---|
fanruan.gc.log | It is the crash log of the FanRuan application. |
thread.txt | It is the thread information statistics report, which includes the status and stack trace information of all threads. It can help you understand the thread usage of the application, and promptly discover and solve thread issues. |
histo.txt | It is the memory usage statistics report, which includes the number, size, and occupied space of objects. It can help you understand the memory usage of the application, and promptly discover and solve issues such as memory leaks. |
hs_err.txt | It is the error report file automatically generated in case of errors during the Java runtime, which includes detailed error information, stack trace information, and thread information. It can help you locate and solve code issues. |
top.txt | It is the output result of the top command on Linux systems, which is used to display relevant information about running processes in the current system, including top-H.txt, top-sortedByCPU.txt, and top-sortedByMEM.txt. 1. top-H.txt: This file contains the output result of the top command, which displays thread information including thread ID, CPU usage, and memory usage. 2. top-sortedByCPU.txt: This file contains the output result of the top command sorted by CPU usage in descending order, including process ID, CPU usage, and memory usage. 3. top-sortedByMEM.txt: This file contains the output result of the top command sorted by memory usage in descending order, including process ID, CPU usage, and memory usage. |
Analysis Log
Application Scenario
When the project crashes, the admin needs to analyze the crash logs to fix existing issues and prevent recurrence.
FineOps supports the analysis of the Java thread stack traces (printed by the stack command), and the contents of histo.txt, fanruan.gc.log, and top-H.txt files, helping users analyze stack errors and providing relevant improvement suggestions.
Automatic O&M Project Analysis
FineOps will automatically generate logs if the conditions in section "Automatic Log Generation" are met, and will analyze the automatically generated stack logs of each node.
1. Log in to FineOps as the admin, click Project Management, and click the specified O&M project.
2. Go to the Log > Performance Stack page. Under the Analysis module, you can view the automatic analysis results.
Manual O&M Project Analysis
Procedure:
1. Log in to FineOps as the admin, click Project Management, and click the specified O&M project. Go to the Log > Performance Stack page.
2. Click the Analyze button under the Analysis module to trigger the generation of stack logs for each node of that O&M project.
3. Check the FineOps analysis results of stack logs of each node in Analysis Record.
Note:
Only one analysis is supported per minute to prevent poor system performance caused by frequent triggering. If analysis requests are too frequent, a prompt will appear: "Frequent generation of stack logs. Retry in 1 minute(s)."
Analysis of Manually Uploaded Logs
You can manually upload local stack logs to the O&M platform for analysis.
Procedure:
1. Prepare the local stack log package.
Only ZIP, TAR, and GZ files are supported. Ensure that the stack logs are generated using the jstack -l command line.
The log package size must not exceed 100 MB.
FineOps will generate an analysis report based on the package's Java thread stack traces (printed by the stack command) and the contents of histo.txt, fanruan.gc.log, and top-H.txt files. Only one node can be analyzed at a time.
2. Upload the local stack log package.
Log in to FineOps as the admin, click the O&M project on the Project Management page, and choose Log > Performance Stack.
Click the Upload Performance Stack Analysis button under the Analysis module, select a local log package, and click Open.
3. Generate an analysis report.
The analysis results are displayed in a new tab instead of in Analysis Record.
Analysis Log Viewing
Log in to FineOps as the admin, click the O&M project on the Project Management page, and choose Log > Performance Stack.
The results of all manually and automatically triggered stack analyses of the O&M project are displayed in the form of a detail table under Analysis > Analysis Record (excluding manually uploaded log analysis records).
1. The content displayed in Analysis Record includes Project Name, Node Name, Triggering Type, Operation Time, Operator, Operator IP, and Analysis Result.
The operator and operator IP address of the manually triggered analysis are empty.
2. An analysis result is generated for each node of a clustered O&M project.
3. Only the latest 50 log analysis records are retained. You can delete a single analysis record and multiple analysis records in a batch, and clear all historical records.
4. Click the View button in Analysis Record to view specific analysis results in a new tab.
The following figure shows an example of performance stack analysis results, which have two parts: Memory Status Detail and Thread CPU Status Detail.
Memory Status Detail is divided into GC Status and Memory Status.
Thread CPU Status Detail:
Log Downloading
Application Scenario
When the project crashes, the admin needs to download the crash log for analysis or provide it to FanRuan technical support staff to help confirm the issues in the project and prevent them from happening again.
Procedure:
Number | Procedure | Notes |
|---|---|---|
| 1 | Log in to FineOps as the admin, click a certain O&M project under Project Management, and choose Log > Performance Stack. | |
| 2 | Selecting Time Range | Selectable range: seven days ago to the current date, with a selected time span not exceeding two days (48 hours) Default time: two hours ago to the current time |
| 3 | Selecting Service Type | Supported types: bi, fr Selectable range: All service types that exist within the selected time. All options are selected by default. Note: 1. Non-containerized O&M projects do not support selecting service types. 2. If none of the types exist, the option will not appear. |
| 4 | Selecting Node | Selectable range: All nodes in the selected O&M project. All options are selected by default. |
| 5 | Selecting Log to be Downloaded | Supported types: Downloadable logs are those generated in section "Log Generation", including: fanruan.gc.log, thread.txt, histo.txt, hs_err.txt, and top.txt Note: 1. If a certain type of log does not exist, downloading a log of that type is not supported. 2. In the O&M project deployed through containerization, if the elasticsearch component is installed for log storage, selecting types will not be supported. Logs will be downloaded in full directly. |
| 6 | Clicking Download to download the generated O&M log files to the local computer | 1. If the file size exceeds 5 GB, the log cannot be exported. A prompt will appear "The size of the following log(s) exceeds the upper limit (5 GB) and the download failed: Log name." 2. If the log file of this type does not exist, the log cannot be exported, and a prompt will appear: "Non-existing log: Log name." 3. When downloading logs, you need to compress files. If the waiting time is too long, you can click the Cancel Download button to stop it. 4. After the downloaded files are decompressed, they are stored according to the structure: Node-Date-Service Type-Log Type. |
Log Generation
Application Scenario
When the project crashes or experiences an anomaly, stack logs can help admins quickly locate the problematic codes.
When the project has performance issues (such as long response times or high CPU usage), stack logs can help admins understand the program's execution path and identify potential performance bottlenecks.
When the project encounters issues, histo logs can help admins track the user's historical input commands, keep operational staff monitoring of user behavior, and trace the root cause of the problem.
FineOps supports manual and automatic generation of stack and histo logs.
Automatic Triggering
1. Automatic log generation triggering
The following three situations will trigger automatic log generation.
Situation | Description |
|---|---|
The O&M platform monitors the project and automatically generates logs when it crashes. | 1. Click an O&M Project. Log in to FineOps as the admin, click Project Management, and click the specified O&M project.
2. Enable Auto Crash Handling. Go to the Crash > Crash Handling Strategy page, enable Auto Crash Handling, and click Save.
3. Configure Automatic Generation Content. Go to the Log > Performance Stack page, under the Generation > Automatic Generation module: You can select generation types: including stack and histo. All options are selected by default. You can select service types: including bi, fr, and fdl. All options are selected by default (Non-containerized O&M projects do not support selecting service types. If none of the types exist, this option will not appear.).
4. When the following situations occur in the O&M project, logs are automatically generated: CPU usage exceeds 80% (fr, fdl, bi-web, and bi-worker). GC occurs for more than 30 seconds (fr, fdl, bi-web, bi-worker, and bi-master). The application is unresponsive for two minutes (fr, fdl, bi-web, bi-worker, and bi-master). Thread pool usage reaches 85% (fr, fdl, bi-web, and bi-worker). High load persists after consecutive FullGCs (five consecutive FullGCs with load score greater than 100) (fr, fdl, bi-web, bi-worker, and bi-master). The total number of FullGCs within five minutes exceeds three times, and the total time exceeds 50 seconds (fr, fdl, bi-web, bi-worker, and bi-master). |
The project itself monitors crashes and triggers log generation. | 1. Enable Auto Crash Handling. Log in to the O&M project as the admin and choose System Management > Intelligent O&M > Downtime Handling > Automatic Downtime Handling Tool. Turn on Auto Resolve Downtime. Turn on Auto Export Memory Stack and choose generation types (stacks, histo, and dump). Click Save.
2. When the following situations occur in the O&M project, logs are automatically generated: CPU usage exceeds 80% (fr, fdl, bi-web, and bi-worker). GC occurs for more than 30 seconds (fr, fdl, bi-web, bi-worker, and bi-master). The application is unresponsive for two minutes (fr, fdl, bi-web, bi-worker, and bi-master). Thread pool usage reaches 85% (fr, fdl, bi-web, and bi-worker). High load persists after consecutive FullGCs (five consecutive FullGCs with load score greater than 100) (fr, fdl, bi-web, bi-worker, and bi-master). The total number of FullGCs within five minutes exceeds three times, and the total time exceeds 50 seconds (fr, fdl, bi-web, bi-worker, and bi-master). |
You trigger analysis log generation manually. | Log in to FineOps as the admin, click Project Management, and click the specified O&M project. Go to the Log > Performance Stack page, and click the Analyze button manually, then you can automatically trigger the generation of a stack log.
|
2. Automatically generated logs
When meeting the above conditions, each node of the O&M project will be automatically triggered to generate a stack log.
Location of the generated file:
Containerized projects: in the logs\FineLog\Date folder of the mounting directory of each engineering node
Non-containerized projects: in the %Tomcat %\logs\FineLog\Date folder of the server where each project node resides
Manual Generation
Prerequisite
To generate a stack log, you must enable the Auto Crash Handling function. Otherwise, a prompt will appear "The function has not been enabled yet. To start it, enable Auto Crash Handling under Crash > Crash Handling Strategy > Auto Crash Handling."
Log in to FineOps as the admin, click an O&M project for which logs need to be generated, choose Crash > Crash Handling Strategy, enable Auto Crash Handling, and click Save.
Procedure:
1. Log in to FineOps as the admin, click the O&M project on the Project Management page, and choose Log > Performance Stack.
2. Select the service type under the Generation > Manual Generation module.
Supported types: bi, fr
Selectable range: The service types existing in all nodes of the current O&M project. Non-containerized O&M projects do not support selecting service types. If none of the types exist, the option will not appear.
3. Then you can trigger the generation of stack logs for each node under that O&M project by clicking Generate.
4. Location of the generated file:
Containerized projects: in the logs\FineLog\Date folder of the mounting directory of each engineering node
Non-containerized projects: in the %Tomcat %\logs\FineLog\Date folder of the server where each project node resides
Note:
Only one stack can be generated per minute to prevent frequent triggers that could degrade system performance.
Generation Record
The historical generation records, including all automatically or manually triggered generation, will be displayed in Performance Stack. See the following figure:
The following information will be displayed in the form of a detail table: Node Name, Export Content, Export Start Time, Export Duration, Triggering Type, Successful or Not, and Reason of Failure.
