4.1.1
Supported the setting of service log level.
Supported the backup, restoration, and cleanup of service logs.
Supported manual/automatic cleanup of the diagnostic logs of scheduled tasks using the function described in the "Cleaning up Operation Logs and Service Logs" section.
4.2.4.3
Applied the configuration of Log Cleanup under System Management > Intelligent O&M > Platform Log > Global Setting to execution records and service logs of real-time tasks to prevent excessive accumulation.
A running FineDataLink project generates log files that record relevant activities.
System Log
It records information generated during system operation.
It is stored in FineDataLink installation directory\logs\fanruan.log by default.
Modifying the log storage location is prohibited.
You can choose System Management > Intelligent O&M > Platform Log > System Log and click Click to download to download logs on the platform.
You can also access the corresponding storage path on the server to retrieve the logs.
Operation Log
It records the activities of regular users and administrators.
It is stored in FineDataLink installation directory\webapps\webroot\logs\cubes.
You can modify the log storage location.
You can obtain logs in the following ways:
You can choose System Management > Intelligent O&M > Platform Log > Operation Log to view logs on the platform.
You can also synchronize data in the LogDB database to other databases for analysis. For details, see LogDB Database.
Service Log
Execution Record
1. It is generated during the use of various business modules.
For FineDataLink, execution records include:
Execution records of pipeline tasks in Data Pipeline
Execution records of scheduled tasks in Data Development
Call records of APIs in Data Service
2. Log level setting is not supported.
For FineDataLink of versions before V4.1.7.3:
Execution records are stored in fine_dp_work_exec_record and fine_dp_pipeline_execute in the FineDB database by default.
They are not subject to the configuration in the "Cleaning up Operation Logs and Service Logs" section. For details about cleanup operations, see Automatic Cleanup of Execution Records in FineDataLink.
For FineDataLink of V4.1.7.3 and later versions:
The task execution records in fine_dp_work_exec_record are subject to the configuration in the "Cleaning up Operation Logs and Service Logs" section.
The most recent execution record stored in fine_dp_work_last_record in the FineDB database can only be cleaned up by using automatic cleanup of execution records in FineDataLink.
You can view logs in FineDataLink.
You can also synchronize data in the LogDB database to other databases for analysis.
Diagnostic Log
1. Diagnostic logs are process logs generated during the use of various business modules.
For FineDataLink, diagnostic logs include:
Read and write process logs and exception and error logs of pipeline tasks in Data Pipeline
Read and write process logs and exception and error logs of scheduled tasks in Data Development
Call records and exception and error logs of APIs in Data Service
2. Log level setting is supported.
Data Pipeline and Data Service:
The diagnostic logs of pipeline tasks in Data Pipeline are stored in fine_pipeline_log and fine_pipeline_sync_information in the built-in LogDB database by default.
The diagnostic logs of the Data Service module are stored in fine_record_fdl_dataservice_monitor_info in the built-in LogDB database by default.
The data in these three tables can be manually or automatically cleaned up using the function introduced in the "Cleaning up Operation Logs and Service Logs" section.
For FineDataLink of V4.1.7.3 and later versions, you can choose O&M Center > Scheduled Task > Running Record and export the execution records of an instance as files. For details, see Scheduled Task O&M - Running Record.
Scheduled Task:
The diagnostic logs are stored as files by default, with each execution record generated as a file. (The storage path is FineDataLink installation directory\webapps\webroot\WEB-INF\dpworks\record.)
FineDataLink of versions before 4.2.1.4 does not support the manual/automatic cleanup function introduced in the "Cleaning up Operation Logs and Service Logs" section.
FineDataLink of V4.2.1.4 and later versions supports the manual/automatic cleanup function introduced in the "Cleaning up Operation Logs and Service Logs" section. Cleanup logic: All record files are deleted except those associated with the records stored in the LogDB fine_dp_work_exec_record table and the FineDB fine_dp_work_last_record table.
System logs record information generated during system operation.
The logs are stored in the file fanruan.log in FineDataLink installation directory\logs and are also generated in the Tomcat server's log file catalina.out.
Modifying the storage location of system logs is not supported.
The file fanruan.log in FineDataLink installation directory\log cannot be cleaned up automatically.
The fanruan.log file is compressed daily at 00:00 to save disk space.
The system logs are classified into five levels during output, namely DEBUG, INFO, WARN, ERROR, and FATAL.
Log levels ranked by severity (from highest to lowest): FATAL > ERROR > WARN > INFO > DEBUG
Log levels ranked by verbosity (from highest to lowest): FATAL < ERROR < WARN < INFO < DEBUG
DEBUG
It indicates fine-grained information events helpful for debugging FineDataLink, mainly used during development to print some runtime information.
INFO
It indicates the coarse-grained operation process of FineDataLink, mainly used to print some noteworthy or important information.
It can be used to output some important information about program execution in production environments, but should not be overused to avoid printing excessive logs.
WARN
It indicates situations where potential errors may occur. While the information does not always indicate errors, it can provide hints to programmers.
ERROR
It indicates that although an error has occurred, it does not affect the continuous operation of the system. It is mainly used to print error and exception information.
To avoid outputting excessive logs, you can use this level.
FATAL
It indicates that an error is severe enough to terminate FineDataLink.
This level is relatively high, representing critical errors that can directly stop the program.
You can log in to FineDataLink as the super admin, choose System Management > Intelligent O&M > Platform Log, click the icon, and adjust log levels as needed, as shown in the following figure.
The operation logs are stored in FineDataLink installation directory\webapps\webroot\logs\cubes by default.
All operation logs are stored in the cubes folder and can be queried for analysis through data connections. For details, see LogDB Database.
Operation logs are displayed in the form of tables under System Management > Intelligent O&M > Platform Log > Operation Log.
All operation logs generated in FineDataLink are displayed here.
Common operation logs generated in FineReport and FineBI are displayed here.
For Windows systems: Go to Tomcat installation directory\bin. Find the setenv.bat file. If the file does not exist, create a setenv.bat file. Add the following line to the file, and restart the project after configuration.
SET "JAVA_OPTS=%JAVA_OPTS% -DSWIFT_HOME=D:\path\logs"
For Linux systems: Go to Tomcat installation directory\bin. Find the setenv.sh file. If the file does not exist, create a setenv.sh file. Add the following line to the file, and restart the project after configuration.
JAVA_OPTS="$JAVA_OPTS -DSWIFT_HOME=\home\path\logs"
Operation logs and service logs are backed up together. Backing up system logs is not supported currently.
For projects deployed via FineOps, if you select the Elasticsearch component during deployment, Elasticsearch will replace swift(logdb) as the log storage component.
If Elasticsearch Connection Configuration is displayed on the General page under System Management > System Setting, ElasticSearch has been enabled as the log storage component.
In this case, the Log Backup function is unavailable. You can back up the Elasticsearch data as needed.
Log in to FineDataLink as the admin, choose System Management > Intelligent O&M > Platform Log, click the icon, and enable Auto Backup, as shown in the following figure.
The tables storing real-time backup of operation logs and service logs:
fine_record_execute
fine_record_operate
fine_record_email
fine_record_write
fine_schedule_record
fine_pipeline_log
fine_pipeline_sync_information
fine_record_fdl_dataservice_monitor_info
With Auto Backup enabled, logs are backed up by timestamp (for example, 20211105) and stored in FineDataLink installation directory\webapps\webroot\logs\swift_backup, as shown in the following figure.
The backup files will be cleaned up according to the log cleanup logic described in the "Cleaning up Operation Logs and Service Logs" section.
Operation logs and service logs can only be restored when Auto Backup is enabled.
Click Backup Restoration. A date selection box will pop up. Select the date range of the logs to be restored, and click OK.
After a short wait, a prompt saying "Backup Restoration Successful" will pop up, indicating the logs have been successfully restored, as shown in the following figure.
1. If you refresh the page during restoration and click Backup Restoration again, a prompt saying "Backing up and restoring..." will pop up. Wait patiently for the process to complete.
2. If insufficient memory is detected during restoration, a prompt will be displayed, indicating that disk cleanup is needed.
3. If a corrupted backup file is detected during restoration, a prompt will notify you of the corruption, and the restoration will not proceed.
Log in to FineDataLink as the super admin, choose System Management > Intelligent O&M > Platform Log, click the icon, and configure the cleanup mechanism of the operation log files and service log files. You can set Auto Cleanup or Manual Cleanup.
1. Auto Cleanup: The system automatically cleans up log files every three months, retaining data generated in the last three months. Therefore, the log files will be retained for at least three months, and actually up to six months at most.
2. Manual Cleanup: The system retains the data generated in the last three months by default. You can set the data retention period to zero day/one day/one week/one month/three months/six months/twelve months, as shown in the following figure.
Note:
The settings of Log Cleanup apply to operation logs and service logs only, not system logs.
Retain Last Period Data refers to retaining data in the specified period preceding the current date. For example, if you set Period to 3 Months and clean up data manually on 2019/08/19, only the data generated from 2019/05/19 to 2019/08/19 will be retained.
After log cleanup, the cubes folder still occupies significant disk space because only files inside the folder were cleaned up, not subfolders. The cubes folder contains numerous subfolders that collectively consume substantial storage.
If you set the reservation strategy to retaining data generated in the past month and perform an upgrade within one month after configuration, all log files will be automatically saved in a folder named after the upgrade date in FineDataLink installation directory\webapps\webroot\WEB-INF\dpworks\record during the upgrade.
In this case, FineDataLink cannot delete log files saved in the folder named after the upgrade date during automatic cleanup one month after the configuration, even if those files were created long before the upgrade.
But one month after the upgrade, the system will automatically clean up the log files in the folder named after the upgrade date in the records folder.
The current log size displayed in Log Size after cleanup does not represent disk space usage, but rather the total size of all files.
3. Cleanup Warning
With Cleanup Warning enabled, the system will notify relevant users when log files are too large, thus they can manually clean up logs and reduce file size.
Enable Cleanup Warning and set Trigger Condition to Log file larger than n MB.
SMS Reminder is available after you enable the SMS function. For details about the settings, see SMS. Email Notification is available after you configure senders. For details about settings, see Mailbox.
Tick SMS Reminder, Platform Message, and Email Notification, set the recipients respectively, and click Save, as shown in the following figure.
The following figure shows the effect of SMS Reminder.
The following figure shows the effect of Platform Message.
The following figure shows the effect of Email Notification.
Problem:
In the "Cleaning up Operation Logs and Service Logs" section, if you have set the data retention period to one month, the fine_dp_work_exec_record table contains records generated in the past one month. However, the %FDL_HOME%\webapps\webroot\WEB-INF\dpworks\record directory still contains log files that were generated much earlier.
Cause:
If the system has been upgraded within the past month, all log files will be automatically saved in a folder named after the upgrade date in the directory %FDL_HOME%\webapps\webroot\WEB-INF\dpworks\record during the upgrade. When FineDataLink automatically cleans up the records generated in the last month, it cannot simultaneously delete the log files saved on the upgrade date, even if those files were created long before the upgrade.
Only after an execution record is deleted will its corresponding log files be deleted. But one month after the upgrade, the system will automatically clean up the log files in the folder named after the upgrade date in the records folder.
1. You can set Service Log Level to ERROR, WARN, or INFO, with WARN selected by default, as shown in the following figure.
2. The configuration of Service Log Level only applies to pipeline tasks and scheduled tasks currently.
3. For pipeline tasks and scheduled tasks, you can adjust the log levels of individual tasks. For details about setting log levels for pipeline tasks and scheduled tasks individually, see Task Control - Task Attribute and Pipeline Task Configuration - Pipeline Control.
4. If you modify the log level of a task and modify the global log level later, the task specific log level setting takes priority over the global log level setting.
For details, see the "Backing up Operation Logs" section.
For details, see the "Cleaning up Operation Logs and Service Logs" section.
滑鼠選中內容，快速回饋問題
滑鼠選中存在疑惑的內容，即可快速回饋問題，我們將會跟進處理。
不再提示
10s後關閉
Submitted successfully
Network busy