Overview
Version
| FineBI Version | Syncing Platform Logs to Other Databases Plugin |
|---|---|
6.0 | V1.0.0 |
Application Scenario
Currently, LogDB does not support migration. It is inconvenient for you to perform complex analysis on log data. You may hope to migrate log data to your own databases for management and analysis.
You may want to operate and manage logs of various systems in the same place.
Function Description
After installing the Syncing Platform Logs to Other Databases plugin, you can configure some table data to be periodically synchronized from LogDB to other databases.
Plugin Introduction
Plugin Installation
Click to download the plugin: Syncing Platform Logs to Other Databases.
For details about how to install the plugin, see Plugin Management.
Databases Supported for Migration
| Database Type | Supported Database Version |
|---|---|
| MySQL | 5.1.73/5.5.31/5.5.46/5.5.56/5.6.22/5.6.31/5.7.16 |
SQL Server | 2000/2005/2008/2012/2014/2016 |
Oracle | 10g/11g/12c |
Log Sync
After the plugin is successfully installed, choose System Management > Intelligent O&M > Platform Log. The Log Sync tab is added to the page, as shown in the following figure.
Example
Selecting the Logs to Be Synced
(1) Log in to the decision-making platform as the admin, choose System Management > Intelligent O&M > Platform Log, and click Set Log Sync Task on the Log Sync tab page, as shown in the following figure.
(2) On the Log Sync Setting page, select Log Sync Start Time, tick the logs that need to be synced, and click Next.
The logs that support sync include:
fine_record_execute (report execution log)
fine_record_login (login log)
fine_record_logout (logout log)
fine_record_email (email sending log)
fine_record_write (data entry log)
fine_record_param (common parameter combination log)
fine_record_sms (SMS sending log)
fine_record_operate (management behavior log)
fine_record_sql (SQL execution log)
fine_real_time_usage (real-time app status log)
fine_record_gc (GC log)
fine_record_conf (conf_entity configuration log)
fine_record_lock (user lock log)
fine_tpl_info (template attribute log)
fine_container_entity (container information log)
fine_function_process (function point information log)
fine_operation_process_log (cloud O&M log)
fine_operation_shutdown_record (semaphore information log)
fine_plugin_usage (plugin use log)
fine_schedule_record (task schedule execution log)
fine_intelli_focus_point (function use log)
fine_intelli_consume_point (performance log)
Configuring Database Information
Enter the relevant information of the database where the log data will be migrated, and click OK, as shown in the following figure.
Effect Display
Sync Status
Note:The plugin performs synchronization every three minutes. Up to 30,000 records can be synced per table each time. If the data to be synced this time is less than 30,000 records, only data within the last three hours can be synced.
For example, if 30 records of data are generated from 13:00:00 to 16:00:00 on 2021-08-02, these 30 records at most can be synced at this time.
The following figure shows the successful sync status.
Notifying the Admin If Sync Fails
If sync fails after the Notify Admin If Sync Fails checkbox is ticked, the admin will receive a platform message.
Pausing Sync
You can click the Pause button to pause the sync.
Editing Sync Settings
If sync is paused or fails, you can click the Edit button to re-edit the sync task.
The following describes how to set Effective Object of Start Time.
If you want to sync all selected tables from the set value of Log Sync Start Time, set Effective Object of Start Time to All Logs to Be Synced.
If you want to sync only added tables from the set value of Log Sync Start Time, and resume the sync of the left tables that are synced before from the breakpoint, set Effective Object of Start Time to Only Effective for Log in This Change.
Viewing Sync Details
Click the View Sync Details button to view the sync details, as shown in the following figure.
Only the operation status of the last sync task will be displayed. The following table describes each field.
| Field Name | Description |
|---|---|
Synced Table | Synced log |
Log Sync End Time | Latest time when logs are stored into the database in the last executed sync task, indicating that all logs from the set value of Log Sync Start Time to this time point have been stored |
Last Sync Date | Start time when the sync task is last executed |
Volume of Data Synced Last Time | Number of log records synced when the sync taks is last executed |
Time Consumed by Last Sync | Time consumed by the last sync |
Operation Result | Operation result, which can be Success or Failure You can click Failure to view the specific error if the operation result is Failure |
Syncing Log Data to the Database
Log data will be synced to the configured database, as shown in the following figure.
Note:1. Field values with a maximum of 255 characters can be stored into the database but with over 255 characters will be truncated.
2. Log cleaning will not affect the log data that has been migrated to external databases.
syncID (auto-increment primary key) and storageTime (timestamp field) will be added to the synced table, as shown in the following figure.
For compatibility with various versions, aliases are created for field names that use database-reserved names. The following table describes the matching between specific fields and alias.
| Table | Original Field Name | Field Name After Migration |
|---|---|---|
| FINE_RECORD_OPERATE | type | operateType |
| resource | operateResource | |
| status | operateStatus | |
FINE_RECORD_EXECUTE | type | executeType |
| sql | query | |
| sqlTime | queryTime | |
FINE_RECORD_WRITE | sql | query |
| sqlTime | queryTime |
Common FAQ
Sync Logic
Content | Description |
Sync Logic | Currently, the sync logic for platform logs is insertion submission. Data synced multiple times from the same time period will be submitted repeatedly. |
Sync Frequency | The plugin will sync logs to a third-party database in batches from the set value of Log Sync Start Time, in chronological order. The plugin syncs tables (each of which can contain a maximum of 30,000 records per sync) every 3 minutes. If the data to be synced this time is lesser than 30,000 records, data within the last 24 hours can be synced. (Before March 23, 2021, only data within the last 3 hours can be synced.) |
Sync-Supported Field Length | Field values with a maximum of 255 characters can be stored into the database but with over 255 characters will be truncated. |
Sync Status | Sync Status displayed on the Log Sync tab page specifies the operation status of the plugin, not the sync status. |
Database Problem
If the sync fails, you can troubleshoot the following database problems.
| Troubleshooting Step | Description |
|---|---|
Checking the Database Version | You can view the supported database types and versions by referring to section“Databases Supported for Migration.” |
Checking Database Information | Check if the database driver, schema, and password are correct. You can first check if the connection works properly under System Management > Data Connection. Database reboot will interrupt log synch. You need to restart the task on the platform when the sync terminates. |
Checking the Database User Permission | Check if you have the permission to create tables and insert data. You need to have the create, delete, alter, update, select, and insert permissions. |
Checking the Database Code | UTF-8 is recommended for MySQL5 database. The utf8mb4 code is not supported. If you are using ApsaraDB RDS for MySQL, you also need to check whether the engine is InnoDB. |
Plugin Problem
If the synchronization fails and any of the following three situations occurs, you can troubleshoot the following plugin problems.
Problem
(1) Duplicate plugin packages are found in the fanruan.log file.
(2) The Log Sync page displays blank content.
(2) The Log Sync page displays internationalization keys.
| Troubleshooting Step | Description |
|---|---|
| Checking Plugin Installation | Check whether the plugin has been installed repeatedly. If so, delete all the plugins, restart the server, and reinstall the plugin. |
Checking the WebSocket Port |
Common Error
If sync fails, you can check the fanruan.log file and locate the problem through error messages containing "com.fr.plugin.sync".
| Error Message | Error Message |
|---|---|
SyncLog: getConfigInstance failed!, retry to getConfigInstance! java.lang.AssertionError: com.fr.plugin.sync.config.status.IsStart has the same tableName with com.fr.plugin.sync.config.status.IsStart | (1) You may have installed different versions of the Syncing Platform Logs to Other Databases plugin. You need to completely delete all plugin files in the plugins folder, restart the server, and reinstall the plugin. (2) You may have installed both the original log data synchronization plugin and the Syncing Platform Logs to Other Databases plugin. You need to delete the log data synchronization plugin. |
Cannot set int field com.fr.plugin.sync.db.entity.Schedule.logType to java.lang.Long | LogDB does not support the int type. You need to change the int type to the long type. Update the plugin to 1.1.2 or later versions. |
SyncLog: get logs failed from XXX | The XXX table being synced is too large, causing Swift to terminate the query, which leads to query failure. The Syncing Platform Logs to Other Databases plugin cannot support the sync of particularly large amounts of log data. You are advised to clean up the table data appropriately before the sync. |
SyncLog: sync failed: com.fr.third.org.hibernate.StaleStateException: Batch update returned unexpected row count from update [0]; actual row count: 0; expected: 1 | Hibernate occasionally fails to save data. Update the plugin to 1.0.8 or later versions. |
java.sql.SQLException: Incorrect string value: '/xE4/xB8/xB4/xE6/x97/xB6...' for column 'userrole' at row 1 | The character set of the database or table is incorrect. UTF-8 is recommended for MySQL5 database. The utf8mb4 code is not supported. If you are using ApsaraDB RDS for MySQL, you also need to check whether the engine is InnoDB. |
QueryTimeoutException- could not extract ResultSet | Uncommitted transactions may exist in the database, which leads to data submission failure. Check if any uncommitted transactions exist., If so, handle them before attempting to perform sync again. |
could not execute statement | Check the schema and encoding type of the database. UTF-8 is recommended for MySQL5 database. The utf8mb4 code is not supported. If you are using ApsaraDB RDS for MySQL, you also need to check whether the engine is InnoDB. Check whether the Oracle schema is set to thin. If so, change it to the username. |
