Data Factory Dataset

Version

Application Scenario

Enterprise data sources are in various formats, among which some data formats cannot be directly applied to FineReport. This plugin provides data format conversion capabilities to connect to JSON data sources/WebService/web crawlers/self-developed data centers, and convert data into types that can be used by FineReport. This plugin also has high scalability to adapt to more complex scenarios through secondary development.

Note:

1. To extend development based on this plugin, you need to have the most basic FineReport plugin development capabilities.

2. To perform upgrade using sub-plugins, you need to disable the sub-plugins first or restart the server after the upgrade.

Function Description

The plugin can be used to personalize data assembly through different loaders and parsers, converting the externally-obtained data into data of formats supported by FineReport.

Plugin Installation

You can download the plugin Data Factory Dataset.

For details about how to install the plugin in the designer, see Designer Plugin Management.

For details about how to install the plugin on the server, see Server Plugin Management.

Procedure

1. Install the plugin. Then a data factory dataset is added on the dataset management panel, as shown in the following figure.

2. Click Data Factory to enter the main data factory panel. The main panel can handle major scenarios. The following describes specific functions of the main panel.

(1) Name: You can set the dataset name here.

(2) Loader/Parser: The loader is responsible for loading data from the data source, while the parser is responsible for converting the loaded data into the data model supported by FineReport.

The plugin has two built-in load methods (HTTP and Single Parameter) and three parse methods (JSON, JS Parser, and Raw Data).

Select a load method and parse method from the drop-down list. For details about the specific loader/parser configuration methods, see section "Loader/Parser Use Example."

(3) Advanced Setting: When the built-in loader and parser cannot meet user requirements, the Advanced Setting function can be used for secondary development to achieve function expansion. For details, see section “Advanced Setting Description."

(4) Cache: Externally-read data can be temporarily stored in the cache. In this case, data can be directly retrieved from the cache instead of interfaces, improving the data reading speed. Cache can be set to None, Disk, or Memory.

• If None is selected, the cache function is disabled.

• If Disk is selected, data is cached to the local disk.

• If Memory is selected, data is cached to the memory.

(5) N Minute(s): Within N minutes, static data is read from the cache. After N minutes, data is read from interfaces again. The newly-read data will still be saved in the cache for N minutes.

Parameter: The parameters in the Parameter column will be divided into two types in the background.

• The first type is the parameters (${Parameter name}) that have appeared on the loader page, which can be automatically recognized by the parameter panel and parsed into the string configured in the loader. In terms of the parameter syntax, parameter values can be obtained by ${Parameter name}. Parameters in various formats such as texts and formulas are supported. Parameters can also be read from reports.

• The second type is the parameters that have not appeared on the loader page before. You can add parameters as request      parameters through the insert button on the parameter panel.

On the parameter panel, the Parameter column records the parameter names. The Value column allows you to select the parameter data types (including the string, integer, double, date, boolean, and formula) by clicking the button and enter the parameter values. The  icons allow you to delete, move up, and move down respectively the rows where the parameters are located.

(6) Preview: You can click the preview button to preview the data after format conversion.

3. Complete the configuration. Then in the FineReport designer, the new data factory dataset appears in the template dataset and can be directly applied to report production.

Built-in Loader

The plugin has two built-in load methods, which are:

• HTTP: Data (for example, JSON services) is loaded from the web by configuring related requests.

• Single Parameter: Data parameters (for example, JSON data, XML data, and other structured data) on the loader page can be originally passed after calculation to the parser, which is mainly for debug purposes. 

HTTP

When you set Load Method to HTTP, the loader content is in JSON format, which contains four fields described in the following table.

The following table describes a specific use example.

{
   url:"http://example.com/",
   type:"GET",
   charset:"UTF-8",
   header:{
       aa:${bb}
       }
  }

{
   url:"http://example.com/",
   type:"POST",
   charset:"UTF-8",
   header:{
       aa:${bb}
      }
  }

{
   url:"http://example.com/",
   type:"JSON",
   charset:"UTF-8",
   header:{
       aa:${bb}
      }
  }

{
   url:"http://example.com/",
   type:"JSON",
   charset:"UTF-8",
   header:{
       aa:${bb}
      },
   body:{
      cc:${dd}
      }
  }

Note:

The http://example.com/ in the example is for demonstration only and needs to be adjusted to the actual interface.

In the 2.5.24 version, GET requests can pass parameters through the body (not applicable to POST requests). The following table decribes the details.

When the request type is GET, the Content-Type parameter is provided to be used with the body.

{
      "url": "http://example.com",
      "type": "GET",
      "charset":   "UTF-8",
      "Content-Type":   "application/x-www-form-urlencoded",
      "header": {},
      "body": {
          "age":   "123"
      }
  }

Single Parameter

Example

If you select Single Parameter from the Load Method drop-down list, enter ${data}${today()} in the loader content box, and select Raw Data from the Parse Method drop-down list (indicating that data is returned originally, which will be explained in section “Raw Data"),  configure the parameters on the parameter panel as follows:

Parameter      Value
data               myData

Click the preview button to get the following results:

Built-in Parser

The plugin has three built-in parse methods, which are:

• JSON: Structured JSON data is parsed to generate the data model supported by FineReport.

Note:

Only JSON data for describing structures can be parsed. JSON data for describing services cannot be parsed. Service JSON data parse depends on the specific service description, which requires a separate parser implementation.

• JS Parser: Data is parsed through JS scripts to generate the data model supported by FineReport.

Note:

Use this parser with caution when dealing with large amounts of data.

• Raw Data: Raw data is returned, which is mainly for debug purposes.

JSON

The JSONPath syntax provided here is not complete. The following defines each field:

• dataPath: The default value is root (processing all data). If the value root.key1.key2.arr[1] is used, data in { key1 : { key2 : { arr:[ {},{<here>} ] }} } is located. In this case, only the JSON content in the specified path is parsed.

• showmap: Column name mapping is provided to convert the original column name (path1) to a custom column name (value1).

JS Parser

According to the JS syntax in the JSON dataset plugin, the $data variable specifies the received raw data. Finally, a data table object as follows needs to be generated through JS.

In the example of the returned object, column specifies column names and content specifies two-dimensional arrays to store the values in the dataset cells.

{
    "column":["col1", "col2", "col3"],
    "content":[
        [1, 2, 3],
        [4, 5, 6]
    ]
}

Example: (Develop the specific code according to the actual scenario.)

Data returned by the loader is as follows:

{
    "monthFactoryWaitingRateDataList": [
        {
            "factory": "HZC01",
            "dayFactoryWaitingRateList": [0,0,0],
            "days": ["2023-04-01","2023-04-02","2023-04-03"],
            "dayStrs": ["04-01","04-02","04-03"]
        },
        {
            "factory": "HZC02",
            "dayFactoryWaitingRateList": [0,0,0],
            "days": ["2023-04-01","2023-04-02","2023-04-03"],
            "dayStrs": ["04-01","04-02","04-03"]
        }
    ]
}

JS parser data entry example:

var x =$data;

var column = Object.keys(x.monthFactoryWaitingRateDataList[0]);
var content = [];

for (var i = 0; i < x.monthFactoryWaitingRateDataList.length; i++) {
    for (var j = 0; j < x.monthFactoryWaitingRateDataList[i].dayFactoryWaitingRateList.length; j++) {
        var contentLine = [];
        contentLine.push(x.monthFactoryWaitingRateDataList[i].factory, x.monthFactoryWaitingRateDataList[i].dayFactoryWaitingRateList[j], x.monthFactoryWaitingRateDataList[i].days[j], x.monthFactoryWaitingRateDataList[i].dayStrs[j]);
        content.push(contentLine);
    }
}

var result = {};
result.column = column;
result.content = content;
return result;

Preview data. The following figure shows the effect.

The JS parser has relatively low performance and needs to be used with caution when dealing with large amounts of data.

Note:

The plugin of the 2.5.13 version has started using the built-in Nashorn engine in JDK for the JS parser. The ES5 syntax is supported by default and the higher-version syntax is not supported currently.

Raw Data

Similar to the single-parameter loader, this method returns a dataset with only one cell. The dataset content is the raw data received from the loader, and the parameter panel is meaningless. For details, see section "Single Parameter."

Use Case

Case One

The FanRuan plugin list needs to be obtained.

Load Method: HTTP

Set Load Method to HTTP to load data from the web.

{
    url:"http://localhost:8075/webroot/FanRuan Plugin List.json",
    type:"GET",
    charset:"UTF-8",
    header:{}
}

Note:

1. The specific loader logic can be customized separately, and there is no unified writing specification.

2. The following figure shows the JSON data actually returned by the interface. (The figure here is for comparison purposes only. This step will not appear during actual plugin use.)

Parse Method: JSON

Set Parse Method to JSON to parse structured JSON data.

{
    dataPath:"root.result",
    showmap:"path1,value1,path2,value2",
}

The following figure shows the main panel configuration.

Click the preview button. The following figure shows the data results after format conversion.

Case Two

Assume that the interface (http://example.com?city=New York) for obtaining the weather of a city exists.

If you want to dynamically retrieve city data based on the template widget content, you can set a widget called city in the template.

Then perform configuration on the data factory panel as follows.

Load Method: HTTP

Method 1: Add the parameter with the same name in the loader and leave the default parameter value empty.

{
    url:"http://example.com?city=${city}",
    type:"GET",
    charset:"UTF-8",
    header:{}
}

Method 2: Manually add the city parameter on the parameter panel and leave the parameter value empty.

{
    url:"http://example.com",
    type:"GET",
    charset:"UTF-8",
    header:{}
}

Assume that the interface returns the following values:

{
    result:[
        {
            district:Manhattan,
            temperature:20,
        },
        {
            district:Brooklyn,
            temperature:23,
        }
    ]
}

Parse Method: JSON

{
    dataPath:"root.result",
    showmap:"district,temperature"
}

The following figure shows the main panel configuration.

Click the preview button. The following figure shows the data results after format conversion.

To use the advanced setting function of the main panel, you must have certain plugin development abilities.

If you think that the built-in load and parse methods are not sufficient to meet your needs, but do not want to develop a new loader/parser, you can click the Advanced Setting button on the main panel of the data factory to customize auxiliary steps.

The Advanced Setting panel is used to add pre-load, post-load, and post-parse events as supplementary load and parse methods, allowing you to add multiple events with same steps.

Processing Flow

On the Advanced Setting panel, you can select Set for Template Separately, Use Template Setting, or Use Global Setting from the top drop-down list.

If you select Set for Template Separately, a list will appear below for you to add, delete, and modify events.

If you select Use Template Setting or Use Global Setting, no list will appear below. To edit specific setting items, click Global Setting or Template Setting on the right.

Function/Interface Description of Each Event

Currently, each event does not have a specific application implemented and provides only an access framework. The advanced steps are enhancement and supplement of load/parse modes and used with development.

Pre-load Event

Additional parameter passing is provided. For example, the token acquisition logic can be written in the custom pre-load event. Multiple pre-load event parameters can be accumulated. The latter will overwrite the former for duplicate parameters.

Pre-load events are used on the main panel with the same method as that of invoking regular parameters. The default values in the Parameter column on the right are invalid.

Scene example: To obtain a token, you can develop a pre-load event for token acquisition, and introduce the obtained token as a parameter into the main logic.

Preprocessor Interface Description for Pre-load Events:

package com.tptj.plugin.hg.fun;
 
import com.fr.script.Calculator;
import com.fr.stable.ParameterProvider;
 
/**
 * Preprocessing stage
 */
public interface Preprocessor extends Configuration {
    String XML_TAG = "TableDataPreprocessor";
    int CURRENT_LEVEL = 1;
 
    /**
     * Preprocessing  Parameter changeable
     * @param calculator Operator
     * @param config Panel configuration content
     * @return Injected parameter
     * @throws Exception
     */
    ParameterProvider[] process(Calculator calculator, String config) throws Exception;
}

Post-load Event

Raw data obtained from the loader can be collated. Generally, the loader returns data of the string type. At this stage, you can additionally collate data. Multiple events are executed in the top-to-bottom order.

Scene example: If data returned in the JSON format is not suitable to be directly parsed using the JSON parser, or parsed using the JS parser with relatively poor performance, you can customize a post-load event to format the structure into that supported by the JSON parser.

Formatter Interface Description for Post-load Events:

package com.tptj.plugin.hg.fun;
 
public interface Formatter extends Configuration{
 
    String XML_TAG = "TableDataFormatter";
 
    int CURRENT_LEVEL = 1;
 
    /**
     * Data formatting
     * @param data Data returned by the loader
     * @param config Configuration
     * @return Formatted data
     * @throws Exception
     */
    Object format(Object data, String config) throws Exception;
} 

Post-parse Event

The dataset returned by the parser can be further processed. For example, you can add/delete columns, rename columns, re-sort data, and filter conditions. At this stage, you can additionally collate data. Multiple events are executed in the top-to-bottom order.

Scene example: You can freely operate dataset content. For example, you can sort data, filter data, take the top n items, take odd-numbered items, and adjust data formats.

Filter Interface Description for Post-parse Events:

package com.tptj.plugin.hg.fun;
 
import com.tptj.plugin.hg.stable.SimpleDataModel;
 
/**
 *
 */
public interface Filter extends Configuration {
 
    String XML_TAG = "TableDataFilter";
 
    int CURRENT_LEVEL = 1;
 
    /**
     * Data collation
     * @param dataModel Data model
     * @param config Configuration
     * @return Collated data
     * @throws Exception
     */
    SimpleDataModel doAction(SimpleDataModel dataModel, String config) throws Exception;
 
}

Description of Other Interfaces

Loader Interface Description for Load Events

package com.tptj.plugin.hg.fun;
 
import com.fr.script.Calculator;
import com.fr.stable.ParameterProvider;
import com.fr.stable.fun.mark.Mutable;
 
public interface Loader extends Mutable {
    
    String XML_TAG = "TableDataLoader";
 
    int CURRENT_LEVEL = 1;
    
    /**
     * Data load
     * @param cal   Current operator
     * @param params    Required parameter
     * @param others        Other configuration items that you may use but do not want to be controled through parameters, namely, loader panel content
     * @return  Object storing data
     */
    Object load( Calculator cal, ParameterProvider[] params, String others);
    
    /**
     * Loader name [displayed in the drop-down list on the data factory configuration page], which is unique and supports internationalization keys
     * @return
     */
    String getName();
    
    /**
     *
     * @return Configuration displayed by default [custom configuration items displayed in the loader configuration text domain on the data factory configuration page, namely, parts that you do not want to be hard-coded or tampered with by parameters]
     */
    String getDefaultConfig();
}

Resolver Interface Description for Parse Events

package com.tptj.plugin.hg.fun;
 
import com.fr.stable.fun.mark.Mutable;
import com.tptj.plugin.hg.stable.SimpleDataModel;
 
public interface Resolver extends Mutable {
    
    String XML_TAG = "TableDataResolver";
 
    int CURRENT_LEVEL = 1;
    
    /**
     * Data parse
     * @param data Raw data obtained from the loader
     * @param others  Configuration that is used during parse and not expected to be parameters, namely, parser panel content
     * @return Parsed two-dimensional data model
     */
    SimpleDataModel parse(Object data, String others );
    
    /**
     * Parser name [displayed in the drop-down list on the data factory configuration page], which is unique and supports internationalization keys
     * @return
     */
    String getName();
     
    /**
     *
     * @return Configuration displayed by default [custom configuration items displayed in the loader configuration text domain on the data factory configuration page, namely, parts that you do not want to be hard-coded or tampered with by parameters]
     */
    String getDefaultConfig();
    
}

Function Point Registration

   <extra-core>
       <TableDataLoader class="com.fr.plugin.tptj.tabledata.factory.demo.DemoLoader"/>
       <TableDataResolver class="com.fr.plugin.tptj.tabledata.factory.demo.DemoResolver"/>
       <TableDataPreprocessor class="com.fr.plugin.tptj.tabledata.factory.demo.DemoPreprocessor"/>
       <TableDataFormatter class="com.fr.plugin.tptj.tabledata.factory.demo.DemoFormatter"/>
       <TableDataFilter class="com.fr.plugin.tptj.tabledata.factory.demo.DemoFilter"/>
   </extra-core>
<dependence>
       <Item key="com.tptj.plugin.hg.tabledata.factory.v10" type="plugin"/>  <!-- Depending on the main framework of the data factory -->
 </dependence>

Abstract classes are usually inherited for development.

package com.fr.plugin.tptj.tabledata.factory.demo;
 
import com.fr.base.Parameter;
import com.fr.json.JSONObject;
import com.fr.script.Calculator;
import com.fr.stable.ParameterProvider;
import com.tptj.plugin.hg.impl.AbstractPreprocessor;
 
public class DemoPreprocessor extends AbstractPreprocessor {
    @Override
    public ParameterProvider[] process(Calculator calculator, String s) {
 
        JSONObject jo = new JSONObject(s);
        ParameterProvider[] results = new ParameterProvider[1];
        results[0] = new Parameter("token", jo.get("token"));
        //jo.get("token");
        return results;
    }
 
    @Override
    public String getName() {
        return "Demo_PreProcess";
    }
 
    @Override
    public String getDefaultConfig() {
        return "{\n" +
                "   token:token\n" +
                "}";
    }
}

package com.fr.plugin.tptj.tabledata.factory.demo;
 
import com.tptj.plugin.hg.impl.AbstractFormatter;
 
public class DemoFormatter extends AbstractFormatter {
    @Override
    public Object format(Object o, String s) {
        String data = (String) o;
        String[] lines = data.split("\n");
        StringBuilder result = new StringBuilder();
        for (int i = 1; i < lines.length; i++) {
            result.append(lines[i]).append("\n");
        }
        return result.toString();
    }
 
    @Override
    public String getName() {
        return "Demo_Formatter";
    }
 
    @Override
    public String getDefaultConfig() {
        return "remove first line";
    }
}

package com.fr.plugin.tptj.tabledata.factory.demo;
 
import com.fr.general.data.TableDataException;
import com.fr.log.FineLoggerFactory;
import com.tptj.plugin.hg.impl.AbstractFilter;
import com.tptj.plugin.hg.stable.SimpleDataModel;
 
import java.util.ArrayList;
import java.util.List;
 
public class DemoFilter extends AbstractFilter {
    @Override
    public SimpleDataModel doAction(SimpleDataModel simpleDataModel, String s) {
        try {
            simpleDataModel.removeColumn(simpleDataModel.getColumnCount() - 1);
            List<Object[]> newData = new ArrayList<Object[]>();
            for(Object[] row : simpleDataModel.getDatas()) {
                Object[] newRow = new Object[row.length - 1];
                System.arraycopy(row, 0, newRow, 0, row.length - 1);
                newData.add(newRow);
            }
            simpleDataModel.setDatas(newData);
            //simpleDataModel.getDatas().remove(simpleDataModel.getColumnCount() - 1);
        } catch (TableDataException e) {
            FineLoggerFactory.getLogger().error(e, "{}", e.getMessage());
        }
        return simpleDataModel;
    }
 
    @Override
    public String getName() {
        return "Demo_Filter";
    }
 
    @Override
    public String getDefaultConfig() {
        return "remove last col";
    }
}

package com.tptj.plugin.hg.tabledata.factory;
 
import com.tptj.plugin.hg.impl.AbstractResolver;
import com.tptj.plugin.hg.stable.SimpleDataModel;
 
public class DemoResolver extends AbstractResolver {
 
    @Override
    public SimpleDataModel parse(Object jsonStr, String others) {
        return new SimpleDataModel();
    }
 
    public static final String KEY = "Demo";
    @Override
    public String getName() {
        return KEY;
    }
 
    @Override
    public String getDefaultConfig() {
 
        return "Is a demo";
    }
 
}

package com.tptj.plugin.hg.tabledata.factory.core.loader;

 
import com.fr.json.JSONObject;
import com.fr.script.Calculator;
import com.fr.stable.ParameterProvider;
import com.tptj.plugin.hg.impl.AbstractLoader;
 
public class SingleParamLoader extends AbstractLoader {
 
    @Override
    public Object load( Calculator calculator, ParameterProvider[] parameters, String others ) {
        try{
            JSONObject obj = new JSONObject(others);
            if( obj.has("data") ){
                return obj.optString("data", "{}");
            }
        }catch(Exception e){
            e.printStackTrace();
        }
        return "";
    }
    public static final String KEY = "Plugin-Factory_Data_Set_Loader_Type_Single_Parameter";
    @Override
    public String getName() {
        return KEY;
    }
 
    @Override
    public String getDefaultConfig() {
        StringBuilder sb = new StringBuilder();
        sb.append("{\r\n")
        .append("    data:\"JSON string to be parsed, supporting parameters and formulas such as ${p1}\",\r\n")
        .append("}\r\n");
        return sb.toString();
    }
 
}

SimpleDataModel inherits the standard dataset format. The data storage structure consists of a two-dimensional table called datas and a one-dimensional table called cols, which specifies the
content and column names respectively. To write data to the dataset, you can invoke the corresponding set and add methods.
You can customize panel interfaces. In many cases, the default text box is too rudimentary for configuration. If you want a richer UI, you can use this interface to replace the default text box.

package com.tptj.plugin.hg.fun;
 
import com.fr.stable.fun.mark.Mutable;
 
import javax.swing.*;
import java.awt.event.FocusListener;
 
public interface ConfigTable<T extends Configuration> extends Mutable {
 
    String XML_TAG = "TableDataConfigTable";
 
    int CURRENT_LEVEL = 1;
 
    String getName();
 
    /**
     * Writing data to JPanal
     * @param config
     */
    void setValue(String config);
 
    /**
     * Reading data to
     * @return
     */
    String getValue();
 
    JPanel getTable(FocusListener listener);
}

Abstract classes can be inherited during use.

package com.tptj.plugin.hg.fun;

import com.fr.stable.fun.mark.Mutable;

import javax.swing.*;
import java.awt.event.FocusListener;

/**
 * Binding Configuration through generic inheritance
 * @param <T>
 */
public interface ConfigTable<T extends Configuration> extends Mutable {


    String XML_TAG = "TableDataConfigTable";

    int CURRENT_LEVEL = 1;

    String getName();

    /**
     * Writing data to JPanal
     * @param config
     */
    void setValue(String config);

    /**
     * Reading data to
     * @return
     */
    String getValue();

    /**
     * Obtaining the configuration panel
     * @deprecated use {@link ConfigTable#getTable(ParameterRefreshAction action)} instead
     * FocusListener triggering conditons are limited, which cannot meet all needs. You are advised to manually bind events using ParameterRefreshAction.
     * @param listener FocusListener Event callback
     * @return
     */
    JPanel getTable(FocusListener listener);
    /**
     * Obtaining the configuration panel
     * @param action Obtaining the refresh event through {@link ParameterRefreshAction#doAction()} and binding the event manually
     * @return
     */
    JPanel getTable(ParameterRefreshAction action);

}

Function point registration
<extra-designer>
      <TableDataConfigTable class="com.tptj.plugin.hg.tabledata.factory.core.filter.DefaultFilterTextTable"/>
  </extra-designer> 

If you suspect that the slow execution of data factory datasets causes long template loading, you can analyze the execution periods of the data factory datasets through logs, including the loader execution period, parser execution period, and total execution period of each data factory dataset.

Test step:

Adjust the log level to the debug level, preview the template or dataset, and search for the following logs in fanruan.log.
(1) table data factory eval start ... (Meaning: The data factory dataset starts to be executed.)
(2) start loading data, config is (Meaning: The loader starts to be executed.)
(3) loading data end, cost (Meaning: The loader execution ends.)
(4) start json resolving data ... (Meaning: The JSON parser starts to be executed.)
(5) resolving data end, cost (Meaning: The parser execution ends.)
(6) table data factory eval end, cost (Meaning: The data factory execution ends.)

The logs show the overall data execution periods of the loader, parser, and data factory datasets. If the loader execution period is too long, it is usually because the response period of the invoked interface is too long. You can use other methods to invoke the interface and check the response period to locate the problem.

Note:

The plugin version must be at least 2.5.21.