Volume limit of self-service dataset

1.1 Version

1.2 Reason for the volume limit of dataset

When using FineBI, it is necessary to avoid the problem that the user's unreasonable operation may affect other users or the system's performance stability.

When creating self-service dataset, FineBI provides data volume limits to avoid such risks and ensure the proper use of FineBI.

Data volume limits for two types of self-service dataset.

Data volume limits for two self-service data sets at the time of real-time data.

Note: When the user's self-service data set exceeds the data volume limit, a prompt will be given on the front-end page, and the data volume can be reduced by reducing the number of fields in the table, filtering, grouping, and aggregation. If the user uses the extracted data and is a processing user, you can directly switch to the "processing self-service dataset" without a data volume limit.

3.1 Row limits for self-service dataset

3.1.1 Problem Phenomenon

Note: It only works for extracted data, real-time data is not restricted.

For example, if you set the generated rows limit to 1 million, the preview reports an error as shown below.

The error is reported in the background as shown below.

3.1.2 Solution

Reduce the amount of data that needs to be updated in a self-service dataset, such as reducing fields, continuing to filter the data, or group summary the data.

3.2 Cell limits for self-service dataset

3.2.1 Problem Phenomenon

Note: Only works on extracted data, real-time data is not restricted.

If you set the cell limit to 1 million, an error is reported when generating the data, and the preview error is shown below.

The error is reported in the background as shown below.

3.2.2 Solution

• Reduce the amount of data that needs to be updated, such as by reducing fields, continuing to filter the data, or grouping the data for aggregation.

• Click"Manage>System>General>Spider Parameter", modify "Analyze user self-dataset disk footprint size"parameter, as shown below.

Note: Use the second method with caution.

3.3 Memory limits for self-service dataset

Note: You need to check the logs to determine the memory usage size.

3.3.1 Exceeded limits were detected before the self-service dataset was calculated

For example, if the memory is set to 900000 and the number of rows is 1.2 million, if the rows exceed the memory limit, the prediction will exceed the limit, and it will be able to detect that the limit is exceeded before the calculation, without the actual memory consumed.

The preview screen reports an error as shown below：

The error is reported in the background as shown below.

3.3.2 Self-service dataset show exceeding limits during calculation

During the calculation it shows that the limit is exceeded, then the background logs will give the actual memory consumed and the preview screen reports an error as shown below.

The error is reported in the background as shown below.

Note: For extracting data using part of the data, you need to check the log to determine the size of the memory occupation, the judgment method is the same as above, assuming that the memory is set very small, then there is a possibility that the use of part of the data exceeds the limit, the use of all data does not exceed the limit of the situation.

3.3.3 Solution

• Reduce self-service dataset steps.

• Reduce fields, do not use all data.

• Continue to filter the data, etc.

3.4 Left and right merge 10 million rows limit

3.4.1 Problem Phenomenon

Note: Both real-time data and extracted data are subject to restrictions.

For example, the preview screen when the actual left and right merge results are exceeded is shown in the following figure.

The error is reported in the background as shown below.

3.4.2 Solution

Reduce the amount of data in the tables used, such as filtering before trying to merge.

3.5 Group summary 1 million cells limit

3.5.1 Problem Phenomenon

Note: Both real-time data and extracted data are subject to restrictions.

A preview of the screen when the actual grouped summary merge results exceed 10 million cells is shown below.

The error is reported in the background as shown below.

3.5.2 Solution

Reduce the amount of data in that result, such as reducing fields, filtering before trying to aggregate.

4. Analysis using ultra-limited self-service dataset

4.1 Scope of application

Real-time data

4.2 Making a self-service dataset using an over-constrained self-service dataset

4.2.1 Problem Phenomenon

1）After using a self-service dataset with an overlimit to make a self-service dataset and select field, the preview area is displayed as "Hint of source table overlimit".

2）In the left and right merge popup the table with the overlimit selected is shown in the preview area as the source table overlimit.

3）If you select a table that exceeds the limit when merges up and down, then the preview area shows detailed information about the merged table exceeding the limit.

4.2.2 Solution

The amount of data in the self-help dataset was adjusted as described in Section 3 of this paper.

4.3 Make dashboards with ultra-limited self-service dataset

4.3.1 Problem Phenomenon

The self-service dataset using overlimit at the dashboard can be selected at component selection, but the preview area on the right side shows "Hints for source table overlimit".

4.3.2 Solution

The amount of data in the self-help dataset was adjusted as described in Section 3 of this paper.