Data Agent allows local large language models (LLMs) and cloud-based LLMs that comply with OpenAI/Azure API specifications to be connected.
(1) Disclaimer: FanRuan only provides recommended LLMs and connection methods for reference and takes no liability for any issues with the LLMs themselves.
(2) The models recommended in this document are only for data query scenarios. The recommended models for report scenarios are still in the testing phase.
If you connect to Data Agent via public cloud LLM APIs hosted by model vendors or cloud service providers, you can refer to the following evaluation results for model selection.
This evaluation uses FanRuan's internal data query benchmark evaluation set to test 7 public cloud LLM APIs in a unified environment. The evaluation includes 200 unified use cases and mainly focuses on two metrics:
Accuracy score: It measures the accuracy of model responses.
Performance score: It measures the model response speed and waiting experience.
Usage Focus
Model Recommended to Be Preferentially Evaluated
Evaluation Performance
More focused on response accuracy
Kimi K3
Accuracy score 80.7, ranking 1st among the evaluated models
More focused on response speeds
DeepSeek-V4-Flash-0731 (Official Version)
Performance score 80.8, ranking 1st among the evaluated models
Balanced between accuracy and response performance
Qwen3.8-Max
Accuracy score 79.4, ranking 2nd
Performance score 75.0, ranking 3rd
The following list sorts models by accuracy score in the descending order. When accuracy scores are the same, the model with the higher performance score ranks higher.
Ranking
Model
Accuracy Score
Performance Score
Evaluation Grade
1
80.7
53.8
A
2
79.4
75.0
3
Qwen 3.8 27B
78.0
66.6
4
DeepSeek-V4-Pro-0813
77.5
47.9
5
76.5
80.8
6
Qwen3.7-Plus
73.4
77.9
B
7
GLM-5.2
71.4
50.5
8
DeepSeek-V4-Pro (Closed Beta)
68.5
72.7
C
The recommended models above all meet the basic usage requirements of Data Agent. If configuring other public cloud LLMs, enterprises are advised to estimate various usage quotas and confirm the following content with the model provider:
Procurement Item
Content to Confirm with the Model Provider
Impact on Actual Usage
Model and Version
Model name, version number, whether to upgrade automatically, and retention period for old versions
After a model version change, the response quality and API behavior may change.
QPM/RPM Quota
Number of allowed LLM invocations per minute, and whether burst traffic limits exist
If the quota is insufficient, rate limiting, queuing, or invocation failures may occur during peak hours.
TPM Quota
Number of input and output tokens allowed to be processed per minute, and whether inputs and outputs have separate quota limits
If the quota is insufficient, complex questions, long contexts, or high-concurrency invocations may fail.
Concurrency Limit
Number of invocations allowed to be processed at the same time, and queuing mechanism when the limit is exceeded
It affects response durations and timeout rates during peak business hours
Context Capability
Maximum context length and maximum output tokens supported per invocation
It affects the processing of complex questions, long prompts, and large-scale metadata.
Network and Security
API access address, network access method, whether data is retained, and related security compliance terms
It determines whether the model service can be successfully connected and whether enterprise security requirements are met.
Related concepts:
QPM (Queries Per Minute): Number of queries allowed to be completed per minute
RPM (Requests Per Minute): Number of API requests allowed to be initiated per minute
TPM (Tokens Per Minute): Number of tokens a model is allowed to process per minute, including both input and output tokens
Concurrency limit: Number of API invocations a model service is allowed to process simultaneously
When answering a user's question, Dora may need to invoke the LLM multiple times. Therefore, you cannot purchase API quotas directly based on the number of user questions.
You can use the following method for a preliminary estimation:
(1) Required invocations per minute
For example, if 10 user questions are asked per minute during peak business hours and Dora needs to invoke the LLM an average of 3 times to answer each question, the LLM is invoked for approximately 30 (10 × 3) times per minute.
In this case, the purchased QPM/RPM quota should be higher than 30 invocations per minute, with a margin reserved for traffic fluctuations, invocation retries, and future business growth.
(2) Required tokens per minute
For example, if the LLM is invoked 30 times per minute during peak hours and each invocation uses an average of 5,000 tokens, approximately 150,000 (30 × 5,000) TPM is required.
If the model service provider limits the input TPM and output TPM separately, you also need to calculate them separately based on actual invocation records, rather than just calculating the total value of input and output tokens.
(1) Accuracy Score
The ratio of passed use cases to evaluated use cases is converted to a score in a 100-point scale. Use cases that cannot be evaluated due to missing data or other reasons are not included in the denominator.
Calculation Formula: Accuracy Score = Number of Passed Cases ÷ Number of Evaluated Cases × 100
The following lists the grades corresponding to the accuracy scores.
Grade
S
Accuracy Score ≧ 85
75 ≤ Accuracy Score < 85
70 ≤ Accuracy Score < 75
60 ≤ Accuracy Score < 70
D
Accuracy Score < 60
(2) Performance Score
The performance score is calculated based on a weighted conversion of the overall response speed, tail latency, and the proportion of long-tail timeout. A higher score indicates a shorter overall wait time and fewer slow queries.
If you cannot transmit data to the public cloud, or need to control the model service and computing resources yourself, you can select private deployment.
Recommended LLM
GPU Requirement
Option One: 8 x H20-3e
Option Two: 8 x A800 (2 x TP4)
Enterprises can select from the following two solutions based on their existing hardware conditions, usage during peak hours, and budget range.
Enterprise Usage Scenario
Deployment Solution
Performance Reference
Enterprise users may use the system intensively in the same period, focusing more on peak processing capacity.
8 x H20-3e
About 20–25 background analysis tasks can be executed simultaneously.
It takes about 40–70 seconds to complete a task during peak hours.
The deployment budget is approximately $477,000 (for reference only).
Enterprises focus more on initial procurement costs and can accept moderate queuing during peak hours.
8 x A800 (2 x TP4)
About 10–14 background analysis tasks can be executed simultaneously.
It takes about 50–70 seconds to complete a task during peak hours.
The deployment budget is approximately $149,000–$290,000 (for reference only).
The following table lists the detailed performance of the two deployment solutions. Enterprises can select one according to the scenario-based recommendations in the previous section.
Peak Capacity Reference for a Single Service
Wait Time Near Capacity Limit
When Traffic Significantly Exceeds Capacity
Deployment Budget (for Reference Only)
18–30 tasks can be completed per minute.
About 40–70 seconds
It takes about 90–140 seconds to execute 50 tasks simultaneously.
About $477,000
About 9–14 tasks can be completed per minute.
About 50–70 seconds
It takes about 110–150 seconds to execute 28 tasks simultaneously.
About $149,000–$290,000
Supplementary Reference for Technical Personnel
The following metrics are primarily used by the deployment team for performance analysis and do not require special attention for general business selection.
Technical Metric
Single-Concurrency Latency
(Time required to complete one task when the service is idle)
About 8–13 seconds
About 18–24 seconds
Single-Concurrency QPM
(Number of tasks that can be completed per minute when the service is idle)
About 5–7 queries
About 2.5–3.3 queries
KV Cache
(Capacity of task contexts that can be stored simultaneously)
6.83 million tokens
About 1.42 million tokens
When processing tasks, the model needs to temporarily remember the prompt, context, and the content that has already been generated. KV Cache can be understood as a temporary workspace used by the model when ongoing tasks are processed, and its capacity is measured in tokens. As the basic unit used by an LLM to calculate text length, a token can be simply understood as a text fragment obtained after the LLM splits the text, but it is not equivalent to a fixed number of characters.
Generally, a larger capacity enables more tasks to be processed simultaneously or tasks with longer contexts to be handled. For example, the longer the context used by a task, the more KV Cache the task typically occupies, resulting in fewer tasks that can be accommodated at the same time.
KV Cache is not a hard drive space, nor is it the capacity for long-term storage of chat records. It also cannot be directly converted into the number of online users. The actual number of tasks that can be processed simultaneously is also subject to factors such as model deployment parameters, task length, VRAM allocation, and inference frameworks.
The final performance of the same hardware model may vary significantly according to different server configurations, inference frameworks, model precisions, parallel strategies, and service parameters.
When procuring a private deployment solution, you are advised to include the following content into the procurement or technical verification checklist:
(1) Model version: Specify the deployment of DeepSeek-V4-Flash-0731 (Official Version), and confirm the model files, licensing method, and upgrade strategy.
(2) Inference service: Confirm the inference framework, deployment parameters, and Azure-compatible API, and complete joint debugging with Data Agent.
(3) Capacity goal: Clarify acceptance metrics based on the estimated concurrency, peak QPM, question length, and target response time.
After deployment is completed according to the private solution, you are advised to verify at least the following content:
(1) Use actual business questions to verify the answer accuracy, rather than just verifying that the model API can return results.
(2) Test the QPM, average response time, tail latency, and timeout rate under the estimated peak concurrency.
(3) Use typical long questions to verify context capacity and VRAM usage.
(4) Simulate burst traffic exceeding the estimated capacity to confirm rate limiting, queuing, and recovery mechanisms.
(5) Complete the verification of model service restarts, node failures, surveillance alerts, and log troubleshooting.
The models or deployment combinations recommended in this document are backed by evaluation data or verification results, but this does not mean that Data Agent can only connect to the recommended models. You can connect to other models that comply with API specifications. But for non-recommended models, FanRuan does not guarantee their compatibility, answer quality, response performance, or running stability.
Using non-recommended models may result in the following situations:
(1) API fields, authentication methods, or response formats are incompatible, resulting in model connection or invocation failures.
(2) Insufficient capabilities in model instruction understanding, tool invocation, or structured output lead to incorrect answers, unstable results, or incomplete tasks.
(3) Insufficient context window, QPM, TPM, or concurrency quotas cause rate limiting, timeouts, queuing, or excessively long responses.
(4) Capabilities or API behavior have changed after model version upgrades, resulting in a decline in original business performance.
(5) Insufficient private hardware resources or deployment parameters lead to insufficient VRAM, low throughput, abnormal services, or increased capacity expansion costs.