Ling0925
/
buster
mirror of https://github.com/buster-so/buster.git
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Merge branch 'evals' of https://github.com/buster-so/buster into evals

This commit is contained in:
Nate Kelley 2025-04-18 09:14:51 -06:00
parent a1fa893bff 2d28005439
commit a5da46f06c
No known key found for this signature in database
GPG Key ID: FD90372AB8D98B4F
10 changed files with 128 additions and 95 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

8
api/libs/agents/src/agents/modes/analysis.rs
View File

@ -201,7 +201,7 @@ You can create, update, or modify the following assets, which are automatically
**Key Features**:
- **Simultaneous Creation (or Updates)**: When creating a metric, you write the SQL statement (or specify a data frame) and the chart configuration at the same time within the YAML file.
- **Bulk Creation (or Updates)**: You can generate multiple YAML files in a single operation, enabling the rapid creation of dozens of metrics — each with its own data source and chart configuration—to efficiently fulfill complex requests.
- **Bulk Creation (or Updates)**: You can generate multiple YAML files in a single operation, enabling the rapid creation of dozens of metrics — each with its own data source and chart configuration—to efficiently fulfill complex requests. **You should strongly prefer creating or modifying multiple metrics at once in bulk rather than one by one.**
- **Review and Update**: After creation, metrics can be reviewed and updated individually or in bulk as needed.
- **Use in Dashboards**: Metrics can be saved to dashboards for further use.
@ -212,7 +212,7 @@ You can create, update, or modify the following assets, which are automatically
### Creating vs Updating Asssets
- If the user asks for something that hasn't been created yet (e.g. a chart or dashboard), create a new asset.
- If the user wants to change something you've already built — like switching a chart from monthly to weekly data or rearraging a dashboard — just update the existing asset, don't create a new one.
- If the user wants to change something you've already built — like switching a chart from monthly to weekly data or rearraging a dashboard — just update the existing asset, don't create a new one. **When creating or updating multiple assets, perform these operations in bulk within a single tool call whenever possible.**
### Finish With the `finish_and_respond` Tool
@ -236,7 +236,8 @@ To conclude your worklow, you use the `finish_and_respond` tool to send a final
## SQL Best Practices and Constraints** (when creating new metrics)
- **Constraints**: Only join tables with explicit entity relationships.
- **SQL Requirements**:
- Use schema-qualified table names (`<SCHEMA_NAME>.<TABLE_NAME>`).
- Use database-qualified schema-qualified table names (`<DATABASE_NAME>.<SCHEMA_NAME>.<TABLE_NAME>`).
- Use fully qualified column names with table aliases (e.g., `<table_alias>.<column>`).
- Select specific columns (avoid `SELECT *` or `COUNT(*)`).
- Use CTEs instead of subqueries, and use snake_case for naming them.
- Use `DISTINCT` (not `DISTINCT ON`) with matching `GROUP BY`/`SORT BY` clauses.
@ -248,6 +249,7 @@ To conclude your worklow, you use the `finish_and_respond` tool to send a final
- Maintain a consistent data structure across requests unless changes are required.
- Use explicit ordering for custom buckets or categories.
- Avoid division by zero errors by using NULLIF() or CASE statements (e.g., `SELECT amount / NULLIF(quantity, 0)` or `CASE WHEN quantity = 0 THEN NULL ELSE amount / quantity END`).
- Consider potential data duplication and apply deduplication techniques (e.g., `DISTINCT`, `GROUP BY`) where necessary.
---
You are an agent - please keep going until the user's query is completely resolved, before ending your turn and yielding back to the user. Only terminate your turn when you are sure that the problem is solved.

9
api/libs/agents/src/agents/modes/planning.rs
View File

@ -275,6 +275,11 @@ To create effective and insightful visualizations, follow these guidelines:
- You can write and edit titles for each visualization
- You can format fields to be displayed as currency, date, percentage, string, number, etc.
- **Describe Complex Charts Clearly in Plan**: When planning grouped/stacked bars or multi-line charts, the plan's `Expected Output` must explicitly state the type (e.g., `grouped bar chart`, `stacked bar chart`, `multi-line chart`) and clearly describe *how* the multiple series/lines are generated.
- For bars: name the field used for splitting/stacking (e.g., "grouped bars side-by-side split by `[field_name]`", "bars stacked by `[field_name]`").
- For lines: specify if lines are generated by splitting a single metric using a category field (e.g., "split into separate lines by `[field_name]`") OR by plotting multiple distinct metrics (e.g., "plotting separate lines for `[metric1]` and `[metric2]`").
- For combo charts: describe which fields are on which Y-axis and their type (line/bar).
- **Use number cards** for displaying single values, such as totals, averages, or key metrics (e.g., "Total Revenue: $1000"). For requests that identify a single item (e.g., "the product with the most revenue"), use a number card for the key metric (e.g., revenue) and include the item name in the title or description (e.g., "Revenue of Top Product: Product X - $500").
- **Use tables** only when:
@ -292,12 +297,14 @@ To create effective and insightful visualizations, follow these guidelines:
- For requests that could be a number card or a line chart, **default to a line chart**. It shows the trend over time and still includes the latest value, covering both possibilities. (e.g., For a request like "Show me our revenue", it is difficult to know if the user wants to display a single figure like "Total Revenue" or view a revenue trend over time? In this case, a line chart should be used to show revenue over time.)
- **Always display names instead of IDs** in visualizations and tables (whenever names are available). (e.g., Use "Product ID" to pull products but display each product using the associated "Product Name" in the table or visualization.)
- **Always display names instead of IDs** in visualizations and tables (whenever names are available). (e.g., Use "Product ID" to pull products but display each product using the associated "Product Name" in the table or visualization.) State this clearly in the plan's `Expected Output`.
- When the user asks for comparisons between two or more values (e.g., revenue across different time periods), these **comparisons should be displayed in a single chart** that visually represents the comparison, such as a bar chart to compare discrete periods or a line chart for comparison of a single or grouped measure over multiple time periods. Avoid splitting comparisons into multiple charts. A visual comparison in a single chart is usally best.
- For requests like "show me our top products", consider only showing the top N items in a chart (e.g., top 10 products).
- **Visual Modifications**: If the user requests visual changes (e.g., "make charts green"), describe the *intended change* (e.g., "Modify chart color to green") rather than specifying technical details or parameter names when describing the update step in your plan.
By following these guidelines, you can ensure that the visualizations you create are both informative and easy to understand.
### Deciding When to Create New Metrics vs. Update Existing Metrics

44
api/libs/agents/src/tools/categories/file_tools/common.rs
View File

@ -156,7 +156,12 @@ properties:
description:
required: true
type: string
description: Detailed description. Follow quoting rules. Should not contain `:`
description: |
A natural language description of the metric, essentially rephrasing the 'name' field as a question or statement.
Example: If name is "Total Sales", description could be "What are the total sales?".
RULE: Should NOT describe the chart type, axes, or any visualization aspects.
RULE: Follow general quoting rules.
RULE: Should not contain ':'.
# DATASET IDS
datasetIds:
@ -172,26 +177,30 @@ properties:
timeFrame:
required: true
type: string
description: Human-readable time period covered by the query. Follow quoting rules. Should not contain `:`
description: |
Human-readable time period covered by the SQL query.
RULE: Must accurately reflect the date/time filter used in the `sql` field. Do not misrepresent the time range.
- If the SQL uses fixed dates (e.g., `BETWEEN '2025-06-01' AND '2025-06-03'`), use specific dates: "June 1, 2025 - June 3, 2025".
- If the SQL uses dynamic relative dates (e.g., `created_at >= NOW() - INTERVAL '3 days'`), use relative terms: "Last 3 days".
- For comparisons between two periods, use the format "Comparison - [Period 1] vs [Period 2]". Examples:
- "Comparison - This Week vs Last Week"
- "Comparison - Q3 2024 vs Q3 2023"
- "Comparison - June 1, 2025 vs August 1, 2025"
RULE: Follow general quoting rules. Should not contain ':'.
# SQL QUERY
### SQL Best Practices and Constraints** (when creating new metrics)
# - **Constraints**: Only join tables with explicit entity relationships.
# - **SQL Requirements**:
# - Use schema-qualified table names (`<SCHEMA_NAME>.<TABLE_NAME>`).
# - Use schema-qualified table names (`<DATABASE_NAME>.<SCHEMA_NAME>.<TABLE_NAME>`).
# - Use fully qualified column names with table aliases (e.g., `<table_alias>.<column>`).
# - Select specific columns (avoid `SELECT *` or `COUNT(*)`).
# - Use CTEs instead of subqueries, and use snake_case for naming them.
# - Use `DISTINCT` (not `DISTINCT ON`) with matching `GROUP BY`/`SORT BY` clauses.
# - Show entity names rather than just IDs.
# - Handle date conversions appropriately.
# - Order dates in ascending order.
# - Reference database identifiers for cross-database queries.
# - Format output for the specified visualization type.
# - Maintain a consistent data structure across requests unless changes are required.
# - Use explicit ordering for custom buckets or categories.
# - When grouping metrics by dates, default to monthly granularity for spans over 2 months, yearly for over 3 years, weekly for under 2 months, and daily for under a week, unless the user specifies a different granularity.
# - Avoid division by zero errors by using NULLIF() or CASE statements (e.g., `SELECT amount / NULLIF(quantity, 0)` or `CASE WHEN quantity = 0 THEN NULL ELSE amount / quantity END`).
###
# - Consider potential data duplication and apply deduplication techniques (e.g., `DISTINCT`, `GROUP BY`) where necessary.
sql:
required: true
type: string
@ -279,7 +288,7 @@ definitions:
-
currency # Note: The "$" sign is automatically prepended.
-
percent # Note: The value is automatically multiplied by 100 and the "%" sign is appended.
percent # Note: "%" sign is appended. You need to use the multiplier to either multiply or divide the number by 100.
- number
- date
- string
@ -297,13 +306,14 @@ definitions:
description: Maximum number of fraction digits to display
multiplier:
type: number
description: Value to multiply the number by before display
description: Value to multiply the number by before display. This is usually required for percentages.
prefix:
type: string
suffix:
type: string
replaceMissingDataWith:
description: Value to display when data is missing, this should be set to null as default.
type: number
description: Value to display when data is missing, needs to be set to 0.
compactNumbers:
type: boolean
description: Whether to display numbers in compact form (e.g., 1K, 1M)
@ -321,10 +331,16 @@ definitions:
description: Whether to interpret dates as UTC
convertNumberTo:
type: string
description: this is useful for converting numberic 1-12 into month names
description: Optional. Convert numeric values to time units or date parts. This is a necessity for time series data when numbers are passed instead of the date.
enum:
- day_of_week
- month_of_year
- quarter
required:
- columnType
- style
- replaceMissingDataWith
# COLUMN VISUAL SETTINGS
column_settings:

4
api/libs/agents/src/tools/categories/file_tools/create_dashboards.rs
View File

@ -396,7 +396,7 @@ async fn get_dashboard_yml_description() -> String {
async fn get_dashboard_name_description() -> String {
if env::var("USE_BRAINTRUST_PROMPTS").is_err() {
return "The name of the dashboard file to be created. Do not include the file extension.".to_string();
return "The natural language name/title for the dashboard, exactly matching the 'name' field within the YML content. This name will identify the dashboard in the UI. Do not include file extensions or use file path characters.".to_string();
}
let client = BraintrustClient::new(None, "96af8b2b-cf3c-494f-9092-44eb3d5b96ff").unwrap();
@ -404,7 +404,7 @@ async fn get_dashboard_name_description() -> String {
Ok(message) => message,
Err(e) => {
eprintln!("Failed to get prompt system message: {}", e);
"The name of the dashboard file to be created. Do not include the file extension.".to_string()
"The natural language name/title for the dashboard, exactly matching the 'name' field within the YML content. This name will identify the dashboard in the UI. Do not include file extensions or use file path characters.".to_string()
}
}
}

12
api/libs/agents/src/tools/categories/file_tools/create_metrics.rs
View File

@ -297,7 +297,7 @@ impl ToolExecutor for CreateMetricFilesTool {
async fn get_create_metrics_description() -> String {
if env::var("USE_BRAINTRUST_PROMPTS").is_err() {
return "Creates metric configuration files with YAML content following the metric schema specification. Before using this tool, carefully consider the appropriate visualization type (bar, line, scatter, pie, combo, metric, table) and its specific configuration requirements. Each visualization has unique axis settings, formatting options, and data structure needs that must be thoroughly planned to create effective metrics.".to_string();
return "Creates metric configuration files with YAML content following the metric schema specification. Before using this tool, carefully consider the appropriate visualization type (bar, line, scatter, pie, combo, metric, table) and its specific configuration requirements. Each visualization has unique axis settings, formatting options, and data structure needs that must be thoroughly planned to create effective metrics. **This tool supports creating multiple metrics in a single call; prefer using bulk creation over creating metrics one by one.**".to_string();
}
let client = BraintrustClient::new(None, "96af8b2b-cf3c-494f-9092-44eb3d5b96ff").unwrap();
@ -305,14 +305,14 @@ async fn get_create_metrics_description() -> String {
Ok(message) => message,
Err(e) => {
eprintln!("Failed to get prompt system message: {}", e);
"Creates metric configuration files with YAML content following the metric schema specification. Before using this tool, carefully consider the appropriate visualization type (bar, line, scatter, pie, combo, metric, table) and its specific configuration requirements. Each visualization has unique axis settings, formatting options, and data structure needs that must be thoroughly planned to create effective metrics.".to_string()
"Creates metric configuration files with YAML content following the metric schema specification. Before using this tool, carefully consider the appropriate visualization type (bar, line, scatter, pie, combo, metric, table) and its specific configuration requirements. Each visualization has unique axis settings, formatting options, and data structure needs that must be thoroughly planned to create effective metrics. **This tool supports creating multiple metrics in a single call; prefer using bulk creation over creating metrics one by one.**".to_string()
}
}
}
async fn get_metric_name_description() -> String {
if env::var("USE_BRAINTRUST_PROMPTS").is_err() {
return "This is a natural language name/title for the metric. It will be used to identify the metric in the UI.".to_string();
return "The natural language name/title for the metric, exactly matching the 'name' field within the YML content. This name will identify the metric in the UI. Do not include file extensions or use file path characters.".to_string();
}
let client = BraintrustClient::new(None, "96af8b2b-cf3c-494f-9092-44eb3d5b96ff").unwrap();
@ -320,7 +320,7 @@ async fn get_metric_name_description() -> String {
Ok(message) => message,
Err(e) => {
eprintln!("Failed to get prompt system message: {}", e);
"This is a natural language name/title for the metric. It will be used to identify the metric in the UI.".to_string()
"The natural language name/title for the metric, exactly matching the 'name' field within the YML content. This name will identify the metric in the UI. Do not include file extensions or use file path characters.".to_string()
}
}
}
@ -328,7 +328,7 @@ async fn get_metric_name_description() -> String {
async fn get_metric_yml_description() -> String {
if env::var("USE_BRAINTRUST_PROMPTS").is_err() {
// Revert to just returning the schema string
return METRIC_YML_SCHEMA.to_string();
return format!("The YAML content for a single metric, adhering to the schema below. Multiple metrics can be created in one call by providing multiple entries in the 'files' array. **Prefer creating metrics in bulk.**\n\n{}", METRIC_YML_SCHEMA);
}
let client = BraintrustClient::new(None, "96af8b2b-cf3c-494f-9092-44eb3d5b96ff").unwrap();
@ -337,7 +337,7 @@ async fn get_metric_yml_description() -> String {
Err(e) => {
eprintln!("Failed to get prompt system message: {}", e);
// Revert to just returning the schema string on error
METRIC_YML_SCHEMA.to_string()
format!("The YAML content for a single metric, adhering to the schema below. Multiple metrics can be created in one call by providing multiple entries in the 'files' array. **Prefer creating metrics in bulk.**\n\n{}", METRIC_YML_SCHEMA)
}
}
}

12
api/libs/agents/src/tools/categories/file_tools/modify_metrics.rs
View File

@ -474,7 +474,7 @@ impl ToolExecutor for ModifyMetricFilesTool {
async fn get_modify_metrics_description() -> String {
if env::var("USE_BRAINTRUST_PROMPTS").is_err() {
return "Updates existing metric configuration files with new YAML content. Provide the complete YAML content for each metric, replacing the entire existing file. This tool is ideal for bulk modifications when you need to update multiple metrics simultaneously. The system will preserve version history and perform all necessary validations on the new content. For each metric, you need its UUID and the complete updated YAML content.".to_string();
return "Updates existing metric configuration files with new YAML content. Provide the complete YAML content for each metric, replacing the entire existing file. This tool is ideal for bulk modifications when you need to update multiple metrics simultaneously. The system will preserve version history and perform all necessary validations on the new content. For each metric, you need its UUID and the complete updated YAML content. **Prefer modifying metrics in bulk using this tool rather than one by one.**".to_string();
}
let client = BraintrustClient::new(None, "96af8b2b-cf3c-494f-9092-44eb3d5b96ff").unwrap();
@ -482,14 +482,14 @@ async fn get_modify_metrics_description() -> String {
Ok(message) => message,
Err(e) => {
eprintln!("Failed to get prompt system message: {}", e);
"Updates existing metric configuration files with new YAML content. Provide the complete YAML content for each metric, replacing the entire existing file. This tool is ideal for bulk modifications when you need to update multiple metrics simultaneously. The system will preserve version history and perform all necessary validations on the new content. For each metric, you need its UUID and the complete updated YAML content.".to_string()
"Updates existing metric configuration files with new YAML content. Provide the complete YAML content for each metric, replacing the entire existing file. This tool is ideal for bulk modifications when you need to update multiple metrics simultaneously. The system will preserve version history and perform all necessary validations on the new content. For each metric, you need its UUID and the complete updated YAML content. **Prefer modifying metrics in bulk using this tool rather than one by one.**".to_string()
}
}
}
async fn get_modify_metrics_yml_description() -> String {
if env::var("USE_BRAINTRUST_PROMPTS").is_err() {
return "Array of metrics to update. Each item requires an 'id' (UUID of the existing metric) and 'yml_content' (complete new YAML content that follows the specification below). You can update multiple metrics in a single operation, making this ideal for bulk updates.".to_string();
return "Array of metrics to update. Each item requires an 'id' (UUID of the existing metric) and 'yml_content' (complete new YAML content that follows the specification below). You can update multiple metrics in a single operation, making this ideal for bulk updates. **Prefer using this for bulk updates rather than modifying metrics individually.**".to_string();
}
let client = BraintrustClient::new(None, "96af8b2b-cf3c-494f-9092-44eb3d5b96ff").unwrap();
@ -497,7 +497,7 @@ async fn get_modify_metrics_yml_description() -> String {
Ok(message) => message,
Err(e) => {
eprintln!("Failed to get prompt system message: {}", e);
"Array of metrics to update. Each item requires an 'id' (UUID of the existing metric) and 'yml_content' (complete new YAML content that follows the specification below). You can update multiple metrics in a single operation, making this ideal for bulk updates.".to_string()
"Array of metrics to update. Each item requires an 'id' (UUID of the existing metric) and 'yml_content' (complete new YAML content that follows the specification below). You can update multiple metrics in a single operation, making this ideal for bulk updates. **Prefer using this for bulk updates rather than modifying metrics individually.**".to_string()
}
}
}
@ -505,7 +505,7 @@ async fn get_modify_metrics_yml_description() -> String {
async fn get_metric_yml_description() -> String {
if env::var("USE_BRAINTRUST_PROMPTS").is_err() {
// Revert to just returning the schema string plus basic instruction
return format!("The complete new YAML content for the metric, following the metric schema specification. This will replace the entire existing content of the file. Ensure all required fields are present and properly formatted according to the schema.\n\n{}", METRIC_YML_SCHEMA);
return format!("The complete new YAML content for the metric, following the metric schema specification. This will replace the entire existing content of the file. Ensure all required fields are present and properly formatted according to the schema. When modifying multiple metrics, provide each in the 'files' array. **Prefer bulk modifications.**\n\n{}", METRIC_YML_SCHEMA);
}
let client = BraintrustClient::new(None, "96af8b2b-cf3c-494f-9092-44eb3d5b96ff").unwrap();
@ -514,7 +514,7 @@ async fn get_metric_yml_description() -> String {
Err(e) => {
eprintln!("Failed to get prompt system message: {}", e);
// Revert to just returning the schema string plus basic instruction on error
format!("The complete new YAML content for the metric, following the metric schema specification. This will replace the entire existing content of the file. Ensure all required fields are present and properly formatted according to the schema.\n\n{}", METRIC_YML_SCHEMA)
format!("The complete new YAML content for the metric, following the metric schema specification. This will replace the entire existing content of the file. Ensure all required fields are present and properly formatted according to the schema. When modifying multiple metrics, provide each in the 'files' array. **Prefer bulk modifications.**\n\n{}", METRIC_YML_SCHEMA)
}
}
}

21
api/libs/agents/src/tools/categories/planning_tools/create_plan_investigative.rs
View File

@ -128,6 +128,7 @@ async fn get_plan_investigative_description() -> String {
const PLAN_INVESTIGATIVE_TEMPLATE: &str = r##"
Use this template to create a clear and actionable plan for investigative data requests using SQL.
Ensure the final plan output is well-formatted with markdown for readability.
**Thought**
Analyze the user's request and outline your approach. Keep it simple. Use a clear, direct style to communicate your thoughts in a simple and natural tone. Consider the goal, the types of visualizations needed, the specific datasets that will be used, etc. You should aim to create lots of visualizations (more than 8) to assess which ones return valuable infromation, and then compile a dashboard.
@ -135,9 +136,9 @@ Analyze the user's request and outline your approach. Keep it simple. Use a clea
**Step-by-Step Plan**
1. **Create [number] visualization(s)**:
- **Title**: [Simple title for the visualization]
- **Type**: [e.g., Bar Chart, Line Chart, Number Card, etc]
- **Type**: [e.g., Bar Chart, Line Chart, Number Card, Grouped Bar Chart, Stacked Bar Chart, Multi-Line Chart, etc.]
- **Datasets**: [Relevant datasets]
- **Expected Output**: [Describe the visualization, e.g., axes and key elements, without SQL details]
- **Expected Output**: [Describe the visualization, e.g., axes and key elements. For grouped/stacked bars or multi-line charts, explicitly state the grouping/stacking/splitting method and the field used. See guidelines below.]
- [Repeat for each visualization]
2. **Create dashboard**:
@ -155,11 +156,15 @@ Add any assumptions, limitations, or clarifications about the analysis and findi
#### Guidelines
- **Visualizations**: Describe what the visualization should show (e.g., "a bar chart with months on the x-axis and sales on the y-axis"). Avoid SQL or technical details. Do not define names for axes labels, just state what data should go on each axis.
- **Create Visualizations in One Step**: All visualizations should be created in a single, bulk step (typically the first step) titled "Create [specify the number] visualizations"
- **For Grouped/Stacked Bars**: Explicitly state if it's a `grouped bar chart` or `stacked bar chart` (or `100% stacked`). Clearly name the field used for splitting/stacking (e.g., "grouped bars side-by-side split by `[field_name]`", "bars stacked by `[field_name]`").
- **For Multi-Line Charts**: Explicitly state it's a `multi-line chart`. Describe *how* the multiple lines are generated: either by splitting a single metric using a category field (e.g., "split into separate lines by `[field_name]`") OR by plotting multiple distinct metrics (e.g., "plotting separate lines for `[metric1]` and `[metric2]`").
- **For Combo Charts**: Describe which fields are on which Y-axis and their corresponding chart type (line or bar).
- **Create Visualizations in One Step**: All visualizations should be created in a single, bulk step (typically the first step) titled "Create [specify the number] visualizations".
- **Review**: Always include a review step to ensure accuracy and relevance.
- **Referencing SQL:** Do not include any specific SQL statements with your plan. The details of the SQL statement will be decided during the workflow. When outlining visualizations, only refer to the visualization title, type, datasets, and expected output.
- **Using Names in Visualizations**: When describing the expected output, specify that names should be displayed instead of IDs whenever possible, as IDs are not meaningful to users. If first and last names are available, indicate that they should be combined into a full name for display (e.g., "with sales rep full names labeling each line").
- **Default Time Range**: If the user does not specify a time range, default to the last 12 months.
- **Use Names instead of IDs**: When visualizations or tables include things like people, customers, vendors, products, categories, etc, you should display names instead of IDs (if names are included in the available datasets). IDs are not meaningful to users. For people, you should combine first and last names if they are available. State this clearly in the `Expected Output` (e.g., "...split into separate lines by sales rep full names").
- **Default Time Range**: If the user does not specify a time range for a visualization, default to the last 12 months.
- **Visual Modifications**: If the user requests visual changes (e.g., "make charts green"), describe the *intended change* (e.g., "Modify chart color to green") rather than specifying technical details or parameter names.
---
@ -182,9 +187,9 @@ Add any assumptions, limitations, or clarifications about the analysis and findi
- **Datasets:** turnover_data, employee_records
- **Expected Output:** A bar chart with departments on the x-axis and the number of employees who left (turnover count) on the y-axis, to identify departments with the highest turnover. This highlights which departments are most affected, allowing the user to focus efforts where turnover is most severe.
- **Title:** Turnover Rate by Department Over Time
- **Type:** Line Chart
- **Type:** Multi-Line Chart
- **Datasets:** turnover_data, employee_records
- **Expected Output:** A line chart with months on the x-axis and turnover rate (percentage of employees who left) on the y-axis, with a separate line for each department. This allows the user to compare turnover trends across departments, revealing whether issues are persistent or emerging in specific areas.
- **Expected Output:** A multi-line chart with months on the x-axis and turnover rate (percentage of employees who left) on the y-axis, split into separate lines by department name. This allows the user to compare turnover trends across departments, revealing whether issues are persistent or emerging in specific areas.
- **Title:** Average Tenure by Department
- **Type:** Bar Chart
- **Datasets:** employee_records
@ -216,7 +221,7 @@ Add any assumptions, limitations, or clarifications about the analysis and findi
- **Title:** Turnover by Department, Segmented by Reason for Leaving
- **Type:** Stacked Bar Chart
- **Datasets:** turnover_data, employee_records
- **Expected Output:** A stacked bar chart with departments on the x-axis and turnover count on the y-axis, with each bar segmented by reasons for leaving (e.g., salary, growth, culture). This highlights the primary reasons employees leave within each department, directly addressing the "why" behind turnover.
- **Expected Output:** A stacked bar chart with departments on the x-axis and turnover count on the y-axis, with bars stacked by reason for leaving (e.g., salary, growth, culture). This highlights the primary reasons employees leave within each department, directly addressing the "why" behind turnover.
2. **Create Dashboard**
- Save all relevant visualizations to a dashboard titled: Employee Turnover Analysis

51
api/libs/agents/src/tools/categories/planning_tools/create_plan_straightforward.rs
View File

@ -128,6 +128,7 @@ async fn get_plan_straightforward_description() -> String {
const PLAN_STRAIGHTFORWARD_TEMPLATE: &str = r##"
Use this template to create a clear and actionable plan tailored to the user's request.
Ensure the final plan output is well-formatted with markdown for readability.
**Thought**
Analyze the user's request and outline your approach. Keep it simple. Use a clear, direct style to communicate your thoughts in a simple and natural tone. Consider the goal, the types of visualizations needed, the specific datasets that will be used, etc. For broad or summary requests (e.g., "summarize sales"), plan to create lots of visualizations (8-12 total) to provide a comprehensive view of the data.
@ -136,9 +137,9 @@ Analyze the user's request and outline your approach. Keep it simple. Use a clea
Outline actionable steps to fulfill the request. Your plan should mirror the exact template below:
1. **Create [number] visualization(s)**:
- **Title**: [Simple title for the visualization]
- **Type**: [e.g., Bar Chart, Line Chart, Number Card, etc]
- **Type**: [e.g., Bar Chart, Line Chart, Number Card, Grouped Bar Chart, Stacked Bar Chart, Multi-Line Chart, etc.]
- **Datasets**: [Relevant datasets]
- **Expected Output**: [Describe the visualization, e.g., axes and key elements, without SQL details. Be specific and opinionated.]
- **Expected Output**: [Describe the visualization, e.g., axes and key elements. For grouped/stacked bars or multi-line charts, explicitly state the grouping/stacking/splitting method and the field used. See guidelines below.]
- [Repeat for each visualization if multiple]
2. **[(Optional) Create dashboard]**:
If creating multiple visualizations, specify how they should be organized into a dashboard (e.g., title, layout).
@ -152,13 +153,17 @@ Add context like assumptions, limitations, or acknowledge unsupported aspects of
#### Guidelines
- **Visualizations**: Describe what the visualization should show (e.g., "a bar chart with months on the x-axis and sales on the y-axis"). Avoid SQL or technical details. Do not define names for axes labels, just state what data should go on each axis.
- **Create Visualizations in One Step**: All visualizations should be created in a single, bulk step (typically the first step) titled "Create [specify the number] visualizations"
- **Broad Requests**: For broad or summary requests (e.g., "summarize assembly line performance", "show me important stuff", "how is the sales team doing?"), you must create at least 8 visualizations to ensure a comprehensive overview. Creating fewer than five visualizations is inadequate for such requests. Aim for 8-12 visualizations to cover various aspects of the data, such as sales trends, order metrics, customer behavior, or product performance, depending on the available datasets. Include lots of trends (time-series data), groupings, segments, etc. This ensures the user receives a thorough view of the requested information.
- **For Grouped/Stacked Bars**: Explicitly state if it's a `grouped bar chart` or `stacked bar chart` (or `100% stacked`). Clearly name the field used for splitting/stacking (e.g., "grouped bars side-by-side split by `[field_name]`", "bars stacked by `[field_name]`").
- **For Multi-Line Charts**: Explicitly state it's a `multi-line chart`. Describe *how* the multiple lines are generated: either by splitting a single metric using a category field (e.g., "split into separate lines by `[field_name]`") OR by plotting multiple distinct metrics (e.g., "plotting separate lines for `[metric1]` and `[metric2]`").
- **For Combo Charts**: Describe which fields are on which Y-axis and their corresponding chart type (line or bar).
- **Create Visualizations in One Step**: All visualizations should be created in a single, bulk step (typically the first step) titled "Create [specify the number] visualizations".
- **Broad Requests**: For broad or summary requests (e.g., "summarize assembly line performance", "show me important stuff", "how is the sales team doing?"), you must create at least 8 visualizations to ensure a comprehensive overview. Creating fewer than five visualizations is inadequate for such requests. Aim for 8-12 visualizations to cover various aspects of the data, such as sales trends, order metrics, customer behavior, or product performance, depending on the available datasets. Include lots of trends (time-series data), groupings, segments, etc. This ensures the user receives a thorough view of the requested information.
- **Review**: Always include a review step to ensure accuracy and relevance.
- **Referencing SQL:** Do not include any specific SQL statements with your plan. The details of the SQL statement will be decided during the workflow. When outlining visualizations, only refer to the visualization title, type, datasets, and expected output.
- **Use Names instead of IDs**: When visualizations or tables include things like people, customers, vendors, products, categories, etc, you should display names instead of IDs (if names are included in the available datasets). IDs are not meaningful to users. For people, you should combine first and last names if they are available.
- **Use Names instead of IDs**: When visualizations or tables include things like people, customers, vendors, products, categories, etc, you should display names instead of IDs (if names are included in the available datasets). IDs are not meaningful to users. For people, you should combine first and last names if they are available. State this clearly in the `Expected Output` (e.g., "...split into separate lines by sales rep full names").
- **Default to Top 10**: If the user requests the "top", "best", etc of any entity (e.g., products, regions, employees) without specifying a number, default to showing the top 10 in the visualization.
- **Default Time Range**: If the user does not specify a time range for a visualization, default to the last 12 months.
- **Visual Modifications**: If the user requests visual changes (e.g., "make charts green"), describe the *intended change* (e.g., "Modify chart color to green") rather than specifying technical details or parameter names.
---
@ -173,12 +178,12 @@ The user wants a bar chart showing total new customers per month over the last y
**Step-by-Step Plan**:
1. **Create 1 Visualization**:
- **Title**: "Monthly New Customers (Last 12 Months)"
- **Type**: Bar Chart
- **Datasets**: `sem.entity_customer`
- **Expected Output**: A bar chart with the last 12 months on the x-axis and the count of new customers on the y-axis.
- **Title**: "Monthly New Customers (Last 12 Months)"
- **Type**: Bar Chart
- **Datasets**: `sem.entity_customer`
- **Expected Output**: A bar chart with the last 12 months on the x-axis and the count of new customers on the y-axis.
2. **Review & Finish**:
- Confirm non-empty results and accurate counts. Respond to the user.
- Confirm non-empty results and accurate counts. Respond to the user.
```
---
@ -188,7 +193,7 @@ The user wants a bar chart showing total new customers per month over the last y
```
**Thought**:
The user wants a dashboard with three specific visualizations to see sales performance over the last 12 months (monthly sales, total sales, and monthly sales by sales rep). I'll use the `sales_data` dataset to create a line chart for monthly sales, a number card for total sales, and a line chart for sales by rep.
The user wants a dashboard with three specific visualizations to see sales performance over the last 12 months (monthly sales, total sales, and monthly sales by sales rep). I'll use the `sales_data` dataset to create a line chart for monthly sales, a number card for total sales, and a multi-line line chart for sales by rep.
**Step-by-Step Plan**:
1. **Create 3 Visualizations**:
@ -201,9 +206,9 @@ The user wants a dashboard with three specific visualizations to see sales perfo
- **Datasets**: `sales_data`
- **Expected Output**: A single-value card showing cumulative sales from the last 12 months, formatted as currency.
- **Title**: Monthly Sales by Sales Rep
- **Type**: Line Chart
- **Type**: Multi-Line Chart
- **Datasets**: `sales_data`
- **Expected Output**: A line chart with the last 12 months on the x-axis and sales amounts on the y-axis, with each line labeled by sales rep full names.
- **Expected Output**: A multi-line chart with the last 12 months on the x-axis and sales amounts on the y-axis, split into separate lines by sales rep full names.
2. **Create Dashboard**:
- Title: "Sales Performance, Last 12 Months"
- Add all three visualizations.
@ -227,37 +232,37 @@ The user is asking for a summary of product performance. This is an ambiguous, o
- **Datasets**: `entity_transaction_history`
- **Expected Output**: A line chart with months on the x-axis and total revenue on the y-axis.
- **Title**: Revenue and Units Sold by Product Category
- **Type**: Grouped Bar Chart
- **Type**: Bar Chart
- **Datasets**: `entity_transaction_history`, `entity_product`
- **Expected Output**: A grouped bar chart with product categories on the x-axis and bars for revenue and units sold.
- **Expected Output**: A bar chart with product categories on the x-axis and bars for revenue and units sold.
- **Title**: Profit Margin by Product Category
- **Type**: Bar Chart
- **Datasets**: `entity_transaction_history`, `entity_product`, `entity_inventory`
- **Expected Output**: A bar chart with categories on the x-axis and profit margin percentage on the y-axis.
- **Expected Output**: A bar chart with product category names on the x-axis and profit margin percentage on the y-axis.
- **Title**: Top 10 Products by Revenue
- **Type**: Bar Chart
- **Datasets**: `entity_transaction_history`, `entity_product`
- **Expected Output**: A bar chart with products on the x-axis and revenue on the y-axis.
- **Expected Output**: A bar chart with product names on the x-axis and revenue on the y-axis, showing only the top 10 products.
- **Title**: New vs. Returning Customers by Product Category
- **Type**: Stacked Bar Chart
- **Datasets**: `entity_transaction_history`, `entity_customer`, `entity_product`
- **Expected Output**: A stacked bar chart with categories on the x-axis and revenue percentage on the y-axis, split by customer type.
- **Expected Output**: A stacked bar chart with product category names on the x-axis and revenue percentage on the y-axis, with bars stacked by customer type (New/Returning).
- **Title**: Revenue by Sales Channel, Stacked by Product Category
- **Type**: Stacked Bar Chart
- **Datasets**: `entity_transaction_history`, `entity_product`, `entity_sales_channel`
- **Expected Output**: A stacked bar chart with channels on the x-axis and revenue on the y-axis, segmented by category.
- **Expected Output**: A stacked bar chart with sales channel names on the x-axis and revenue on the y-axis, with bars stacked by product category names.
- **Title**: Revenue Trend by Product Category
- **Type**: Line Chart
- **Type**: Multi-Line Chart
- **Datasets**: `entity_transaction_history`, `entity_product`
- **Expected Output**: A line chart with months on the x-axis and revenue on the y-axis, with lines per category.
- **Expected Output**: A multi-line chart with months on the x-axis and revenue on the y-axis, split into separate lines by product category names.
- **Title**: Percentage Change in Revenue by Product Category
- **Type**: Bar Chart
- **Datasets**: `entity_transaction_history`, `entity_product`
- **Expected Output**: A bar chart with categories on the x-axis and percentage change on the y-axis.
- **Expected Output**: A bar chart with product category names on the x-axis and percentage change in revenue on the y-axis.
- **Title**: Average Customer Satisfaction by Product Category
- **Type**: Bar Chart
- **Datasets**: `entity_customer_feedback`, `entity_product`
- **Expected Output**: A bar chart with categories on the x-axis and satisfaction scores on the y-axis.
- **Expected Output**: A bar chart with product category names on the x-axis and average satisfaction scores on the y-axis.
2. **Create Dashboard**:
- Title: "Product Performance Summary"
- Add all visualizations.

41
api/libs/database/src/types/metric_yml.rs
View File

@ -7,12 +7,12 @@ use diesel::{
sql_types::Jsonb,
};
use indexmap::IndexMap;
use lazy_static::lazy_static;
use regex::Regex;
use serde::{Deserialize, Serialize};
use serde_json::Value;
use serde_json::json;
use std::io::Write;
use uuid::Uuid;
use regex::Regex;
use lazy_static::lazy_static;
// Helper function to sanitize string values for YAML
fn sanitize_yaml_string(value: &str) -> String {
@ -22,6 +22,7 @@ fn sanitize_yaml_string(value: &str) -> String {
.replace('\"', "") // Remove double quotes
.replace('\n', " ") // Replace newlines with spaces
.replace('\t', " ") // Replace tabs with spaces
.replace('%', "Percent") // Replace % with Percent
.trim() // Trim leading/trailing whitespace
.to_string()
}
@ -76,7 +77,6 @@ pub enum ChartConfig {
Table(TableChartConfig),
}
#[derive(Debug, Serialize, Deserialize, Clone)]
#[serde(rename_all = "camelCase")]
#[serde(untagged)]
@ -201,7 +201,7 @@ impl ColumnLabelFormat {
multiplier: None,
prefix: None,
suffix: None,
replace_missing_data_with: None,
replace_missing_data_with: Some(json!(0)),
compact_numbers: None,
currency: None,
date_format: None,
@ -528,22 +528,23 @@ impl MetricYml {
for (index, line) in yml_content.lines().enumerate() {
// Store SQL line info for potential timeFrame insertion
if sql_line_index.is_none() {
if let Some(caps) = SQL_KEY_RE.captures(line) {
if let Some(caps) = SQL_KEY_RE.captures(line) {
sql_line_index = Some(index);
sql_line_indent = Some(caps.get(1).map_or("", |m| m.as_str()).to_string());
}
}
let current_indent = INDENT_RE.captures(line).map_or(0, |caps| {
caps.get(1).map_or(0, |m| m.as_str().len())
});
let current_indent = INDENT_RE
.captures(line)
.map_or(0, |caps| caps.get(1).map_or(0, |m| m.as_str().len()));
// --- Colors Block Logic ---
if in_colors_block && colors_indent.map_or(false, |indent| current_indent <= indent) {
in_colors_block = false;
colors_indent = None;
}
if !in_colors_block { // Only check for start if not already in block
if !in_colors_block {
// Only check for start if not already in block
if let Some(caps) = COLORS_START_RE.captures(line) {
in_colors_block = true;
colors_indent = Some(caps.get(1).map_or(0, |m| m.as_str().len()));
@ -558,14 +559,14 @@ impl MetricYml {
processed_lines.push(format!("{}'{}'", marker_part, color_part));
continue;
} else {
processed_lines.push(line.to_string()); // Add line within color block as is if not a hex item
continue;
processed_lines.push(line.to_string()); // Add line within color block as is if not a hex item
continue;
}
}
// --- End Colors Block Logic ---
// --- String Sanitization & timeFrame Check ---
if let Some(caps) = SANITIZE_KEYS_RE.captures(line) {
if let Some(caps) = SANITIZE_KEYS_RE.captures(line) {
let indent = caps.name("indent").map_or("", |m| m.as_str());
let key = caps.name("key").map_or("", |m| m.as_str());
let value = caps.name("value").map_or("", |m| m.as_str());
@ -575,13 +576,9 @@ impl MetricYml {
}
let sanitized_value = sanitize_yaml_string(value);
// Reconstruct line, potentially quoting if value was empty after sanitizing?
// For now, just place the sanitized value. YAML might handle empty strings okay.
// If the value needs quotes (e.g., contains special chars AFTER sanitization, though unlikely now)
// we might need more complex logic. Let's assume simple value placement is fine.
processed_lines.push(format!("{}{}: {}", indent, key, sanitized_value));
} else {
// Add lines that don't match any processing rules
// Add lines that don't match any processing rules (like the sql block)
processed_lines.push(line.to_string());
}
}
@ -589,15 +586,15 @@ impl MetricYml {
// Insert default timeFrame if not found
if !time_frame_found {
if let Some(index) = sql_line_index {
// Use the indent captured from the sql line
// Use the indent captured from the sql line
let indent = sql_line_indent.unwrap_or_else(|| " ".to_string()); // Default indent if sql indent capture failed
processed_lines.insert(index, format!("{}timeFrame: 'all_time'", indent));
} else {
// Fallback: append if sql key wasn't found (shouldn't happen for valid metric)
// Or maybe error out?
eprintln!("Warning: sql key not found in metric YAML, cannot insert default timeFrame correctly.");
// Append at end with default indent - might break YAML structure
processed_lines.push(" timeFrame: 'all_time'".to_string());
eprintln!("Warning: sql key not found in metric YAML, cannot insert default timeFrame correctly.");
// Append at end with default indent - might break YAML structure
processed_lines.push(" timeFrame: 'all_time'".to_string());
}
}

21
api/libs/handlers/src/chats/post_chat_handler.rs
View File

@ -2280,21 +2280,22 @@ pub fn normalize_asset_fields(request: &ChatCreateNewChat) -> (Option<Uuid>, Opt
// Constants for title generation
const TITLE_GENERATION_PROMPT: &str = r#"
You are a conversation title generator. Your task is to generate a clear, concise, and descriptive title for a conversation based on the user messages and assistant responses provided.
You are a conversation title generator. Your task is to generate a clear, concise title (3-10 words) that summarizes the **core subject matter** of the conversation, primarily based on the **most recent user message**.
Guidelines:
1. The title should be 3-10 words and should capture the core topic or intent of the conversation
2. Focus on key topics, questions, or themes from the conversation
3. Be specific rather than generic when possible
4. Avoid phrases like "Conversation about..." or "Discussion on..."
5. Don't include mentions of yourself in the title
6. The title should make sense out of context
7. Pay attention to the most recent messages to guide topic changes, etc.
1. **Focus on the Topic:** Identify the key nouns, concepts, or goals mentioned by the user, especially in their latest message. What is the conversation *about*?
2. **Prioritize Recent Request:** The title should strongly reflect the subject of the most recent user input.
3. **Be Specific & Concise:** Capture the essence in 3-10 words.
4. **AVOID Action Verbs/File Types:** Do NOT use words like "creating", "modifying", "updating", "dashboard", "metric", "file", "chart", "visualization" unless the user's core request was *specifically* about the process of creation/modification itself (rare). Instead, focus on *what* is being created or modified (e.g., "Monthly Sales Goals", not "Creating Metric for Monthly Sales Goals").
5. **Natural Language:** Phrase the title naturally.
6. **No Self-Reference:** Do not mention yourself (the assistant).
7. **Context Independent:** The title should make sense on its own.
8. **Example:** If the last user message is "Show me total revenue for Q3 as a line chart", a good title would be "Q3 Total Revenue" or "Quarter 3 Revenue Analysis". A bad title would be "Creating Q3 Revenue Line Chart".
Conversation:
Conversation History (Most recent messages are most important):
{conversation_messages}
Return only the title text with no additional formatting, explanation, quotes, new lines, special characters, etc.
Return ONLY the generated title text. Do not include quotes, explanations, or any other text.
"#;
/// Generates a title for a conversation by processing user and assistant messages.