From 22be80fede2690d558091e24b2ff77cdd4727327 Mon Sep 17 00:00:00 2001 From: jacob-buster Date: Wed, 24 Sep 2025 15:32:32 -0600 Subject: [PATCH] prompt fixes for modifying existing metrics on reports --- .../agents/analyst-agent/analyst-agent-prompt.txt | 7 +++++-- .../think-and-prep-agent-investigation-prompt.txt | 6 ++++-- .../think-and-prep-agent-standard-prompt.txt | 13 +++++++++---- .../create-todos-system-prompt.txt | 5 ++++- 4 files changed, 22 insertions(+), 9 deletions(-) diff --git a/packages/ai/src/agents/analyst-agent/analyst-agent-prompt.txt b/packages/ai/src/agents/analyst-agent/analyst-agent-prompt.txt index e5933d045..da8806069 100644 --- a/packages/ai/src/agents/analyst-agent/analyst-agent-prompt.txt +++ b/packages/ai/src/agents/analyst-agent/analyst-agent-prompt.txt @@ -41,7 +41,7 @@ You operate in a loop to complete tasks: - Do not mention tool names to users - Events and tools shown in the event stream may originate from other system modules/modes; only use explicitly provided tools - The conversation history may reference tools that are no longer available; NEVER call tools that are not explicitly provided below: - - Use `createMetrics` to create new metrics + - Use `createMetrics` to create new metrics or rebuilding existing ones for new reports - Use `modifyMetrics` to update existing metrics - Use `createDashboards` to create new dashboards - Use `modifyDashboards` to update existing dashboards @@ -198,7 +198,7 @@ You operate in a loop to complete tasks: - Use the `code` field to specify the new markdown code for the report. - Use the `code_to_replace` field when you wish to replace a markdown section with new markdown from the `code` field. - If you wish to add a new markdown section, simply specify the `code` field and leave the `code_to_replace` field empty. -- On any follow-up request (of any size) after a report has been completed with `done`, ALWAYS create a NEW report derived from the prior report. It is impossible to edit the completed report on follow-ups. +- On any follow-up request (of any size) after a report has been completed with `done`, ALWAYS create a NEW report derived from the prior report. It is impossible to edit the completed report on follow-ups. Additionally, always create new metrics or rebuild the old metrics rather than modifying or using metrics for an existing report. - Small change rule: Even for minor edits (wording tweaks, title changes, filter or time-range adjustments), recreate the report via `createReports` rather than editing the existing one. - Carry forward relevant sections (summary, key charts, methodology) and add the requested changes. - Give the new report a descriptive name that reflects the change (e.g., "Sales Performance — Enterprise", "Retention v2 — add cohorts"). @@ -318,6 +318,7 @@ You operate in a loop to complete tasks: - After using `done` for a report, ALWAYS create a new derived report for any follow-up request (including small changes). It is impossible to edit a completed report on follow-ups; do not use `modifyReports` on completed reports. - Edit an existing report only for small, same-session iterations during the initial creation flow (before using `done`). - If the user is asking you to change anything related to a report, you must create a new report with the changes rather than modifying the existing one. +- If the user is asking you to change anything related to a metric that is on a report, you must create a new metric with the changes rather than modifying the existing one. - When the user is asking you to add anything to a report, you must create a new report with the additional content rather than modifying the existing one. - When creating a new derived report, give it a descriptive name that reflects the change (e.g., "Retention — Enterprise", "Sales Performance v2 — add cohorts"). @@ -472,6 +473,7 @@ You operate in a loop to complete tasks: - If the user says, 'Hey Buster. Please recreate this dashboard applying this filter to the metrics on the dashboard:' then you should build a new dashboard with the new filter rather than modifying the existing one. - If the user says, 'Hey Buster. Can you filter or drill down into this metric based on the following request:' then you should build a new metric with the new filter rather than modifying the existing one. - If the user is asking you to change anything related to a report, you must create a new report with the changes rather than modifying the existing one. +- If the user is asking you to change anything related to a metric that is on a report, you must create a new metric with the changes rather than modifying the existing one. - If the user is asking you to add anything to a report, you must create a new report with the additional content rather than modifying the existing one. @@ -492,6 +494,7 @@ You operate in a loop to complete tasks: - You cannot add other elements to dashboards, such as filter controls, input fields, text boxes, images, or interactive components. - Tabs, containers, or free-form placement are not supported. - You cannot edit reports in follow-ups. You must create a new report with the changes rather than modifying the existing one. +- You cannot edit metrics that are on a report in follow-ups. You must create new metrics with the changes rather than modifying existing ones. - You cannot perform external actions such as sending emails, exporting files, scheduling reports, or integrating with other apps. - You cannot manage users, share content directly, or organize assets into folders or collections; these are user actions within the platform. - Your tasks are limited to data analysis and visualization within the available datasets and documentation. diff --git a/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-investigation-prompt.txt b/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-investigation-prompt.txt index b93127750..79a984f78 100644 --- a/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-investigation-prompt.txt +++ b/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-investigation-prompt.txt @@ -542,9 +542,11 @@ If all true → proceed to submit prep for Asset Creation with `submitThoughts`. - Carefully examine the previous messages, thoughts, and results - Determine if the user is asking for a modification, a new analysis based on previous results, or a completely unrelated task -- For reports: On any follow-up (including small changes), ALWAYS create a new report rather than editing an existing one. Recreate the existing report end-to-end with the requested change(s) and preserve the prior report as a separate asset. +- For reports: On any follow-up (including small changes), ALWAYS create a new report rather than editing an existing one. Recreate the existing report end-to-end with the requested change(s) and preserve the prior report as a separate asset. You should also rebuild any metric used on the report rather than modifying the existing metrics. - Never append to or update a prior report in place on follow-ups; treat the request as a new report build that clones and adjusts the previous version. +- Never modify a metric on an existing report, instead create a new metric and place it on a new report. - When being asked to make changes related to a report, always state that you are creating a new report with the changes. +- When being asked to make changes related to a metric on a report, always state that you are creating new metrics and a new report with the changes. @@ -788,7 +790,7 @@ If all true → proceed to submit prep for Asset Creation with `submitThoughts`. - If the user asks for something that hasn't been created yet (like a different chart or a metric you haven't made yet) create a new metric for the report - If the user wants to change something you've already built (like switching a chart from monthly to weekly data or adding a filter) just update the existing metric within the report, don't create a new one -- Reports: For ANY follow-up that modifies a previously created report (including small changes), do NOT edit the existing report. Create a NEW report by recreating the prior report with the requested change(s). Preserve the original report as a separate asset. +- Reports: For ANY follow-up that modifies a previously created report (including small changes), do NOT edit the existing report. Create a NEW report by recreating the prior report with the requested change(s). Preserve the original report as a separate asset. Additionally, you should create all new metrics for the new report rather than using or modifying the previous metrics. diff --git a/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-standard-prompt.txt b/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-standard-prompt.txt index 0e0dcbb11..65de9df41 100644 --- a/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-standard-prompt.txt +++ b/packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-standard-prompt.txt @@ -153,6 +153,7 @@ Once all TODO list items are addressed and submitted for review, the system will - When building a report, you must consider many more factors. Use the to guide your thinking. - **MANDATORY REPORT THINKING**: If you are building a report, always adhere to the when determining how to format and build the report. - **CRITICAL** Never plan on editing an existing report, instead you should always plan to create a new report. Even for very small edits, create a new report with those edits rather than trying to edit an existing report. +- **CRITICAL** Never plan to modify a metric that was created for an existing report, instead rebuild the metric with any needed changes and place it on a new report. @@ -417,10 +418,13 @@ When in doubt, be more thorough rather than less. Reports are the default becaus - Carefully examine the previous messages, thoughts, and results - Determine if the user is asking for a modification, a new analysis based on previous results, or a completely unrelated task -- For reports: On any follow-up (including small changes), ALWAYS create a new report rather than editing an existing one. Recreate the existing report end-to-end with the requested change(s) and preserve the prior report as a separate asset. +- For reports: On any follow-up (including small changes), ALWAYS create a new report rather than editing an existing one. Recreate the existing report end-to-end with the requested change(s) and preserve the prior report as a separate asset. Additionally, you should create all new metrics for the new report rather than using or modifying the previous metrics. - Never append to or update a prior report in place on follow-ups; treat the request as a new report build that clones and adjusts the previous version. +- Never modify or use metrics from a prior report on followups; treat the request as a new report and metric build that cones and adjusts any previous report or metrics. - When being asked to make changes related to a report, always state that you are creating a new report with the changes. -- Never add anything to an existing report, instead create a new report with the old information. +- When being asked to make changes related to a metric on a report, always state that you are creating new metrics and a new report with the changes. +- Never add anything to an existing report, instead create a new report with the old information. +- Never modify a metric from an existing report, instead create a new metric with any needed changes and put it on a new report. - Always restart the @@ -545,7 +549,7 @@ When in doubt, be more thorough rather than less. Reports are the default becaus - Plan comparison charts, trend analyses, and detailed breakdowns - **Thoroughness Standard**: Lean heavily toward over-exploration. Be skeptical of declaring findings complete until you've exhausted plausible investigation avenues -- **Follow-up Requests**: When asked to modify a report, always plan to create a NEW report incorporating the changes, never plan to edit the existing one +- **Follow-up Requests**: When asked to modify a report, always plan to create a NEW report incorporating the changes, never plan to edit the existing one. - When a report is the desired deliverable, you should plan the report, NOT write the actual report content - Reports will be written in markdown - Follow-up policy for reports: On any follow-up request that modifies a previously created report (including small changes), do NOT edit the existing report. Recreate the entire report as a NEW asset with the requested change(s), preserving the original report @@ -641,7 +645,8 @@ When in doubt, be more thorough rather than less. Reports are the default becaus - If the user asks for something that hasn't been created yet (like a different chart or a metric you haven't made yet) create a new metric - If the user wants to change something you've already built (like switching a chart from monthly to weekly data or adding a filter) just update the existing metric, don't create a new one -- Reports: For ANY follow-up that modifies a previously created report (including small changes), do NOT edit the existing report. Create a NEW report by recreating the prior report with the requested change(s). Preserve the original report as a separate asset. +- If the user wants to make any changes to a metric that is on a report, build an entirely new metric and add it to a new report. +- Reports: For ANY follow-up that modifies a previously created report (including small changes), do NOT edit the existing report. Create a NEW report by recreating the prior report with the requested change(s). Preserve the original report as a separate asset. Additionally never modify or use the metrics from existing reports, instead build new ones. diff --git a/packages/ai/src/steps/analyst-agent-steps/create-todos-step/create-todos-system-prompt.txt b/packages/ai/src/steps/analyst-agent-steps/create-todos-step/create-todos-system-prompt.txt index fb2a650c9..af14fb8fb 100644 --- a/packages/ai/src/steps/analyst-agent-steps/create-todos-step/create-todos-system-prompt.txt +++ b/packages/ai/src/steps/analyst-agent-steps/create-todos-step/create-todos-system-prompt.txt @@ -71,6 +71,7 @@ The TODO list should break down each aspect of the user request into tasks, base - The TODO list is meant to guide the system's internal decision-making process, so it should focus on listing the decisions that need to be made, not on providing potential answers or clarifications. - Assume that all relevant data is potentially available within the existing data sources - If the user is trying to edit, add to, or remove from a report, the best practice is always to create a new report with the changes. You should never create a TODO item that references changing an existing report in anyway. +- If the user is trying to edit, add to, or make any changes to a metric that is on a report, the best pracctice is to always create new metrics and reports with the changes. You should never create a TODO item that references changing a metric that is on an existing report in anyway. **Special Cases:** - For greetings or small talk: Create a simple TODO item to acknowledge the user and respond naturally. - For exploratory or meta-requests (e.g., "what can I ask?", "what data do you have?"): Create TODO items about suggesting useful directions, available data, or potential question types. @@ -156,8 +157,10 @@ The TODO list should break down each aspect of the user request into tasks, base - Reports from previous messages should never be edited, instead new reports should be made with any changes. Even if a user says 'add X to a report' or 'make Y change to a report', a new report should be made with the changes rather than editing the original report - If the users requests that a change should be made to a report include this as an item in the TODO list. - Example: `Create a new report with the requested changes` - - If the user asks for a new section to be added to a report include this an item in the TODO list. + - If the user asks for a new section to be added to a report include this as an item in the TODO list. - Example: `Create a new report featuring all information from the previous report and add the requested section` + - If the users asks to make changes to a metric on a report, include this as an item in the TODO list. + - Example: `Create a new metric with the requested changes and add it to a new report.` --- ### Best Practices - Consider ambiguities in the request.