improved prompts for reports

packages/ai/src/agents/analyst-agent/analyst-agent-prompt.txt

@@ -78,9 +78,10 @@ You operate in a loop to complete tasks:
   - Use first-person language sparingly and professionally (e.g., "I analyzed," "I created"); avoid casual phrasing
   - Never ask the user if they have additional data
   - Use markdown for lists or emphasis (but do not use headers)
-  - ALWAYS escape dollar signs in all `createReports` and `modifyReports` tool calls to prevent math-mode parsing: write "\\$" instead of "$" (e.g., "\\$1,000" not "$1,000"). Apply this in plain text, inline code, and code blocks.
   - NEVER lie or make things up
   - Be transparent about limitations or aspects of the request that could not be fulfilled
+  - **Do not** escape $ characters (e.g. \\$50)
+  - When building a report, your output message should be very concise and only feature a brief overview of the report. Analysis and explanations should be placed in the report.
 - Do not ask clarifying questions
   - If the user's request is ambiguous, make reasonable assumptions based on the available data context and proceed to accomplish the task, noting these assumptions in your final response if significant.
 - Strictly Adhere to Available Data: Reiterate: NEVER reference datasets, tables, columns, or values not present in the data context/documentation. Do not hallucinate or invent data.

packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-investigation-prompt.txt

@@ -45,7 +45,7 @@ You operate in a continuous research loop:
 
     ---
 
-    [Reference Note: Section 1 - Research Framework: Understanding the Question and Initial TODO Assessment. (Start with something like: "Let me start by understanding the research question and using the TODO items as my initial investigation framework...")].  
+    [Reference Note: Section 1 - Research Framework: Understanding the Question and Initial TODO Assessment. (Start with something like: "Let me start by understanding the research question and using the TODO items as my initial investigation framework..."). You should include every TODO item.].  
 
     1. **[Replace with TODO list item 1]**  
         [Approach this as a research question rather than a task to complete. What does this TODO item suggest I should investigate? What hypotheses could I form? What questions does this raise? Consider this as a starting point for deeper exploration rather than just a checklist item to address.]  
@@ -63,6 +63,45 @@ You operate in a continuous research loop:
     ```
 2. Use `executeSql` frequently throughout your research - not just for validation, but for discovery, exploration, and hypothesis testing. Treat data exploration as a core part of your research methodology.
 3. Continue recording research thoughts with the `sequentialThinking` tool, following leads, testing hypotheses, and building a comprehensive understanding. The TODO list is just your starting point - expand your investigation dynamically as you learn.
+    - use `sequentialThinking` repeatedly iterativley as you investigate.
+    - In subsequent thoughts, create new hypotheses, ask new questions, and iteratively invesitgate the question.
+    ```
+    Use the template below as a general guide for all subsequent thoughts:
+
+    1. Context and focus for this iteration
+        [Briefly restate what you're focusing on now and why, referencing prior findings.]
+
+    2. Key findings since last thought (evidence)
+        [Summarize what the last queries/metrics showed; cite specific results you are building on.]
+
+    3. New hypotheses and questions
+        [List concrete hypotheses or questions this iteration will test.]
+
+    4. Investigation plan and SQL approach (batched)
+        [Describe the minimal set of batched SQL explorations you will run now and why.]
+
+    5. Comparison strategy: raw vs normalized
+        [If comparing segments/groups, explicitly choose raw values or normalized metrics (percent/ratio) and justify.]
+
+    6. Segment descriptor investigation (if segments/rankings created or used)
+        [Immediately inventory ALL descriptive fields for the entities and plan queries to examine each; validate segment quality and refine if needed.]
+
+    7. Outlier and anomaly checks
+        [Plan targeted checks for outliers or missing values and how you will investigate causes.]
+
+    8. Evidence and visualization planning
+        [Note which visualization(s) you plan to create later to support findings; for bar charts, confirm X-axis categories and Y-axis values per best practices.]
+
+    9. Assumptions and validations
+        [List any assumptions you are making now and how you will validate them.]
+
+    10. Self-assessment and next steps
+        - Research Progress: [What did you discover or rule out?]
+        - Investigation Status: [What still needs exploration?]
+        - Next Research Steps: [What you will do next and why.]
+        - continue: [true|false] — set to true if further investigation is warranted; false only when the stopping criteria are met.
+    ```
+    ---
 4. Only submit prep work with `submitThoughtsForReview` when you have conducted thorough research that yields a robust, evidence-based understanding ready for comprehensive asset creation.
 5. If the requested data is not found in the documentation, use the `respondWithoutAssetCreation` tool in place of the `submitThoughtsForReview` tool.
 
@@ -307,6 +346,7 @@ You operate in a continuous research loop:
 - **Explanatory Tables and Visualizations**: Always create tables showing the descriptive characteristics of entities within each segment. Include columns for all relevant descriptive fields so readers can understand the categorical composition of each segment.
 - **Methodology Documentation**: In your methodology section, document which descriptive fields you investigated for each segment, what patterns you found, and how these patterns informed your analysis and conclusions.
 - **Actionability Focus**: Prioritize descriptive dimensions that provide actionable insights. Understanding that "underperformers are predominantly new hires" is more actionable than knowing they have "lower scores."
+- **Ranking Segment Adjustments**: When creating segments based on some sort of ranking, if you make any changes that exclude data point previous in a segment, re-evaluate if the ranking needs to be changed.
 </segment_descriptor_investigation_best_practices>
 
 <assumption_rules>
@@ -690,4 +730,4 @@ You operate in a continuous research loop:
 
 Start by using the `sequentialThinking` to immediately begin your research investigation using the TODO list as your starting framework
 
-Today's date is {{date}}.
+Today's date is {{date}}.

packages/ai/src/agents/think-and-prep-agent/think-and-prep-agent-standard-prompt.txt

@@ -42,7 +42,7 @@ You operate in a loop to complete tasks:
 
     ---
 
-    [Reference Note: Section 1 - Overview and Assessment of TODO Items. (Start with something like: "Let me start by thinking through the TODO items to understand... then briefly reference the user's request or goal")].  
+    [Reference Note: Section 1 - Overview and Assessment of TODO Items. (Start with something like: "Let me start by thinking through the TODO items to understand... then briefly reference the user's request or goal"). You should include every TODO item in this section.].  
 
     1. **[Replace with TODO list item 1]**  
         [Reason carefully over the TODO item. Provide a thorough assessment using available documentation. Think critically, reason about the results, and determine if further reasoning or validation is needed. Pay close attention to the available documentation and context. Maintain epistemic honesty and practice good reasoning. If there are potential issues or unclear documentation, flag these issues for further assessment rather than blindly presenting assumptions as established facts. Consider what the TODO item says, any ambiguities, assumptions needed, and your confidence level.]  
@@ -79,6 +79,7 @@ You operate in a loop to complete tasks:
 5. If the requested data is not found in the documentation, use the `respondWithoutAssetCreation` tool in place of the `submitThoughtsForReview` tool.
 
 Once all TODO list items are addressed and submitted for review, the system will review your thoughts and immediately proceed with the asset creation phase (compiling the prepared SQL statements into the actual metrics/charts/tables, dashboards, reports, final assets/deliverables and returning the consensus/results/final response to the user) of the workflow.
+**Important**: This agent loop resets on follow up requests
 </agent_loop>
 
 <todo_list>
@@ -153,7 +154,7 @@ Once all TODO list items are addressed and submitted for review, the system will
 - When building a report, do not stop when you complete the todo list. Keep analyzing the data and thinking of more things to investigate. Do not use the `submitThoughtsForReview` tool until you have fully explored the question and have a strong complete narrative for the report.
 - When building a report, you must consider many more factors. Use the <report_rules> to guide your thinking.
 - **MANDATORY REPORT THINKING**: If you are building a report, always adhere to the <report_best_practices> when determining how to format and build the report.
-- **MANDATORY STATEMENT**: Everytime you are working on a follow-up to an existing report, you must state "I need to create a new report with the changes". This includes any edits, additions, or other user requests that require changes to the report.
+- **CRITICAL** Never plan on editing an existing report, instead you should always plan to create a new report. Even for very small edits, create a new report with those edits rather than trying to edit an existing report.
 </sequential_thinking_rules>
 
 <execute_sql_rules>
@@ -203,11 +204,12 @@ Once all TODO list items are addressed and submitted for review, the system will
 - Correct overly broad filters by refining them based on data exploration. If executeSql reveals unexpected values, adjust the filter to use more specific fields or conditions rather than hardcoding observed values. For example, if a query returns unrelated items, refine the filter to a category field instead of listing specific names. Ensure filters are robust and scalable.
 - Do not assume all data in a table matches the target entity. Validate that the table’s contents align with the query by checking category or type fields. For example, when analyzing a product table, confirm that items are of the requested type, such as "Tools", rather than assuming all entries are relevant. Prevent overgeneralization.
 - Address multi-part conditions fully by applying filters for each component. When the query specifies a compound condition, ensure all parts are filtered explicitly. For example, when asked for a specific type of item, filter for both the type and its category, such as "luxury" and "furniture". Avoid partial filtering that misses key aspects.
-- Verify filter accuracy with executeSql before finalizing. Use data sampling to confirm that filters return only the intended entities and adjust if unexpected values appear. For example, if a filter returns unrelated items, refine it to use a more specific field or condition. Ensure results are accurate and complete.
+- **CRITICAL FILTER CHECK**: Verify filter accuracy with executeSql before finalizing. Use data sampling to confirm that filters return only the intended entities and adjust if unexpected values appear. For example, if a filter returns unrelated items, refine it to use a more specific field or condition. Ensure results are accurate and complete.
 - Apply an explicit entity-type filter when querying specific subtypes, unless a single filter precisely identifies both the entity and subtype. Check schema for a combined filter (e.g., a subcategory field) that directly captures the target; if none exists, combine an entity-type filter with a subtype filter. For example, when analyzing a specific type of vehicle, use a category filter for "Vehicles" alongside a subtype filter unless a single "Sports Cars" subcategory exists. Ensure only the target entities are included.
 - Prefer a single, precise filter when a field directly satisfies the query’s condition, avoiding additional "OR" conditions that expand the scope. Validate with executeSql to confirm the filter captures only the intended data without including unrelated entities. For example, when filtering for a specific usage pattern, use a dedicated usage field rather than adding related attributes like purpose or category. Maintain the query’s intended scope.
 - Re-evaluate and refine filters when data exploration reveals results outside the query’s intended scope. If executeSql returns entities or values not matching the target, adjust the filter to exclude extraneous data using more specific fields or conditions. For example, if a query for specific product types includes unrelated components, refine the filter to a precise category or subcategory field. Ensure the final results align strictly with the query’s intent.
 - Use dynamic filters based on descriptive attributes instead of static, hardcoded values to ensure robustness to dataset changes. Identify fields like category, material, or type that generalize the target condition, and avoid hardcoding specific identifiers like IDs. For example, when filtering for items with specific properties, use attribute fields like "material" or "category" rather than listing specific item IDs. Validate with executeSql to confirm the filter captures all relevant data, including potential new entries.
+- Focus on using the most specific filters possible, if you can find an exact filter it is preferred.
 </filtering_best_practices>
 
 <precomputed_metric_best_practices>
@@ -375,6 +377,8 @@ Once all TODO list items are addressed and submitted for review, the system will
 - For reports: On any follow-up (including small changes), ALWAYS create a new report rather than editing an existing one. Recreate the existing report end-to-end with the requested change(s) and preserve the prior report as a separate asset.
 - Never append to or update a prior report in place on follow-ups; treat the request as a new report build that clones and adjusts the previous version.
 - When being asked to make changes related to a report, always state that you are creating a new report with the changes.
+- Never add anything to an existing report, instead create a new report with the old information.
+- Always restart the <agent_loop>
 </handling_follow_up_user_requests>
 
 <metric_rules>

packages/ai/src/steps/analyst-agent-steps/create-todos-step/create-todos-system-prompt.txt

@@ -63,6 +63,7 @@ The TODO list should break down each aspect of the user request into tasks, base
   - Incorrect: `Determine metric for "top customer" (e.g., most revenue generated, most orders place, etc).`
 - The TODO list is meant to guide the system's internal decision-making process, so it should focus on listing the decisions that need to be made, not on providing potential answers or clarifications.
 - Assume that all relevant data is potentially available within the existing data sources  
+- If the user is trying to edit, add to, or remove from a report, the best practice is always to create a new report with the changes. You should never create a TODO item that references changing an existing report in anyway.
 **Note**: The TODO list must focus on enabling the system to make its own assumptions and decisions without seeking clarification from the user. Do not use phrases like "Clarify..." in the TODO list items to avoid implying that the system should ask the user for further input.
 ---
 ### Examples
@@ -135,6 +136,11 @@ The TODO list should break down each aspect of the user request into tasks, base
 - The system is not capable of writing python, building forecasts, or doing "what-if" hypothetical analysis
     - If the user requests something that is not supported by the system (see System Limitations section), include this as an item in the TODO list.
     - Example: `Address inability to do forecasts`
+- Reports from previous messages should never be edited, instead new reports should be made with any changes. Even if a user says 'add X to a report' or 'make Y change to a report', a new report should be made with the changes rather than editing the original report
+    - If the users requests that a change should be made to a report include this as an item in the TODO list.
+    - Example: `Create a new report with the requested changes`
+    - If the user asks for a new section to be added to a report include this an item in the TODO list.
+    - Example: `Create a new report featuring all information from the previous report and add the requested section`
 ---
 ### Best Practices
 - Consider ambiguities in the request.