diff --git a/packages/ai/src/steps/post-processing/flag-chat-step.ts b/packages/ai/src/steps/post-processing/flag-chat-step.ts index fdfa24940..fdb6c8054 100644 --- a/packages/ai/src/steps/post-processing/flag-chat-step.ts +++ b/packages/ai/src/steps/post-processing/flag-chat-step.ts @@ -52,9 +52,9 @@ const CREATE_FLAG_CHAT_SYSTEM_PROMPT = ` - Your role is to review the chat history between Buster and the user, identify signs of user frustration or issues, and flag chats for review by the data team. - The user only sees the final response and delivered assets (e.g., charts, dashboards), not intermediate steps or errors. - Your tasks include: - - Analyzing the chat history for signals of potential user frustration or issues. - - Flagging chats that meet the criteria for review. - - Providing a simple summary message for the data team's Slack channel when a chat is flagged. + - Analyzing the chat history for signals of potential user frustration or issues. + - Flagging chats that meet the criteria for review. + - Providing a simple summary message for the data team's Slack channel when a chat is flagged. @@ -99,10 +99,10 @@ Look for these signals indicating user frustration or issues: - Analyze Buster's thoughts for uncertainty or confusion. - Check for unresolved issues or incomplete tasks. - Identify major assumptions that could significantly impact results, such as: - - Introducing undefined concepts, metrics, segments, or filters. - - Choosing between similar fields or methods without clear guidance. - - Making decisions based on incomplete documentation. - - Assumptions where errors could substantially alter outcomes. + - Introducing undefined concepts, metrics, segments, or filters. + - Choosing between similar fields or methods without clear guidance. + - Making decisions based on incomplete documentation. + - Assumptions where errors could substantially alter outcomes. - Consider errors only if they affect the final response or assets. @@ -120,18 +120,18 @@ Flag the chat if any of these conditions are met: - If flagging the chat, use the \`flagChat\` tool to provide a summary and title. - - Include a 3-6 word title for the summary message. - - Write a simple summary message: - - Start with the user's first name and a brief, accurate description of their request (e.g., "Kevin requested a \"total count of customers\""). + - Include a 3-6 word title for the summary message. + - Write a simple summary message: + - Start with the user's first name and a brief, accurate description of their request (e.g., "Kevin requested a "total count of customers""). - Follow with a list of bullet points (each starting with "•") describing each issue and its implication. - Use two new lines between the intro sentence and the first bullet point, and one new line between bullet points. - Write in the first person as Buster, using 'I' to refer to yourself. - Use backticks for specific fields or calculations (e.g., \`sales.revenue\` or \`(# of orders delivered on or before due date) / (Total number of orders) * 100\`). - - Do not use bold, headers, or emojis in the title or summary. - - The title and summary should be written using a JSON string format. + - Do not use bold, headers, or emojis in the title or summary. + - The title and summary should be written using a JSON string format. - Example of \`flagChat\` fields: - - Summary Message: "Nate requested \"recent returns for Retail Ready customers with Canadian shipping addresses\".\n\n• Found no matching records.\n• The conversation history doesn't show a final response was sent. Likely encountered an error." - - Title: "No Final Response Sent" + - Summary Message: "Nate requested \"recent returns for Retail Ready customers with Canadian shipping addresses\".\n\n- Found no matching records.\n- The conversation history doesn't show a final response was sent. Likely encountered an error." + - Title: "No Final Response Sent" - If no issues, use the \`noIssuesFound\` tool. `; diff --git a/packages/ai/src/steps/post-processing/format-initial-message-step.ts b/packages/ai/src/steps/post-processing/format-initial-message-step.ts index 6a1eb8284..07866fadb 100644 --- a/packages/ai/src/steps/post-processing/format-initial-message-step.ts +++ b/packages/ai/src/steps/post-processing/format-initial-message-step.ts @@ -25,9 +25,9 @@ const initialMessageInstructions = ` - Your role is to review these assumptions and issues, then generate a concise summary for the data team via Slack. - The data team will use this summary to understand the user's request, assumptions made, issues encountered, and areas needing clarification in the documentation. - Your tasks: - - Analyze the flagged assumptions and issues. - - Provide a simple, direct summary message for the data team's Slack channel. - - Provide a 3-6 word title for the summary message. + - Analyze the flagged assumptions and issues. + - Provide a simple, direct summary message for the data team's Slack channel. + - Provide a 3-6 word title for the summary message. @@ -44,58 +44,48 @@ You operate in a loop: - Use the \`generateSummary\` tool to provide a summary and title. - - Include a 3-6 word title for the summary message. - - Write a simple summary message: - - Start with the user's first name and a brief, accurate description of their request (e.g., "Kevin requested a \"total count of customers\""). - - Follow with a list of bullet points (each starting with "•") describing each assumption or issue and its implication. + - Include a 3-6 word title for the summary message. + - Write a simple summary message: + - Start with the user's first name and a brief, accurate description of their request (e.g., "Kevin requested a "total count of customers""). + - Follow with a list of bullet points describing each assumption or issue and its implication. - Use two new lines between the intro sentence and the first bullet point, and one new line between bullet points. - Write in the first person as Buster, using 'I' to refer to yourself. - Use backticks for specific fields or calculations (e.g., \`sales.revenue\` or \`(# of orders delivered on or before due date) / (Total number of orders) * 100\`). - - Do not use bold, headers, or emojis in the title or summary. - - The title and summary should be written using a JSON string format. + - Do not use bold, headers, or emojis in the title or summary. + - The title and summary should be written using a JSON string format. Below are concise examples of summary messages and titles: - - Example #1 - - Summary Message: "Scott requested the \"total count of customers\".\n\n• I included all customer records, regardless of status (active, inactive, deleted). If incorrect, this likely inflates the count." + - Summary Message: "Scott requested the \"total count of customers\".\n\n- I included all customer records, regardless of status (active, inactive, deleted). If incorrect, this likely inflates the count." - Title: "Customer Count Includes All Statuses" - - Example #2 - - Summary Message: "John requested \"team IDs and company names for coverage AB tests starting January 15, 2025 or later\".\n\n• I assumed a coverage AB test is any test with treatments where \`RETURNS_ENABLED = true\`. If wrong, the analysis may be inaccurate." + - Summary Message: "John requested \"team IDs and company names for coverage AB tests starting January 15, 2025 or later\".\n\n- I assumed a coverage AB test is any test with treatments where \`RETURNS_ENABLED = true\`. If wrong, the analysis may be inaccurate." - Title: "Assumed Coverage AB Test Definition" - - Example #3 - - Summary Message: "Elisa requested \"merchants with HubSpot deals under $10k\".\n\n• Assumed deal amounts originate from HubSpot; if wrong, values are incorrect.\n• Assumed \`FIRST_CLOSED_WON_DEAL_AMOUNT\` is the correct field; if not, values are wrong.\n• Assumed to include only merchants with \`INCLUDE_IN_REVENUE_REPORTING = TRUE\`; this may exclude relevant merchants." + - Summary Message: "Elisa requested \"merchants with HubSpot deals under $10k\".\n\n- Assumed deal amounts originate from HubSpot; if wrong, values are incorrect.\n- Assumed \`FIRST_CLOSED_WON_DEAL_AMOUNT\` is the correct field; if not, values are wrong.\n- Assumed to include only merchants with \`INCLUDE_IN_REVENUE_REPORTING = TRUE\`; this may exclude relevant merchants." - Title: "HubSpot Data Assumptions" - - Example #4 - - Summary Message: "Nate requested \"recent returns for Retail Ready customers with Canadian shipping addresses\".\n\n• Found no matching records.\n• My conversation history doesn't show a final response was sent. Likely encountered an error." + - Summary Message: "Nate requested \"recent returns for Retail Ready customers with Canadian shipping addresses\".\n\n- Found no matching records.\n- My conversation history doesn't show a final response was sent. Likely encountered an error." - Title: "No Final Response Sent" - - Example #5 - - Summary Message: "Marcell requested \"total cost of labels paid for Target since using Resupply Inc\".\n\n• Assumed \`TOTAL_COST\` represents costs paid by Resupply Inc; if wrong, calculation is incorrect.\n• Assumed \`STG_SHIPMENT_INVOICES\` joins correctly to \`STG_FULFILLMENT_GROUPS\` via \`SHIPMENT_ID\`; if incorrect, data is likely missing in the results." + - Summary Message: "Marcell requested \"total cost of labels paid for Target since using Resupply Inc\".\n\n- Assumed \`TOTAL_COST\` represents costs paid by Resupply Inc; if wrong, calculation is incorrect.\n- Assumed \`STG_SHIPMENT_INVOICES\` joins correctly to \`STG_FULFILLMENT_GROUPS\` via \`SHIPMENT_ID\`; if incorrect, data is likely missing in the results." - Title: "Shipping Cost and Join Assumptions" - - Example #6 - - Summary Message: "Tiffany requested \"breakdown of Hint’s completed returns by type\".\n\n• Used \`return_line_item\` instead of \`return\`, which may skew percentages if returns have multiple line items.\n• Excluded other return types (\`repair\`, \`green_return\`, \`managed\`, etc) from the pie chart. Only shows returns with a \`completed\` status, making it redundant." + - Summary Message: "Tiffany requested \"breakdown of Hint’s completed returns by type\".\n\n- Used \`return_line_item\` instead of \`return\`, which may skew percentages if returns have multiple line items.\n- Excluded other return types (\`repair\`, \`green_return\`, \`managed\`, etc) from the pie chart. Only shows returns with a \`completed\` status, making it redundant." - Title: "Return Level and Chart Issues" - - Example #7 - - Summary Message: "Leslie requested \"users and their referral_ids for the Northwest team\".\n\n• Couldn't find \`referral_ids\` in the schema or documentation; returned only user list (without the requested \`referral_ids\`)." + - Summary Message: "Leslie requested \"users and their referral_ids for the Northwest team\".\n\n- Couldn't find \`referral_ids\` in the schema or documentation; returned only user list (without the requested \`referral_ids\`)." - Title: "Missing Referral IDs" - - Example #8 - - Summary Message: "Jacob requested \"overview of bike orders\".\n\n• Defined \"bike orders\" as orders with at least one bike (rather than bike-only orders or majority-bike orders); may be incorrect.\n• Calculated \`average bikes per order\` as mean across bike-containing orders; may differ from how this is calculated internally." + - Summary Message: "Jacob requested \"overview of bike orders\".\n\n- Defined \"bike orders\" as orders with at least one bike (rather than bike-only orders or majority-bike orders); may be incorrect.\n- Calculated \`average bikes per order\` as mean across bike-containing orders; may differ from how this is calculated internally." - Title: "Bike Order Definition Issues" - - Example #9 - - Summary Message: "Savanna requested \"analysis distinguishing competitive vs non-competitive cyclists\".\n\n• Assumed \`filter_purchase_motivation\` can be used to indicate if buyer is a competitive cyclist or not; if this is a poor assumption, the analysis is misleading.\n• Assumed \`Fitness\`, \`Recreation\`, and \`Transportation\` motivations indicate non-competitive behavior, classifying all other motivations as \"competitive\"; if this is a poor assumption, the analysis is misleading." + - Summary Message: "Savanna requested \"analysis distinguishing competitive vs non-competitive cyclists\".\n\n- Assumed \`filter_purchase_motivation\` can be used to indicate if buyer is a competitive cyclist or not; if this is a poor assumption, the analysis is misleading.\n- Assumed \`Fitness\`, \`Recreation\`, and \`Transportation\` motivations indicate non-competitive behavior, classifying all other motivations as \"competitive\"; if this is a poor assumption, the analysis is misleading." - Title: "Cyclist Classification Assumptions" - - Example #10 - - Summary Message: "Landen requested \"heat map of monthly sales by customer region\".\n\n• Heat maps are not supported; returned a table instead.\n• Didn't communicate heat map limitation in final response.\n• Assumed sales is calculated as \`SUM(subtotal + taxamt + freight)\` from \`sales_order_header\`; may differ from internal calculation method." + - Summary Message: "Landen requested \"heat map of monthly sales by customer region\".\n\n- Heat maps are not supported; returned a table instead.\n- Didn't communicate heat map limitation in final response.\n- Assumed sales is calculated as \`SUM(subtotal + taxamt + freight)\` from \`sales_order_header\`; may differ from internal calculation method." - Title: "Visualization and Calculation Issues" `;