Commit Graph

282 Commits

Author SHA1 Message Date
dal 743c256dbc
added in some testing work and changed some rules 2025-02-14 11:32:33 -07:00
dal e797f24e23
bug fix and prompt updates for o series 2025-02-14 09:19:43 -07:00
dal e68353b61d
now looking for optional database field. 2025-02-14 08:49:19 -07:00
dal 9e420c3201
hide cargo for now 2025-02-14 08:32:54 -07:00
dal 51b6d00159
message fkey error still propagating. 2025-02-14 08:32:06 -07:00
dal da8eb26931
silent on parsing errors since we will have a ton with the optimistic parsing. 2025-02-13 16:36:04 -07:00
dal eac2529f29
ok we are on our way there... 2025-02-13 16:28:59 -07:00
dal 3bd5516486
ok thread is being inserted 2025-02-13 16:15:09 -07:00
dal 977b5eb6de
mdc 2025-02-13 15:05:10 -07:00
dal 2d47c2c539
mdc commit 2025-02-13 14:50:24 -07:00
dal bdeaccdca8
cursor rules 2025-02-13 13:31:53 -07:00
dal 9fc5011b75
cursor rules 2025-02-13 08:38:14 -07:00
dal 5416418955
cargo optimization for faster builds 2025-02-13 08:31:14 -07:00
dal c6d70b62dc
ddl on query 2025-02-12 11:07:59 -07:00
dal 09ab45bbb5
cli release code 2025-02-12 09:00:02 -07:00
dal 37854342da
ok now generating descriptions ayo 2025-02-12 08:41:17 -07:00
dal 0c597e7b0f
snowflake type mapping correct 2025-02-12 08:19:38 -07:00
dal 59054cfefa
tweak to the diffing bug 2025-02-12 08:14:38 -07:00
dal 422b4c5da5
some temp fixes while we prep the new chat 2025-02-12 06:12:56 -07:00
dal 361c48fade
database_identifier during creation 2025-02-12 05:27:08 -07:00
dal ffaa373ec3
Merge remote-tracking branch 'origin/staging' into dal/simplify-deploy-endpoint 2025-02-12 05:25:09 -07:00
dal 61b6cfeffc
database mapping correct 2025-02-12 05:22:36 -07:00
dal c7a631de40
tweak on database from modelfiles 2025-02-12 05:04:34 -07:00
dal 2805d7ed70
implemented database name for snowflake warehouse. temp fix 2025-02-12 04:52:32 -07:00
dal 45739b73f2
get rid of unused code 2025-02-11 17:04:11 -07:00
dal 89912fed59
ITS WORKING 2025-02-11 16:57:44 -07:00
dal 29900149dd
almost there 2025-02-11 16:38:46 -07:00
dal 041859ec28
change thought_pills to thoughts 2025-02-11 15:45:05 -07:00
dal d17cb7e49a
bulk work on deploy 2025-02-11 15:43:07 -07:00
dal e7fedd0a59
working except for bulk in the validation 2025-02-11 15:21:53 -07:00
dal bf24b249d4
Refactor message and thread handling with explicit UUID references 2025-02-11 12:46:21 -07:00
dal 8e8140246d
Update message transformation to include chat and message IDs 2025-02-11 12:34:06 -07:00
dal 8b96ec01fb
Refactor message transformation with new container types and UUIDs 2025-02-11 11:53:06 -07:00
dal bf05c7f06b
Add new thread events for chat generation progress tracking 2025-02-11 11:21:57 -07:00
dal e7b96d9bd5
make a chart card 2025-02-11 10:58:25 -07:00
dal 41a7146a8d
Add initial chat response for new thread creation 2025-02-11 10:05:20 -07:00
dal 6fcfc4fba6
ids matching for stream messages 2025-02-11 09:57:19 -07:00
dal 2376153459
ids and initial message repeat handled 2025-02-11 09:36:28 -07:00
dal 973e9b41ce
ids are set 2025-02-11 09:21:08 -07:00
dal fcc15a0e6f
fix tool name transform path 2025-02-11 08:23:58 -07:00
dal 8940fbf3b6
using name as id for file type reasoning message 2025-02-11 08:21:32 -07:00
dal 8de08323fa
consistent message id for text stream 2025-02-11 08:10:58 -07:00
dal bcf1ac1a65
file stream is working 👍🏼 for create file. 2025-02-11 07:49:22 -07:00
dal afb40c90be
ok restructured the transformer to get vecs back 2025-02-11 07:40:07 -07:00
dal 90931fc029
Add support for chat routes and update thread route handling 2025-02-11 07:27:56 -07:00
dal d75931dcb0
We did it joe 2025-02-11 07:14:06 -07:00
dal 43e2cf44f4
its working, but not sure if it's how I want it... 2025-02-10 17:09:01 -07:00
dal 61153020ba
save point 2025-02-10 16:08:08 -07:00
dal 8c6c1d83ee
still working on the darn stream 2025-02-10 14:49:22 -07:00
dal fd65408e30
clean up to try different impl 2025-02-10 14:22:13 -07:00
dal 2593536efa
So close, just a few more tweaks to clean up stream. 2025-02-10 13:35:57 -07:00
dal 118ef9c691
ok streaming it back 2025-02-10 12:29:13 -07:00
dal 8c8372b50e
ok everything sending back except create and modify 2025-02-10 12:15:21 -07:00
dal d0400b5226
transforms for events 2025-02-10 11:53:19 -07:00
dal 233b580e1c
data catalog search transformation 2025-02-10 11:29:08 -07:00
dal 6a67931667
message transform 2025-02-10 10:47:16 -07:00
dal 93206fcf5c
added in duration and passed back req params 2025-02-10 09:21:07 -07:00
dal 9169a1c9e8
small tweaks 2025-02-10 08:26:25 -07:00
dal c849a22b4f
added in progress on tools 2025-02-10 08:24:56 -07:00
dal 8bd14e0ee7
added in progress on stream messages 2025-02-10 08:17:14 -07:00
dal 789b22fe1e
Improve message streaming and tool call processing
- Enhance message streaming with more precise content and tool call handling
- Add logic to only send and store meaningful assistant messages and tool calls
- Prevent sending empty or redundant messages during stream processing
- Improve tool call and content update tracking in agent stream method
- Optimize message inclusion in recursive thread generation
2025-02-10 07:30:52 -07:00
dal 641527114c
refactor(agent): Enhance LLM request handling with improved tool call processing
- Add support for initial non-tool response and subsequent tool-enabled processing
- Implement `PendingToolCall` to manage incremental tool call updates
- Update `Delta` and related types to support more flexible tool call streaming
- Modify agent stream processing to handle tool calls with improved state management
- Add robust handling for tool call deltas and function call arguments
2025-02-10 07:17:14 -07:00
dal 373a83efea
refactor(agent): Extract agent prompt to constant and update date
- Move hardcoded agent prompt to a constant `AGENT_PROMPT` in the `agent_thread.rs` file
- Update the prompt's date from January 27, 2025 to February 7, 2025
- Simplify thread initialization by referencing the new constant
- Maintain existing prompt structure and guidelines
2025-02-07 14:18:02 -07:00
dal 4b743fe5ec
feat(agent): Add recursion depth limit to prevent infinite processing
- Implement a maximum recursion depth of 30 for agent thread processing
- Add recursion depth tracking to prevent potential infinite loops
- Provide user-friendly message when maximum recursion depth is reached
- Update debug logging to include current recursion depth
- Modify both synchronous and streaming thread processing methods
2025-02-07 11:35:13 -07:00
dal 8b51618afd
refactor(agent): Implement recursive stream processing with improved tool execution
- Refactor agent stream processing to use a recursive approach for handling tool calls
- Enhance tool execution with more robust error handling and result tracking
- Improve stream chunk processing with detailed state management
- Add support for recursive thread generation based on tool call results
- Implement cloning for LiteLLMClient to support stream processing tasks
2025-02-07 11:22:43 -07:00
dal bb4e4ca9d8
refactor(ai_tools): Standardize search tool response format and requirements
- Update search data catalog and file search tools to use consistent JSON response structure
- Enforce strict response format with mandatory "results" key and comprehensive field requirements
- Add explicit guidelines for result composition and handling of empty result sets
- Improve response predictability and parsing reliability for AI-powered search tools
2025-02-07 11:12:08 -07:00
dal 8054bedf1a
fix(ai_tools): Update LLM request parameters and improve response handling
- Add `stream: Some(false)` to file search and data catalog tools
- Make `json_schema` optional in `ResponseFormat` serialization
- Enhance logging in search file tool with debug and warning messages
- Improve error context when parsing LLM JSON responses
2025-02-07 11:05:39 -07:00
dal 864257bc24
refactor(agent): Improve thread-safe tool management with Arc
- Wrap agent tools in an Arc for safe concurrent access
- Modify tool addition methods to work with Arc-wrapped HashMap
- Ensure thread-safe tool registration and retrieval
- Update stream processing to use Arc-cloned tools reference
2025-02-07 10:53:26 -07:00
dal 2bf27a9eda
feat(agent): Enhance agent thread processing with improved debugging and user-specific streaming
- Add comprehensive debug logging for agent thread processing
- Modify agent thread to use user ID for streaming subscription
- Update stream processing to include detailed tool call and content logging
- Improve error handling and visibility in stream processing
- Add user context to agent thread initialization
- Remove redundant tool execution handling in stream processing
2025-02-07 10:39:15 -07:00
dal a2f3433555
refactor(tools): Implement ValueToolExecutor for generic tool output conversion
- Add `ValueToolExecutor` to convert tool outputs to `serde_json::Value`
- Introduce `IntoValueTool` trait for easy value type conversion
- Update agent tool addition methods to use new value conversion mechanism
- Simplify tool registration by automatically converting tool outputs
- Remove previous manual boxing and type conversion logic
2025-02-07 10:15:55 -07:00
dal 2090e0b7d7
refactor(post_thread): Simplify thread creation with agent-based approach
- Remove complex AI-driven thread generation modules
- Introduce a new `AgentThreadHandler` to manage thread creation
- Streamline post thread logic by delegating to agent handler
- Remove deprecated SQL generation and AI-related modules
- Reduce code complexity and improve maintainability
2025-02-07 09:15:58 -07:00
dal adba6d6954
feat(data_catalog): Implement AI-powered dataset search tool
- Add comprehensive dataset search functionality using LLM for intelligent dataset matching
- Implement search across datasets with relevance ranking based on YML content
- Create structured search result output with dataset metadata
- Add robust error handling, logging, and parsing for search operations
- Include test coverage for search result validation
- Enhance tool with flexible query parameter support and detailed response messages
2025-02-07 08:08:37 -07:00
dal 372694bf1f
feat(file_search): Implement advanced AI-powered file search tool
- Add comprehensive file search functionality using LLM for intelligent file matching
- Implement search across metric and dashboard files with relevance ranking
- Create structured search result output with file metadata
- Add robust error handling and logging for search operations
- Include test coverage for search result parsing
2025-02-07 07:47:35 -07:00
dal 94c1635a34
refactor(file_tools): Enhance file modification and creation processes
- Implement robust line-based content modification for metric and dashboard files
- Add comprehensive error handling and validation for file modifications
- Improve modification tracking with detailed modification results
- Optimize file processing with batch insertion and validation
- Add extensive test coverage for modification and validation logic
2025-02-07 01:29:49 -07:00
dal 0e9075ca2c
Added Line Number Formatting on File outputs 2025-02-07 00:29:11 -07:00
dal 900eb28a67
removed bool on create_file tool 2025-02-06 23:54:11 -07:00
dal 711bbe899a
refactor(tools): Enhance ToolExecutor trait and file-related tools
- Add generic Output type to ToolExecutor trait
- Update file tools to use strongly-typed output structs
- Modify agent and tool implementations to support generic output
- Improve error handling and result reporting in file-related tools
- Add more detailed status messages for file operations
2025-02-06 23:45:48 -07:00
dal 4ec6e78648
refactor(database): Update metric and dashboard file models and migrations
- Rename file type modules from `metric_file` and `dashboard_file` to `metric_yml` and `dashboard_yml`
- Modify metric files migration to use a verification enum instead of boolean
- Update messages_to_files junction table with UUID primary key and additional timestamp columns
- Adjust database models to support new file and message structures
- Refactor file creation utility to use new model structures
2025-02-06 17:07:52 -07:00
dal b0e40007b0
Merge branch 'staging' into dal/goat-chat 2025-02-06 16:23:12 -07:00
dal 3ec059a99e
feat(prompts): Enhance SQL generation prompt with database identifier guidance
- Add instruction about paying attention to database identifier in SQL generation prompt
- Clarify potential cross-database referencing considerations
- Update SQL generation guidelines to improve clarity and flexibility
2025-02-06 16:21:53 -07:00
dal 33d5990907
feat(datasets): Add database_identifier support for dataset creation and deployment
- Extend Dataset model and schema to include optional database_identifier field
- Update dataset creation and deployment routes to handle new database_identifier parameter
- Modify dataset DDL generation to use database_identifier for schema resolution when available
2025-02-06 15:18:40 -07:00
dal c7a6e2788a
Merge branch 'staging' into dal/simplify-deploy-endpoint 2025-02-06 15:00:23 -07:00
dal 2ec0b7743f
feat(snowflake): Improve timestamp handling and JSON processing
- Add support for parsing Snowflake timestamp structs with epoch and fraction fields
- Implement handling of Snowflake timestamp logical types (with and without timezone)
- Enhance JSON value processing to detect and convert Snowflake timestamp objects
- Add error handling and logging for timestamp parsing
2025-02-06 14:09:39 -07:00
dal ef685d87a4
Fix x axis intervals (#105)
* xAxisTickinerval

* fix(visualization): Add x-axis time unit configuration for bar and combo charts

- Extend chart configuration to support optional x-axis time unit
- Update modify visualization agent to dynamically set x-axis time interval
- Modify bar line and combo chart prompts to include x_axis_time_unit parameter

* only use valid time units

* refactor(visualization): Simplify x-axis time unit configuration

- Modify modify_visualization_agent to extract and remove x-axis time unit more efficiently
- Update global styling result structure for x-axis time interval
- Adjust format_label_prompt comment to clarify date format default behavior

---------

Co-authored-by: Nate Kelley <nate@buster.so>
2025-02-06 11:58:25 -08:00
dal c27c27a7e8
fix: Improve column reference validation in model expression checks
- Update column validation to use model-defined columns instead of dataset columns
- Enhance error message to clarify column reference context
- Refine validation logic for expression column references
2025-02-06 10:46:12 -07:00
dal 4e2e9795b6
refactor: Simplify dataset deployment and validation process
- Restructure deploy_datasets function to separate concerns
- Improve column and relationship validation in dataset deployment
- Enhance error handling and validation result generation
- Add support for more comprehensive column and relationship checks
- Refactor validation logic to handle multiple error types
2025-02-05 23:42:51 -07:00
dal 08ecb44de1
Merge branch 'staging' into dal/simplify-deploy-endpoint 2025-02-05 21:56:45 -07:00
dal e144377ada
Tweaked the fix sql to return a json output so we don't get parse errors. 2025-02-05 21:55:19 -07:00
dal b872cf63a4
feat(prompts): Enhance SQL query generation and error handling instructions
- Update failed SQL fix prompt to emphasize query output format
- Add clarification to dataset selector prompt about selecting multiple datasets
2025-02-05 18:24:34 -07:00
dal fa480f6797
feat: enhance column metadata retrieval across database sources
- Add support for capturing source type (table, view, materialized view)
- Improve column metadata queries for Postgres, MySQL, BigQuery, and Snowflake
- Include more comprehensive column information during dataset import
- Extend DatasetColumnRecord to include source_type field
2025-02-05 18:21:40 -07:00
dal 6e5c299389
feat: improve dataset column validation and deployment process
- Add comprehensive column validation before dataset deployment
- Validate existence of all required columns in source database
- Simplify column type and nullability retrieval
- Enhance error reporting for missing columns
- Update deployment logic to use pre-validated column information
2025-02-05 17:20:11 -07:00
dal f081f3e16e
feat: enhance dataset validation and deployment error handling
- Add detailed validation error logging in CLI
- Improve type compatibility checks in dataset validation
- Modify deployment process to handle and report validation errors more comprehensively
- Add Hash derive for Verification enum
- Update API and CLI to support more informative validation results
2025-02-05 17:04:13 -07:00
dal fb75c1f554
fix(search): Add organization_id filter to semantic and terms search queries 2025-02-05 16:23:24 -07:00
dal 7bcd7d81bc
make sure the output of fix sql is delimited 2025-02-05 16:19:23 -07:00
dal 3c82ac0774
feat: add dataset validation and improved deployment process 2025-02-05 15:00:52 -07:00
dal 2c7ef16956
fix the search value table 2025-02-05 12:30:51 -07:00
dal d9973a13dd
bugfix(datasets): add delete dataset route
Implement a new DELETE route for removing datasets by their ID
2025-02-05 11:58:43 -07:00
dal 2be7383656
you learn something new every day... a schema in pg can't start with a number. 2025-02-05 11:36:31 -07:00
dal 960c89ab84
fix: janky check for values 2025-02-04 17:26:57 -07:00
dal 87a6225f1d
remove the limit query bc mixing things up 2025-02-04 17:18:02 -07:00