- Enhance message streaming with more precise content and tool call handling
- Add logic to only send and store meaningful assistant messages and tool calls
- Prevent sending empty or redundant messages during stream processing
- Improve tool call and content update tracking in agent stream method
- Optimize message inclusion in recursive thread generation
- Add support for initial non-tool response and subsequent tool-enabled processing
- Implement `PendingToolCall` to manage incremental tool call updates
- Update `Delta` and related types to support more flexible tool call streaming
- Modify agent stream processing to handle tool calls with improved state management
- Add robust handling for tool call deltas and function call arguments
- Move hardcoded agent prompt to a constant `AGENT_PROMPT` in the `agent_thread.rs` file
- Update the prompt's date from January 27, 2025 to February 7, 2025
- Simplify thread initialization by referencing the new constant
- Maintain existing prompt structure and guidelines
- Implement a maximum recursion depth of 30 for agent thread processing
- Add recursion depth tracking to prevent potential infinite loops
- Provide user-friendly message when maximum recursion depth is reached
- Update debug logging to include current recursion depth
- Modify both synchronous and streaming thread processing methods
- Refactor agent stream processing to use a recursive approach for handling tool calls
- Enhance tool execution with more robust error handling and result tracking
- Improve stream chunk processing with detailed state management
- Add support for recursive thread generation based on tool call results
- Implement cloning for LiteLLMClient to support stream processing tasks
- Update search data catalog and file search tools to use consistent JSON response structure
- Enforce strict response format with mandatory "results" key and comprehensive field requirements
- Add explicit guidelines for result composition and handling of empty result sets
- Improve response predictability and parsing reliability for AI-powered search tools
- Add `stream: Some(false)` to file search and data catalog tools
- Make `json_schema` optional in `ResponseFormat` serialization
- Enhance logging in search file tool with debug and warning messages
- Improve error context when parsing LLM JSON responses
- Wrap agent tools in an Arc for safe concurrent access
- Modify tool addition methods to work with Arc-wrapped HashMap
- Ensure thread-safe tool registration and retrieval
- Update stream processing to use Arc-cloned tools reference
- Add comprehensive debug logging for agent thread processing
- Modify agent thread to use user ID for streaming subscription
- Update stream processing to include detailed tool call and content logging
- Improve error handling and visibility in stream processing
- Add user context to agent thread initialization
- Remove redundant tool execution handling in stream processing
- Add `ValueToolExecutor` to convert tool outputs to `serde_json::Value`
- Introduce `IntoValueTool` trait for easy value type conversion
- Update agent tool addition methods to use new value conversion mechanism
- Simplify tool registration by automatically converting tool outputs
- Remove previous manual boxing and type conversion logic
- Add comprehensive dataset search functionality using LLM for intelligent dataset matching
- Implement search across datasets with relevance ranking based on YML content
- Create structured search result output with dataset metadata
- Add robust error handling, logging, and parsing for search operations
- Include test coverage for search result validation
- Enhance tool with flexible query parameter support and detailed response messages
- Add comprehensive file search functionality using LLM for intelligent file matching
- Implement search across metric and dashboard files with relevance ranking
- Create structured search result output with file metadata
- Add robust error handling and logging for search operations
- Include test coverage for search result parsing
- Implement robust line-based content modification for metric and dashboard files
- Add comprehensive error handling and validation for file modifications
- Improve modification tracking with detailed modification results
- Optimize file processing with batch insertion and validation
- Add extensive test coverage for modification and validation logic
- Add generic Output type to ToolExecutor trait
- Update file tools to use strongly-typed output structs
- Modify agent and tool implementations to support generic output
- Improve error handling and result reporting in file-related tools
- Add more detailed status messages for file operations
- Rename file type modules from `metric_file` and `dashboard_file` to `metric_yml` and `dashboard_yml`
- Modify metric files migration to use a verification enum instead of boolean
- Update messages_to_files junction table with UUID primary key and additional timestamp columns
- Adjust database models to support new file and message structures
- Refactor file creation utility to use new model structures
- Extend Dataset model and schema to include optional database_identifier field
- Update dataset creation and deployment routes to handle new database_identifier parameter
- Modify dataset DDL generation to use database_identifier for schema resolution when available
- Add support for parsing Snowflake timestamp structs with epoch and fraction fields
- Implement handling of Snowflake timestamp logical types (with and without timezone)
- Enhance JSON value processing to detect and convert Snowflake timestamp objects
- Add error handling and logging for timestamp parsing
* xAxisTickinerval
* fix(visualization): Add x-axis time unit configuration for bar and combo charts
- Extend chart configuration to support optional x-axis time unit
- Update modify visualization agent to dynamically set x-axis time interval
- Modify bar line and combo chart prompts to include x_axis_time_unit parameter
* only use valid time units
* refactor(visualization): Simplify x-axis time unit configuration
- Modify modify_visualization_agent to extract and remove x-axis time unit more efficiently
- Update global styling result structure for x-axis time interval
- Adjust format_label_prompt comment to clarify date format default behavior
---------
Co-authored-by: Nate Kelley <nate@buster.so>
- Restructure deploy_datasets function to separate concerns
- Improve column and relationship validation in dataset deployment
- Enhance error handling and validation result generation
- Add support for more comprehensive column and relationship checks
- Refactor validation logic to handle multiple error types
- Add support for capturing source type (table, view, materialized view)
- Improve column metadata queries for Postgres, MySQL, BigQuery, and Snowflake
- Include more comprehensive column information during dataset import
- Extend DatasetColumnRecord to include source_type field
- Add comprehensive column validation before dataset deployment
- Validate existence of all required columns in source database
- Simplify column type and nullability retrieval
- Enhance error reporting for missing columns
- Update deployment logic to use pre-validated column information
- Add detailed validation error logging in CLI
- Improve type compatibility checks in dataset validation
- Modify deployment process to handle and report validation errors more comprehensively
- Add Hash derive for Verification enum
- Update API and CLI to support more informative validation results
- Introduce helper functions for processing string and JSON values
- Implement case-insensitive string and JSON value transformations
- Add robust timestamp parsing with error handling
- Enhance Snowflake query result processing with consistent data normalization
- Refactor stored values processing in dataset deployment to use background task
- Add `StoredValueColumn` struct to encapsulate column processing details
- Implement `process_stored_values_background` for parallel and resilient value storage
- Add logging for successful and failed stored value processing
- Update CLI to handle optional SQL definitions and improve file processing
* fix: add stored values support for dataset columns
This commit introduces stored values functionality for dataset columns, including:
- Adding a `stored_values` flag to column deployment requests
- Implementing a mechanism to store column values during dataset deployment
- Updating data analyst and SQL generation agents to leverage stored values
- Creating a new utility module for stored values search and management
* refactor(stored_values): improve stored values implementation and schema management
This commit enhances the stored values functionality with several key improvements:
- Update schema and table creation to use organization ID as schema name
- Modify stored values storage to include column ID
- Improve value extraction and embedding generation process
- Remove unnecessary distance calculation in search results
- Clean up unused values_engine module
- Added implementation for creating metric files with database insertion
- Introduced `create_metric_file()` and `create_dashboard_file()` functions
- Updated `CreateFilesTool` to handle different file types
- Added `data_metadata` field to `MetricFile` struct
- Implemented basic file type validation and creation logic
- Added robust error handling for JSON parameter parsing in `CreateFilesTool`
- Updated file type description to clarify naming conventions
- Prepared for file creation logic implementation
- Moved file-related types to a new `file_types` module
- Removed redundant `types.rs` file
- Removed individual tool files for bulk file modifications, file creation, file opening, data catalog search, file search, and sending to user
- Created a new `file_tools` module in `tools/mod.rs` to centralize file-related tool implementations
- Commented out individual tool module imports in preparation for future implementation
- Simplified the tools module structure for better organization and maintainability
- Added environment variable-based LLM client initialization in `Agent::new()`
- Introduced `add_tool()` and `add_tools()` methods for more flexible tool registration
- Implemented new `get_name()` method for `ToolExecutor` trait
- Added comprehensive test cases for Agent with and without tools
- Updated `AgentThread` with a convenient constructor method
- Temporarily commented out unused tool modules
- Added debug print in LiteLLM client for response logging
- Renamed existing messages table to `messages_deprecated`
- Created new `messages` table with updated schema and additional indexes
- Updated Diesel schema to reflect new table structure and relationships
- Added new foreign key constraints for threads and users
- Prepared for migration of existing message data
- Created `messages_to_files` junction table to link messages with dashboard and metric files
- Added foreign key constraints and indexes for efficient file-message relationships
- Updated Diesel schema to include new `messages_to_files`, `dashboard_files`, and `metric_files` tables
- Removed unnecessary timestamp triggers from migration files
- Added support for retrieving and passing data source type during SQL agent generation
- Updated SQL generation prompts to include data source type in system messages
- Modified `generate_sql_agent` function to fetch and utilize data source type information
- Improved SQL generation context by dynamically incorporating data source type details
- Renamed `search_datasets.rs` to `search_data_catalog.rs`
- Updated `mod.rs` to reflect the new module and tool name
- Removed the placeholder `SearchDatasetsTool` implementation
- Prepared for future implementation of data catalog search functionality
- Relocated `ToolExecutor` trait from `agent/types.rs` to `tools/mod.rs`
- Updated import paths in `agent/agent.rs` to reflect new trait location
- Simplified `types.rs` by removing unused imports and trait definition
- Prepared for more comprehensive tool management in the tools module
- Renamed `Thread` to `AgentThread` for clarity
- Modified `ToolExecutor` to return `serde_json::Value` instead of `String`
- Updated message processing to handle new message structures
- Improved content handling in streaming and tool call scenarios
- Simplified message content extraction and serialization
- Uncommented and activated module implementations for `LiteLLMClient` and `Agent`
- Updated test cases to use new message and tool call structures
- Simplified test assertions for chat completion and tool call scenarios
- Restored module exports in `mod.rs` files
- Commented out imports and module exports for `LiteLLMClient` and `Agent`
- Removed active module implementations in `mod.rs` files
- Simplified type definitions and test cases
- Prepared for potential future reimplementation or restructuring
- Simplified message and tool structures in `types.rs`
- Updated `agent.rs` to use new type structures
- Enhanced error handling in agent tests with detailed error printing
- Updated `.cursorrules` with debugging recommendation
- Removed redundant fields and improved type flexibility
- Streamlined serialization and deserialization of AI client types
- Updated `types.rs` to support more complex message and response structures
- Added support for multi-content messages with `Content` struct
- Introduced optional fields and improved serialization handling
- Enhanced type flexibility for AI client responses
- Updated test cases to reflect new type structures
- Added `Default` implementation for `LiteLLMClient`
- Improved support for tool calls and streaming responses
- Updated `.env.example` with new LLM configuration variables
- Modified `LiteLLMClient` to support API key and base URL from environment variables
- Enhanced client initialization to use env vars with optional parameter overrides
- Added comprehensive test case for environment variable configuration
dal