- Improved the `closeIncompleteJson` function to handle empty inputs and incomplete values more robustly, ensuring better completion of JSON structures.
- Enhanced regex patterns for extracting values to accommodate scientific notation and various boolean representations.
- Added comprehensive tests for edge cases, including special characters, escape sequences, and deeply nested structures, to ensure reliability in parsing.
- Introduced stress tests to validate performance with large datasets and complex scenarios, ensuring the parser can handle high loads effectively.
- Implemented retry mechanism for SQL execution and validation in `execute-sql`, `create-metrics-file-tool`, and `modify-metrics-file-tool` to handle transient errors and timeouts.
- Improved error handling and logging for better debugging and user feedback.
- Updated `ChunkProcessor` to ensure proper handling of SQL statements, including parsing JSON strings and filtering valid statements.
- Adjusted connection pool settings in `SnowflakeAdapter` to support higher concurrency and load management.
- Added `normalizeEscapedText` and related functions to handle double-escaped characters in streaming text, preventing issues during chunk processing.
- Updated `ChunkProcessor` to utilize the new normalization functions for thought and thoughts messages, ensuring proper formatting.
- Introduced comprehensive tests for escape normalization functionality, verifying correct handling of various escape sequences.
- Enhanced the streaming utility exports to include normalization functions for broader accessibility.
- Introduced logic to defer the doneTool response when completed files exist, ensuring proper sequencing of response history.
- Added a new method, `addFileAndDoneToolResponses`, to combine file messages and the deferred doneTool response.
- Updated tests to verify the correct behavior of deferred responses and ensure robust handling of file processing.
- Enhanced type safety and clarity in the ChunkProcessor class with additional checks for completed files.
- Updated the `extractFilesFromReasoning` function to utilize test-specific types for improved error handling and validation.
- Added comprehensive tests to ensure files are correctly excluded based on failure indicators and missing properties.
- Refactored test cases to simulate various error conditions, enhancing robustness in file extraction logic.
- Improved type safety by integrating new test types for reasoning messages and files.
- Introduced TypeScript types and enums to represent the DataMetadata structure, including ColumnMetaData and its properties.
- Implemented the createDataMetadata function to analyze query results and generate metadata, calculating min/max values and unique counts for each column.
- Updated metric file tools to utilize the new createDataMetadata function for improved data handling and reporting.
- Enhanced type safety and clarity in the metrics and dashboards file tools by integrating the new metadata analysis.
- Introduced multi-layer safety checks in `extractFilesFromReasoning` to prevent inclusion of files from entries with failure indicators.
- Added validation functions to check for failure indicators at both entry and file levels, ensuring only valid files are processed.
- Enhanced logging for rejected files to aid in debugging and traceability.
- Updated tests to verify the new failure detection logic, ensuring robustness in file extraction.
- Introduced functions to convert `MetricYml` and `DashboardYml` from API format (snake_case) to storage format (camelCase).
- Updated `createMetricVersion` and `createDashboardVersion` to store content directly in the new format, removing the enum wrapper.
- Removed generic version management functions in favor of specific functions for metrics and dashboards, enhancing clarity and maintainability.
- Added new storage format schemas for `MetricYml` and `DashboardYml` to ensure type safety and consistency.
- Updated version history functions to use unified types for metrics and dashboards, enhancing consistency and type safety.
- Renamed and refactored functions to improve clarity, including `createMetricVersion`, `createDashboardVersion`, and their respective history functions.
- Introduced validation functions for `MetricYml` and `DashboardYml` schemas to ensure data integrity.
- Streamlined version management by consolidating version entry structures and updating related types in `version-history-types.ts`.
- Introduced comprehensive tests for the `createFileResponseMessages` function to ensure it uses actual file IDs instead of generating random UUIDs.
- Verified consistent ID mapping across multiple calls and handled edge cases with various ID formats.
- Ensured that file metadata structure is preserved with actual IDs, demonstrating the fix for ID mismatch issues.
- Updated `createFileResponseMessages` to use actual file IDs instead of generating new UUIDs.
- Added methods in `ChunkProcessor` to handle file creation/modification tools, updating dummy IDs with actual IDs from tool results.
- Introduced error handling and logging for ID extraction and updating processes.
- Improved type safety and clarity in file ID management across relevant components.
- Replaced manual version history handling with helper functions to streamline the creation and modification of version histories for dashboards and metrics.
- Updated `createDashboardFiles` and `modifyDashboardFiles` to utilize `createInitialDashboardVersionHistory` and `addDashboardVersionToHistory`.
- Enhanced `createMetricFiles` and `modifyMetricFiles` to leverage `createInitialMetricVersionHistory` and `addMetricVersionToHistory`.
- Improved type safety and clarity in version history handling across relevant tools.
- Updated the retryableAgentStream and retryableAgentStreamWithHealing functions to remove console logs and replace them with comments for better readability.
- Enhanced integration tests for Analyst and Think and Prep agents to ensure proper handling of conversation history and response structures.
- Improved type definitions in various test files for better type safety and clarity.
- Refactored message accumulation and deduplication tests to ensure accurate message processing and verification.
- Reformatted the query method signatures in various database adapters (MySQL, SQL Server, Snowflake, etc.) to enhance code clarity and maintainability.
- Updated related SQL execution logic to ensure consistent handling of parameters, including optional timeout and maxRows.
- Adjusted unit tests to reflect changes in method signatures, ensuring comprehensive coverage and accuracy.
- Added timeout parameter to the query method in various database adapters (MySQL, PostgreSQL, SQL Server, etc.) to prevent long-running queries from hanging.
- Implemented timeout handling logic in each adapter to ensure queries respect the specified timeout duration, defaulting to 30 seconds if not provided.
- Updated the validation tool to include timeout in SQL query execution for improved performance and reliability.
Nate Kelley
dal