- Added shortcuts routes to the API.
- Enhanced chat handler to process shortcuts in user prompts.
- Introduced shortcuts table schema in the database.
- Updated relevant queries and shared types to include shortcuts.
- Configured environment variables for database connection.
- Enhanced the CLI with new linter rules and updated file structures for better organization.
- Modified package.json to reflect changes in dependencies and ensure compatibility.
- Improved the vitest configuration for better testing practices.
- Refactored various components for consistency and readability, including imports and error handling.
- Updated the server API and database queries to streamline authentication processes.
- Removed deprecated code and cleaned up unused files to maintain a tidy codebase.
- Added new CLI commands for authentication management and improved default action handling.
- Updated package.json to include new dependencies and scripts for building and running the CLI.
- Integrated authentication features into the SDK, allowing for API key validation.
- Modified the SDK configuration to use a new default API URL.
- Expanded the server API to include authentication routes.
- Updated database queries to support new API key functionalities.
- Improved code structure and readability across CLI and SDK components.
- Added routes for creating, retrieving, and deleting S3 integrations in the API.
- Introduced handlers for S3 integration operations, including validation of user permissions and storage credentials.
- Updated database schema to support S3 integrations, including a new table and associated queries.
- Integrated storage provider logic to handle S3, R2, and GCS configurations.
- Enhanced error handling and response structures for integration operations.
This commit lays the groundwork for managing storage integrations within the application, allowing users to connect and manage their S3 storage solutions.
- Added checks to ensure toolCallId exists and is a valid string before processing tool results.
- Preserves the original part if toolCallId is missing or invalid, improving robustness in message conversion.
- Updated logic to prioritize cache as the source of truth during streaming, with asynchronous database updates for persistence.
- Improved error handling for background database updates, ensuring cache validity even if DB update fails.
- Cleaned up mergeResponseMessages and mergeReasoningMessages functions by removing redundant code.
- Introduced lru-cache version 11.1.0 in package.json and pnpm-lock.yaml.
- Refactored updateMessageEntries to utilize TypeScript-based merge logic with write-through caching.
- Improved error handling for fetching existing message entries and updated database logic to merge new entries with existing ones.
- Updated GET reports endpoint to use `getReportsWithPermissions` for improved access control.
- Added support for filtering reports based on `shared_with_me` and `only_my_reports` flags.
- Introduced `getReportsWithPermissions` function to handle complex permission logic in the database layer.
- Enhanced report metadata retrieval to ensure proper access validation before fetching report details.
- Simplified the `GeneratingContent` component to display a static loading message.
- Updated `ReportPageController` to determine if a report is being generated based on the current message context.
- Adjusted report status handling in the delta creation process to maintain 'loading' status during streaming.
- Enhanced logic in the modify reports helper to ensure file status reflects loading during streaming and only updates to completed or failed when all edits are finalized.
- Modified database query to preserve input order for new messages during updates.
- Added '@buster/data-source' dependency to the server and pnpm lock files.
- Enhanced the getMetricDataHandler function to support versioning and data source integration.
- Updated the metric data query schema to include an optional version number parameter.
- Modified the API endpoint to accommodate the new versioning feature for metric data retrieval.
- Changed the wording of elapsed time in reasoning messages from "min" to "minutes" for better readability.
- Updated the final reasoning message format to specify "Reasoned for" instead of "Total workflow time," enhancing user understanding of the output.
These changes contribute to a more user-friendly experience and maintain consistency in messaging across different tools.
- Added permission checks to metric file download handler
- Added permission checks to export-metric-data trigger task
- Created comprehensive tests for access control integration
- Updated AssetType enums to include report_file
- Resolved export conflicts in server-shared package
- Fixed AI package imports for new getPermissionedDatasets API
- Introduced start time tracking for tool operations in create, modify, and report tools to improve performance monitoring.
- Updated reasoning entries to include elapsed time for completed operations, enhancing user feedback.
- Refactored tool logic to ensure consistent handling of file statuses and improved clarity in reasoning messages.
- Enhanced logging for better traceability during tool execution.
These changes improve the overall user experience and maintainability of the toolset.
- Combined asset type enums from both branches
- Merged all database query exports
- Integrated access-controls package into server-shared
- Resolved conflicts in AI package instructions
- Updated type definitions for better clarity and type safety in metric file download and export tasks.
- Enhanced error handling in the metric files API to utilize HTTPException for more consistent error responses.
- Added environment variable validation in export tasks to ensure necessary credentials are present before execution.
- Improved type assertions for content in metric export queries to ensure proper handling of JSONB data.
- Introduced new API endpoint for downloading metric files as CSV.
- Implemented export logic to handle large datasets (up to 1 million rows) and generate presigned URLs for secure downloads.
- Added cleanup task to remove exported files from R2 storage after 60 seconds.
- Updated environment configuration to include Cloudflare R2 credentials.
- Enhanced error handling for various export scenarios.
- Refactored related database queries and schemas for better integration.
- Updated documentation for new features and usage instructions.
- Introduced a new streaming mechanism for generating TODO lists, allowing for real-time updates and improved user experience.
- Added context handling for datasets in both `AnalystAgent` and `ThinkAndPrepAgent`, ensuring relevant data is included in system messages.
- Implemented new helper functions for managing TODO state and reasoning messages, improving code organization and clarity.
- Updated integration tests to validate the new streaming functionality and ensure consistent message roles.
These changes significantly enhance the functionality and maintainability of the TODO creation process within the agent workflow.
- Eliminated `responseMessages`, `reasoning`, and `rawLlmMessages` from chat creation and handling functions to streamline data processing.
- Updated the `generateJsonbArraySql` function to accept pre-stringified JSON for better performance and clarity.
These changes enhance the efficiency of chat message management and reduce unnecessary complexity in the codebase.
Introduce 'replace-report-content' and 'append-report-content' exports to enhance report content management capabilities, allowing for more flexible report updates and modifications.
- Optimized N+1 query in get-permissioned-datasets.ts using inArray for batch lookup
- Removed unused bulkRemoveAssetPermissions import
- Fixed import organization in find-user-by-email.ts
- Updated CLAUDE.md to reflect tests are written and fixed API example
- Clarified TODO comment in lookup.ts to prevent potential infinite recursion
These are style and performance improvements that don't affect functionality.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed permission hierarchy: full_access (5) now correctly ranks higher than viewer (3)
- Removed 'querier' role from admin permission check as it was too permissive
- Added organizationId parameter to dashboard access checks to prevent unauthorized access
- Fixed public dashboard expiry date check to verify dates are in the future
- Added cycle detection for collection permissions to prevent infinite recursion
- Fixed bulk remove permissions to correctly filter by identityIds array
- Updated date comparison to use ISO string format for PostgreSQL timestamp columns
These fixes address critical security and logic issues identified in the PR review.
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Fixed AssetTypeSchema to include all active asset types (excluding deprecated 'metric' and 'dashboard')
- Added missing 'restricted_querier' role to UserOrganizationRole type
- Fixed dataset access caching to store full result object including accessPath and userRole
- Updated count query to use SQL COUNT for better performance
- Fixed IdentityType consistency across dataset permissions
- Removed unused 'ne' import from list-asset-permissions.ts
- Updated comments to correctly reference 6 access paths instead of 5
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
This PR migrates the Rust and libraries to TypeScript in the new package.
## Changes
### New Access Controls Package
- Implemented complete asset permissions system with CRUD operations
- Added dataset permissions with all 5 access paths (admin, direct user, user→group, team→direct, team→group, org→default)
- Implemented cascading permissions (metrics inherit from dashboards/chats/collections)
- Added LRU caching to replace Redis (30-second TTL)
- Comprehensive test coverage (148 tests passing, 3 skipped pending createUser)
### Database Package Updates
- Fixed AssetType enum to include all 14 asset types
- Added missing database queries for permissions
- Fixed timestamp fields to use ISO strings
- Added getUserOrganizationsByUserId function
- Created organized query structure by table
### AI Package Updates
- Updated getPermissionedDatasets API calls to new format
- Changed property references from ymlFile to ymlContent
- Fixed all test mocks to match new API structure
### Trigger App Updates
- Updated to use new access controls API
- Fixed test mocks to match DatasetListResult structure
- Added proper type imports
### Server Shared Updates
- Added request/response types for access control endpoints
- Exported types for API consistency
## Testing
- All packages build successfully with turbo build:dry-run
- All tests pass with turbo test:unit
- Full type safety maintained throughout
## Migration Notes
- Drop-in replacement for Rust libraries
- Same permission model and roles
- Cache invalidation strategies implemented
- Soft delete pattern maintained
🤖 Generated with [Claude Code](https://claude.ai/code)
Co-Authored-By: Claude <noreply@anthropic.com>
- Add server-shared types for GetTitleRequest and GetTitleResponse with Zod validation
- Create database query functions for each asset type (chat, metric, collection, dashboard)
- Implement GET endpoint with zValidator middleware and exhaustive switch statement
- Add proper permission checks (publiclyAccessible OR organizationId match)
- Export new functions from database query index files
- Add title route to v2 API index
- Follow existing Hono API patterns with standardErrorHandler
Co-Authored-By: nate@buster.so <nate@buster.so>
- Make installationId and githubOrgId NOT NULL (required GitHub API fields)
- Change user foreign key to ON DELETE SET NULL to prevent blocking user deletion
- Add index on github_org_id for better query performance
- Maintain unique constraint on organization_id + installation_id pair
Addresses greptile-apps review comments on PR #574
Co-Authored-By: Dallin Bentley <dallinbentley98@gmail.com>
- Introduced a new function `checkForDuplicateMessages` in the database queries to verify if a message already exists in the same chat.
- Integrated the duplicate check in the `slackAgentTask` to prevent processing of duplicate messages, enhancing task efficiency and logging warnings for duplicates detected.
- Fixed AI package unit tests that were previously failing
- Updated database package.json with environment variables for tests
- Fixed snowflake adapter test issues in data-source package
Co-Authored-By: Dallin Bentley <dallinbentley98@gmail.com>
- Added '@buster/ai' as a workspace dependency in pnpm-lock.yaml and package.json.
- Updated database-migrations.yml to trigger on changes in the database package.
- Refined the cancelChatHandler function to include detailed message cleanup and trigger cancellation logic.
- Improved response handling in the chat cancellation endpoint to return a success message.
- Enhanced updateMessageFields to support marking messages as completed.
- Introduced `updateDefaultChannel` method in Slack helpers to manage default channel settings.
- Updated SlackHandler to utilize the new method for updating integration settings.
- Enhanced integration tests to cover new functionality and ensure proper handling of active integrations.
- Removed obsolete SQL migration for default channel as it is now managed in code.
* type fixes
* biome clean on ai
* add user to flag chat
* attempt to get vercel deployed
* Update tsup.config.ts
* Update pnpm-lock.yaml
* Add @buster/server2 Hono API app with Vercel deployment configuration
* slack oauth integration
* mainly some clean up and biome formatting
* slack oauth
* slack migration + snapshot
* remove unused files
* finalized docker image for porter
* Create porter_app_buster-server_3155.yml file
* Add integration tests for Slack handler and refactor Slack OAuth service
- Introduced integration tests for the Slack handler, covering OAuth initiation, callback handling, and integration status retrieval.
- Refactored Slack OAuth service to improve error handling and ensure proper integration state management.
- Updated token storage implementation to use a database vault instead of Supabase.
- Enhanced existing tests for better coverage and reliability, including cleanup of test data.
- Added new utility functions for managing vault secrets in the database.
* docker image update
* new prompts
* individual tests and a schema fix
* server build
* final working dockerfile
* Update Dockerfile
* new messages to slack messages (#369)
* Update dockerfile
* Update validate-env.js
* update build pipeline
* Update the dockerfile flow
* finalize logging for pino
* stable base
* Update cors middleware logger
* Update cors.ts
* update docker to be more imformative
* Update index.ts
* Update auth.ts
* Update cors.ts
* Update cors.ts
* Update logger.ts
* remove logs
* more cors updates
* build server shared
* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies. (#370)
* tons of file parsing errors (#371)
* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies.
* tons of file parsing errors
* Dev mode updates
* more stable electric handler
* Dal/agent-self-healing-fixes (#372)
* change to 6 min
* optmizations around saving and non-blocking actions.
* stream optimizations
* Dal/agent-self-healing-fixes (#373)
* change to 6 min
* optmizations around saving and non-blocking actions.
* stream optimizations
* change porter staging deploy to mastra-braintrust.
* new path for porter deploy
* deploy to staging fix
* Create porter_app_mastra-braintrust-api_3155.yml file (#375)
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
* Update sizing and opacity
* supe up the instance for mastra
* environment staging
* ssl script
* copy path
* Update list padding
* no throttle and the anthropic cached
* move select to the top
* Update margin inline start
* shrink reasoning vertical space to 2px
* semi bold font for headers
* update animation timing
* haiku
* Add createTodoList tool and integrate into create-todos-step
* chat helper on post chat
* only trigger cicd when change made
* Start created streaming text components
* Refactor analyst agent task to initialize Braintrust logging asynchronously and parallelize database queries for improved performance. Adjusted cleanup timeout for Braintrust traces to reduce delays.
* fixed reasoned for X, so that it rounds down to the minute
* Update users page
* update build pipeline for new web
* document title update
* Named chats for page
* Datasets titles
* Refactor visualization tools and enhance error handling in retryable agent stream. Removed unused metricValueLabel from metrics file tool, updated metric configuration schemas, and improved healing mechanism for tool errors during streaming.
* analyst
* document title updates
* Update useDocumentTitle.tsx
* Refactor tool choice configuration in create-todos-step to use structured object. Remove exponential backoff logic from retryable agent stream for healable errors. Introduce new test for real-world healing scenarios in retryable agent stream.
* Refactor SQL validation logic in modify-metrics-file-tool to skip unnecessary checks when SQL has not changed. Enhance error handling and update validation messages. Clean up code formatting for improved readability.
* update collapse for filecard
* chevron collapse
* Jacob prompt changes (#376)
* prompt changes to improve filtering logic and handle priv/sec errors
* prompt changes to make aggregation better and improved filter best practices
* Update packages/ai/src/steps/create-todos-step.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update packages/ai/src/agents/think-and-prep-agent/think-and-prep-instructions.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* Update packages/ai/src/steps/create-todos-step.ts
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
---------
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: dal <dallin@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
* think and prep
* change header and strong fonts weights
* Update get collection
* combo chart x axis update
* Create a chart schemas as types
* schema types
* simple unit tests for line chart props
* fix the response file ordering iwth active selection.
* copy around reasoning messages taken care of
* fix nullable user message and file processing and such.
* update ticks for chart config
* fix todo parsing.
* app markdown update
* Update splitter to use border instead of width
* change ml
* If no file is found we should auto redirect
* Refactor database connection handling to support SSL modes. Introduced functions to extract SSL parameters and manage connections based on SSL requirements, including a custom verifier for unverified connections.
* black box message update
* chat title updates
* optimizations for trigger.
* some keepalive logic on the anthropic cached
* keep title empty until new one
* no duplicate messages
* null user message on asset pull
* posthog error handling
* 20 sec idle timeout on anthropic
* null req message
* fixed modificiation names missing
* Refactor tool call handling to support new content array format in asset messages and context loaders
* cache most recent file from workflow
* Enhance date and number detection in createDataMetadata function to improve data type handling for metrics files
* group hover effect for message
* logging for chat
* Add messageId handling and file association tracking in dashboard and metrics tools
- Updated runtime context to include messageId in create and modify dashboard and metrics file tools.
- Implemented file association tracking based on messageId in create and modify functions for both dashboards and metrics.
- Ensured type consistency by using AnalystRuntimeContext in runtime context parameters.
* logging for chat
* message type update
* Route to first file instead
* trigger moved to catalog
* Enhance file selection logic to support YAML parsing and improve logging
- Updated `extractMetricIdsFromDashboard` to first attempt JSON parsing, falling back to a regex-based YAML parsing for metric IDs.
- Added detailed debug logging in `selectFilesForResponse` to track file selection process, including metrics and dashboards involved.
- Introduced tests for various scenarios in `file-selection.test.ts` to ensure correct behavior with dashboard context and edge cases.
* trigger dev v4-beta
* Retry + Self Healing (#381)
* Refactor retry logic in analyst and think-and-prep steps
Co-authored-by: dallin <dallin@buster.so>
* some fixes
* console log error
* self healing
* todos retry
---------
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* remove lots of logs
* Remove chat streaming
* Remove chat streaming
* timeout
* Change to updated at field
* link to home
* Update timeout settings for HTTP and HTTPS agents from 20 seconds to 10 seconds for improved responsiveness.
* Add utils module and integrate message conversion in post_chat_handler
* Implement error handling for extract values (#382)
* Remove chat streaming
* Improve error handling and logging in extract values and chat title steps
Co-authored-by: dallin <dallin@buster.so>
---------
Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
* loading icon for buster avatar
* finalize tooltip cache
* upgrade mastra
* increase retries
* Add redo functionality for chat messages
- Introduced `redoFromMessageId` parameter in `handleExistingChat` to allow users to specify a message to redo from.
- Implemented validation to ensure the specified message belongs to the current chat.
- Added `softDeleteMessagesFromPoint` function to soft delete a message and all subsequent messages in the same chat, facilitating the redo feature.
* fix electric potential memory leak
* tooltip cache and chart cleanup
* Update bullet to be more indented
* latest version number
* add support endpoint to new server
* Fix jank in combo bar charts
* index check for dashboard
* Collapse only if there are metrics
* Is finished reasoing back
* Update dependencies and enhance chat message handling
- Upgraded `@mastra/core` to version 0.10.8 and added `node-sql-parser` at version 5.3.10 in the lock file.
- Improved integration tests for chat message redo functionality, ensuring correct behavior when deriving `chat_id` from `message_id`.
- Enhanced error handling and validation in the `initializeChat` function to manage cases where `chat_id` is not provided.
* Update pnpm-lock and enhance chat message integration tests
- Added `node-sql-parser` version 5.3.10 to dependencies and updated the lock file.
- Improved integration tests for chat message redo functionality, ensuring accurate deletion and retrieval of messages.
- Enhanced the `initializeChat` function to derive `chat_id` from `message_id` when not provided, improving error handling and validation.
* remove .env import breaking build
* add updated at to the get chat handler
* zmall runtime error fix
* permission tests passing
* return updated at on the get chat handler now
* slq parser fixes
* Implement chat access control logic and add comprehensive tests
- Developed the `canUserAccessChat` function to determine user access to chats based on direct permissions, collection permissions, creator status, and organizational roles.
- Introduced helper functions for checking permissions and retrieving chat information.
- Added integration tests to validate access control logic, covering various scenarios including direct permissions, collection permissions, and user roles.
- Created unit tests to ensure the correctness of the access control function with mocked database interactions.
- Included simple integration tests to verify functionality with existing database data.
* sql parser and int tests working.
* fix test and lint issues
* comment to kick off deployment lo
* access controls on datasets
* electric context bug fix with sql helpers.
* permission and read only
* Add lru-cache dependency and export cache management functions
- Added `lru-cache` as a dependency in the access-controls package.
- Exported new cache management functions from `chats-cached` module, including `canUserAccessChatCached`, `getCacheStats`, `resetCacheStats`, `clearCache`, `invalidateAccess`, `invalidateUserAccess`, and `invalidateChatAccess`.
* packages deploy as well
* wrong workflow lol
* Update AppVerticalCodeSplitter.tsx
* Add error handling for query run and SQL save operations
Co-authored-by: natemkelley <natemkelley@gmail.com>
* Trim whitespace from input values before sending chat prompts
Co-authored-by: natemkelley <natemkelley@gmail.com>
* type in think-and-prep
* use the cached access chat
* update package version
* new asset import message
* Error fallback for login
* Update BusterChart.BarChart.stories.tsx
* Staging changes to fix number card titles, combo chart axis, and using dynamic filters (#386)
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
* db init command pass through
* combo chart fixes (#387)
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
* clarifying question and connection logic
* pino pretty error fix
* clarifying is a finishing tool
* change update latest version logic
* Update support endpoint
* fixes for horizontal bar charts and added the combo chart logic to update metrics (#388)
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
* permission fix on dashboard metric handlers for workspace and data admin
* Add more try catches
* Hide avatar is no more
* Horizontal bar fixes (#389)
* fixes for horizontal bar charts and added the combo chart logic to update metrics
* hopefully fixed horizontal bar charts
---------
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
* reasoning shimmer update
* Make the embed flow work with versions
* new account warning update
* Move support modal
* compact number for pie label
* Add final reasoning message tracking and workflow start time to chunk processor and related steps
- Introduced `finalReasoningMessage` to schemas in `analyst-step`, `mark-message-complete-step`, and `create-todos-step`.
- Updated `ChunkProcessor` to calculate and store the final reasoning message based on workflow duration.
- Enhanced various steps to utilize the new `workflowStartTime` for better tracking of execution duration.
- Improved database update logic to include `finalReasoningMessage` when applicable.
* 9 digit cutoff for pie
* trigger update
* test on mastra braintrust
* test deployment
* testing
* pnpm install
* pnpm
* node 22
* pnpm version
* trigger main
* get initial chat file
* hono main deploymenbt
* clear timeouts
* Remove console logs
* migration test to staging
* db url
* try again
* k get rid of tls var
* hmmm lets try this
* mark migrations
* fix migration file?
* drizzle-kit upgrade
* tweaks to the github actions
---------
Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>
dal
Nate Kelley
Devin AI
Cursor Agent
jacob-buster
Nate Kelley