Commit Graph

254 Commits

Author SHA1 Message Date
dal 5157b4a44a
revert env.mjs back 2025-07-21 01:08:30 -06:00
dal 2405242950
skip validate env 2025-07-21 00:43:45 -06:00
dal 474dffc1d3
Try again 2025-07-21 00:39:08 -06:00
dal 5ae39a15b3
k lets try this 2025-07-21 00:36:01 -06:00
dal ec4262b87c
ci and a random lint 2025-07-21 00:28:39 -06:00
Devin AI 9e461f9948 Fix Docker context paths in GitHub workflows - change ./api to ./apps/api
Co-Authored-By: Dallin Bentley <dallinbentley98@gmail.com>
2025-07-17 15:42:19 +00:00
dal a36bba3b40
Add '@buster/ai' dependency and enhance chat cancellation logic
- Added '@buster/ai' as a workspace dependency in pnpm-lock.yaml and package.json.
- Updated database-migrations.yml to trigger on changes in the database package.
- Refined the cancelChatHandler function to include detailed message cleanup and trigger cancellation logic.
- Improved response handling in the chat cancellation endpoint to return a success message.
- Enhanced updateMessageFields to support marking messages as completed.
2025-07-11 09:51:51 -06:00
dal 3a200a1029
get rid of old workflows 2025-07-09 07:44:57 -06:00
dal 1e812447ae
trigger deployment fix 2025-07-07 12:21:21 -06:00
dal 482009c01f
trigger upgrade and deployment update 2025-07-07 08:07:46 -06:00
porter-deployment-app[bot] 186290aee4
Create porter_app_staging-hono-server_3155.yml file 2025-07-03 20:20:47 +00:00
dal 2da197697d
Merge branch 'main' into staging 2025-07-03 09:48:04 -06:00
porter-deployment-app[bot] ed4e4c9b33
Create porter_app_main-hono-server_3155.yml file (#401)
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
2025-07-03 07:18:02 -07:00
porter-deployment-app[bot] 78081eebe9
Delete porter_app_main-hono-server_3155.yml file (#400)
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
2025-07-03 07:13:44 -07:00
dal 47f0d3c0e0
deploy main hono on push to main (#399) 2025-07-03 06:44:42 -07:00
dal 3db2cf72b2
Deployment (#397)
* Mastra braintrust (#391)

* type fixes

* biome clean on ai

* add user to flag chat

* attempt to get vercel deployed

* Update tsup.config.ts

* Update pnpm-lock.yaml

* Add @buster/server2 Hono API app with Vercel deployment configuration

* slack oauth integration

* mainly some clean up and biome formatting

* slack oauth

* slack migration + snapshot

* remove unused files

* finalized docker image for porter

* Create porter_app_buster-server_3155.yml file

* Add integration tests for Slack handler and refactor Slack OAuth service

- Introduced integration tests for the Slack handler, covering OAuth initiation, callback handling, and integration status retrieval.
- Refactored Slack OAuth service to improve error handling and ensure proper integration state management.
- Updated token storage implementation to use a database vault instead of Supabase.
- Enhanced existing tests for better coverage and reliability, including cleanup of test data.
- Added new utility functions for managing vault secrets in the database.

* docker image update

* new prompts

* individual tests and a schema fix

* server build

* final working dockerfile

* Update Dockerfile

* new messages to slack messages (#369)

* Update dockerfile

* Update validate-env.js

* update build pipeline

* Update the dockerfile flow

* finalize logging for pino

* stable base

* Update cors middleware logger

* Update cors.ts

* update docker to be more imformative

* Update index.ts

* Update auth.ts

* Update cors.ts

* Update cors.ts

* Update logger.ts

* remove logs

* more cors updates

* build server shared

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies. (#370)

* tons of file parsing errors (#371)

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies.

* tons of file parsing errors

* Dev mode updates

* more stable electric handler

* Dal/agent-self-healing-fixes (#372)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* Dal/agent-self-healing-fixes (#373)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* change porter staging deploy to mastra-braintrust.

* new path for porter deploy

* deploy to staging fix

* Create porter_app_mastra-braintrust-api_3155.yml file (#375)

Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>

* Update sizing and opacity

* supe up the instance for mastra

* environment staging

* ssl script

* copy path

* Update list padding

* no throttle and the anthropic cached

* move select to the top

* Update margin inline start

* shrink reasoning vertical space to 2px

* semi bold font for headers

* update animation timing

* haiku

* Add createTodoList tool and integrate into create-todos-step

* chat helper on post chat

* only trigger cicd when change made

* Start created streaming text components

* Refactor analyst agent task to initialize Braintrust logging asynchronously and parallelize database queries for improved performance. Adjusted cleanup timeout for Braintrust traces to reduce delays.

* fixed reasoned for X, so that it rounds down to the minute

* Update users page

* update build pipeline for new web

* document title update

* Named chats for page

* Datasets titles

* Refactor visualization tools and enhance error handling in retryable agent stream. Removed unused metricValueLabel from metrics file tool, updated metric configuration schemas, and improved healing mechanism for tool errors during streaming.

* analyst

* document title updates

* Update useDocumentTitle.tsx

* Refactor tool choice configuration in create-todos-step to use structured object. Remove exponential backoff logic from retryable agent stream for healable errors. Introduce new test for real-world healing scenarios in retryable agent stream.

* Refactor SQL validation logic in modify-metrics-file-tool to skip unnecessary checks when SQL has not changed. Enhance error handling and update validation messages. Clean up code formatting for improved readability.

* update collapse for filecard

* chevron collapse

* Jacob prompt changes (#376)

* prompt changes to improve filtering logic and handle priv/sec errors

* prompt changes to make aggregation better and improved filter best practices

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/agents/think-and-prep-agent/think-and-prep-instructions.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: dal <dallin@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* think and prep

* change header and strong fonts weights

* Update get collection

* combo chart x axis update

* Create a chart schemas as types

* schema types

* simple unit tests for line chart props

* fix the response file ordering iwth active selection.

* copy around reasoning messages taken care of

* fix nullable user message and file processing and such.

* update ticks for chart config

* fix todo parsing.

* app markdown update

* Update splitter to use border instead of width

* change ml

* If no file is found we should auto redirect

* Refactor database connection handling to support SSL modes. Introduced functions to extract SSL parameters and manage connections based on SSL requirements, including a custom verifier for unverified connections.

* black box message update

* chat title updates

* optimizations for trigger.

* some keepalive logic on the anthropic cached

* keep title empty until new one

* no duplicate messages

* null user message on asset pull

* posthog error handling

* 20 sec idle timeout on anthropic

* null req message

* fixed modificiation names missing

* Refactor tool call handling to support new content array format in asset messages and context loaders

* cache most recent file from workflow

* Enhance date and number detection in createDataMetadata function to improve data type handling for metrics files

* group hover effect for message

* logging for chat

* Add messageId handling and file association tracking in dashboard and metrics tools

- Updated runtime context to include messageId in create and modify dashboard and metrics file tools.
- Implemented file association tracking based on messageId in create and modify functions for both dashboards and metrics.
- Ensured type consistency by using AnalystRuntimeContext in runtime context parameters.

* logging for chat

* message type update

* Route to first file instead

* trigger moved to catalog

* Enhance file selection logic to support YAML parsing and improve logging

- Updated `extractMetricIdsFromDashboard` to first attempt JSON parsing, falling back to a regex-based YAML parsing for metric IDs.
- Added detailed debug logging in `selectFilesForResponse` to track file selection process, including metrics and dashboards involved.
- Introduced tests for various scenarios in `file-selection.test.ts` to ensure correct behavior with dashboard context and edge cases.

* trigger dev v4-beta

* Retry + Self Healing (#381)

* Refactor retry logic in analyst and think-and-prep steps

Co-authored-by: dallin <dallin@buster.so>

* some fixes

* console log error

* self healing

* todos retry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* remove lots of logs

* Remove chat streaming

* Remove chat streaming

* timeout

* Change to updated at field

* link to home

* Update timeout settings for HTTP and HTTPS agents from 20 seconds to 10 seconds for improved responsiveness.

* Add utils module and integrate message conversion in post_chat_handler

* Implement error handling for extract values (#382)

* Remove chat streaming

* Improve error handling and logging in extract values and chat title steps

Co-authored-by: dallin <dallin@buster.so>

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* loading icon for buster avatar

* finalize tooltip cache

* upgrade mastra

* increase retries

* Add redo functionality for chat messages

- Introduced `redoFromMessageId` parameter in `handleExistingChat` to allow users to specify a message to redo from.
- Implemented validation to ensure the specified message belongs to the current chat.
- Added `softDeleteMessagesFromPoint` function to soft delete a message and all subsequent messages in the same chat, facilitating the redo feature.

* fix electric potential memory leak

* tooltip cache and chart cleanup

* Update bullet to be more indented

* latest version number

* add support endpoint to new server

* Fix jank in combo bar charts

* index check for dashboard

* Collapse only if there are metrics

* Is finished reasoing back

* Update dependencies and enhance chat message handling

- Upgraded `@mastra/core` to version 0.10.8 and added `node-sql-parser` at version 5.3.10 in the lock file.
- Improved integration tests for chat message redo functionality, ensuring correct behavior when deriving `chat_id` from `message_id`.
- Enhanced error handling and validation in the `initializeChat` function to manage cases where `chat_id` is not provided.

* Update pnpm-lock and enhance chat message integration tests

- Added `node-sql-parser` version 5.3.10 to dependencies and updated the lock file.
- Improved integration tests for chat message redo functionality, ensuring accurate deletion and retrieval of messages.
- Enhanced the `initializeChat` function to derive `chat_id` from `message_id` when not provided, improving error handling and validation.

* remove .env import breaking build

* add updated at to the get chat handler

* zmall runtime error fix

* permission tests passing

* return updated at on the get chat handler now

* slq parser fixes

* Implement chat access control logic and add comprehensive tests

- Developed the `canUserAccessChat` function to determine user access to chats based on direct permissions, collection permissions, creator status, and organizational roles.
- Introduced helper functions for checking permissions and retrieving chat information.
- Added integration tests to validate access control logic, covering various scenarios including direct permissions, collection permissions, and user roles.
- Created unit tests to ensure the correctness of the access control function with mocked database interactions.
- Included simple integration tests to verify functionality with existing database data.

* sql parser and int tests working.

* fix test and lint issues

* comment to kick off deployment lo

* access controls on datasets

* electric context bug fix with sql helpers.

* permission and read only

* Add lru-cache dependency and export cache management functions

- Added `lru-cache` as a dependency in the access-controls package.
- Exported new cache management functions from `chats-cached` module, including `canUserAccessChatCached`, `getCacheStats`, `resetCacheStats`, `clearCache`, `invalidateAccess`, `invalidateUserAccess`, and `invalidateChatAccess`.

* packages deploy as well

* wrong workflow lol

* Update AppVerticalCodeSplitter.tsx

* Add error handling for query run and SQL save operations

Co-authored-by: natemkelley <natemkelley@gmail.com>

* Trim whitespace from input values before sending chat prompts

Co-authored-by: natemkelley <natemkelley@gmail.com>

* type in think-and-prep

* use the cached access chat

* update package version

* new asset import message

* Error fallback for login

* Update BusterChart.BarChart.stories.tsx

* Staging changes to fix number card titles, combo chart axis, and using dynamic filters (#386)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* db init command pass through

* combo chart fixes (#387)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* clarifying question and connection logic

* pino pretty error fix

* clarifying is a finishing tool

* change update latest version logic

* Update support endpoint

* fixes for horizontal bar charts and added the combo chart logic to update metrics (#388)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* permission fix on dashboard metric handlers for workspace and data admin

* Add more try catches

* Hide avatar is no more

* Horizontal bar fixes (#389)

* fixes for horizontal bar charts and added the combo chart logic to update metrics

* hopefully fixed horizontal bar charts

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* reasoning shimmer update

* Make the embed flow work with versions

* new account warning update

* Move support modal

* compact number for pie label

* Add final reasoning message tracking and workflow start time to chunk processor and related steps

- Introduced `finalReasoningMessage` to schemas in `analyst-step`, `mark-message-complete-step`, and `create-todos-step`.
- Updated `ChunkProcessor` to calculate and store the final reasoning message based on workflow duration.
- Enhanced various steps to utilize the new `workflowStartTime` for better tracking of execution duration.
- Improved database update logic to include `finalReasoningMessage` when applicable.

* 9 digit cutoff for pie

* trigger update

* test on mastra braintrust

* test deployment

* testing

* pnpm install

* pnpm

* node 22

* pnpm version

* trigger main

* get initial chat file

* hono main deploymenbt

* clear timeouts

* Remove console logs

* migration test to staging

* db url

* try again

* k get rid of tls var

* hmmm lets try this

* mark migrations

* fix migration file?

* drizzle-kit upgrade

* tweaks to the github actions

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>

* cert location copy moved (#392)

* biome clean on ai

* add user to flag chat

* attempt to get vercel deployed

* Update tsup.config.ts

* Update pnpm-lock.yaml

* Add @buster/server2 Hono API app with Vercel deployment configuration

* slack oauth integration

* mainly some clean up and biome formatting

* slack oauth

* slack migration + snapshot

* remove unused files

* finalized docker image for porter

* Create porter_app_buster-server_3155.yml file

* Add integration tests for Slack handler and refactor Slack OAuth service

- Introduced integration tests for the Slack handler, covering OAuth initiation, callback handling, and integration status retrieval.
- Refactored Slack OAuth service to improve error handling and ensure proper integration state management.
- Updated token storage implementation to use a database vault instead of Supabase.
- Enhanced existing tests for better coverage and reliability, including cleanup of test data.
- Added new utility functions for managing vault secrets in the database.

* docker image update

* new prompts

* individual tests and a schema fix

* server build

* final working dockerfile

* Update Dockerfile

* new messages to slack messages (#369)

* Update dockerfile

* Update validate-env.js

* update build pipeline

* Update the dockerfile flow

* finalize logging for pino

* stable base

* Update cors middleware logger

* Update cors.ts

* update docker to be more imformative

* Update index.ts

* Update auth.ts

* Update cors.ts

* Update cors.ts

* Update logger.ts

* remove logs

* more cors updates

* build server shared

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies. (#370)

* tons of file parsing errors (#371)

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies.

* tons of file parsing errors

* Dev mode updates

* more stable electric handler

* Dal/agent-self-healing-fixes (#372)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* Dal/agent-self-healing-fixes (#373)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* change porter staging deploy to mastra-braintrust.

* new path for porter deploy

* deploy to staging fix

* Create porter_app_mastra-braintrust-api_3155.yml file (#375)

Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>

* Update sizing and opacity

* supe up the instance for mastra

* environment staging

* ssl script

* copy path

* Update list padding

* no throttle and the anthropic cached

* move select to the top

* Update margin inline start

* shrink reasoning vertical space to 2px

* semi bold font for headers

* update animation timing

* haiku

* Add createTodoList tool and integrate into create-todos-step

* chat helper on post chat

* only trigger cicd when change made

* Start created streaming text components

* Refactor analyst agent task to initialize Braintrust logging asynchronously and parallelize database queries for improved performance. Adjusted cleanup timeout for Braintrust traces to reduce delays.

* fixed reasoned for X, so that it rounds down to the minute

* Update users page

* update build pipeline for new web

* document title update

* Named chats for page

* Datasets titles

* Refactor visualization tools and enhance error handling in retryable agent stream. Removed unused metricValueLabel from metrics file tool, updated metric configuration schemas, and improved healing mechanism for tool errors during streaming.

* analyst

* document title updates

* Update useDocumentTitle.tsx

* Refactor tool choice configuration in create-todos-step to use structured object. Remove exponential backoff logic from retryable agent stream for healable errors. Introduce new test for real-world healing scenarios in retryable agent stream.

* Refactor SQL validation logic in modify-metrics-file-tool to skip unnecessary checks when SQL has not changed. Enhance error handling and update validation messages. Clean up code formatting for improved readability.

* update collapse for filecard

* chevron collapse

* Jacob prompt changes (#376)

* prompt changes to improve filtering logic and handle priv/sec errors

* prompt changes to make aggregation better and improved filter best practices

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/agents/think-and-prep-agent/think-and-prep-instructions.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: dal <dallin@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* think and prep

* change header and strong fonts weights

* Update get collection

* combo chart x axis update

* Create a chart schemas as types

* schema types

* simple unit tests for line chart props

* fix the response file ordering iwth active selection.

* copy around reasoning messages taken care of

* fix nullable user message and file processing and such.

* update ticks for chart config

* fix todo parsing.

* app markdown update

* Update splitter to use border instead of width

* change ml

* If no file is found we should auto redirect

* Refactor database connection handling to support SSL modes. Introduced functions to extract SSL parameters and manage connections based on SSL requirements, including a custom verifier for unverified connections.

* black box message update

* chat title updates

* optimizations for trigger.

* some keepalive logic on the anthropic cached

* keep title empty until new one

* no duplicate messages

* null user message on asset pull

* posthog error handling

* 20 sec idle timeout on anthropic

* null req message

* fixed modificiation names missing

* Refactor tool call handling to support new content array format in asset messages and context loaders

* cache most recent file from workflow

* Enhance date and number detection in createDataMetadata function to improve data type handling for metrics files

* group hover effect for message

* logging for chat

* Add messageId handling and file association tracking in dashboard and metrics tools

- Updated runtime context to include messageId in create and modify dashboard and metrics file tools.
- Implemented file association tracking based on messageId in create and modify functions for both dashboards and metrics.
- Ensured type consistency by using AnalystRuntimeContext in runtime context parameters.

* logging for chat

* message type update

* Route to first file instead

* trigger moved to catalog

* Enhance file selection logic to support YAML parsing and improve logging

- Updated `extractMetricIdsFromDashboard` to first attempt JSON parsing, falling back to a regex-based YAML parsing for metric IDs.
- Added detailed debug logging in `selectFilesForResponse` to track file selection process, including metrics and dashboards involved.
- Introduced tests for various scenarios in `file-selection.test.ts` to ensure correct behavior with dashboard context and edge cases.

* trigger dev v4-beta

* Retry + Self Healing (#381)

* Refactor retry logic in analyst and think-and-prep steps

Co-authored-by: dallin <dallin@buster.so>

* some fixes

* console log error

* self healing

* todos retry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* remove lots of logs

* Remove chat streaming

* Remove chat streaming

* timeout

* Change to updated at field

* link to home

* Update timeout settings for HTTP and HTTPS agents from 20 seconds to 10 seconds for improved responsiveness.

* Add utils module and integrate message conversion in post_chat_handler

* Implement error handling for extract values (#382)

* Remove chat streaming

* Improve error handling and logging in extract values and chat title steps

Co-authored-by: dallin <dallin@buster.so>

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* loading icon for buster avatar

* finalize tooltip cache

* upgrade mastra

* increase retries

* Add redo functionality for chat messages

- Introduced `redoFromMessageId` parameter in `handleExistingChat` to allow users to specify a message to redo from.
- Implemented validation to ensure the specified message belongs to the current chat.
- Added `softDeleteMessagesFromPoint` function to soft delete a message and all subsequent messages in the same chat, facilitating the redo feature.

* fix electric potential memory leak

* tooltip cache and chart cleanup

* Update bullet to be more indented

* latest version number

* add support endpoint to new server

* Fix jank in combo bar charts

* index check for dashboard

* Collapse only if there are metrics

* Is finished reasoing back

* Update dependencies and enhance chat message handling

- Upgraded `@mastra/core` to version 0.10.8 and added `node-sql-parser` at version 5.3.10 in the lock file.
- Improved integration tests for chat message redo functionality, ensuring correct behavior when deriving `chat_id` from `message_id`.
- Enhanced error handling and validation in the `initializeChat` function to manage cases where `chat_id` is not provided.

* Update pnpm-lock and enhance chat message integration tests

- Added `node-sql-parser` version 5.3.10 to dependencies and updated the lock file.
- Improved integration tests for chat message redo functionality, ensuring accurate deletion and retrieval of messages.
- Enhanced the `initializeChat` function to derive `chat_id` from `message_id` when not provided, improving error handling and validation.

* remove .env import breaking build

* add updated at to the get chat handler

* zmall runtime error fix

* permission tests passing

* return updated at on the get chat handler now

* slq parser fixes

* Implement chat access control logic and add comprehensive tests

- Developed the `canUserAccessChat` function to determine user access to chats based on direct permissions, collection permissions, creator status, and organizational roles.
- Introduced helper functions for checking permissions and retrieving chat information.
- Added integration tests to validate access control logic, covering various scenarios including direct permissions, collection permissions, and user roles.
- Created unit tests to ensure the correctness of the access control function with mocked database interactions.
- Included simple integration tests to verify functionality with existing database data.

* sql parser and int tests working.

* fix test and lint issues

* comment to kick off deployment lo

* access controls on datasets

* electric context bug fix with sql helpers.

* permission and read only

* Add lru-cache dependency and export cache management functions

- Added `lru-cache` as a dependency in the access-controls package.
- Exported new cache management functions from `chats-cached` module, including `canUserAccessChatCached`, `getCacheStats`, `resetCacheStats`, `clearCache`, `invalidateAccess`, `invalidateUserAccess`, and `invalidateChatAccess`.

* packages deploy as well

* wrong workflow lol

* Update AppVerticalCodeSplitter.tsx

* Add error handling for query run and SQL save operations

Co-authored-by: natemkelley <natemkelley@gmail.com>

* Trim whitespace from input values before sending chat prompts

Co-authored-by: natemkelley <natemkelley@gmail.com>

* type in think-and-prep

* use the cached access chat

* update package version

* new asset import message

* Error fallback for login

* Update BusterChart.BarChart.stories.tsx

* Staging changes to fix number card titles, combo chart axis, and using dynamic filters (#386)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* db init command pass through

* combo chart fixes (#387)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* clarifying question and connection logic

* pino pretty error fix

* clarifying is a finishing tool

* change update latest version logic

* Update support endpoint

* fixes for horizontal bar charts and added the combo chart logic to update metrics (#388)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* permission fix on dashboard metric handlers for workspace and data admin

* Add more try catches

* Hide avatar is no more

* Horizontal bar fixes (#389)

* fixes for horizontal bar charts and added the combo chart logic to update metrics

* hopefully fixed horizontal bar charts

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* reasoning shimmer update

* Make the embed flow work with versions

* new account warning update

* Move support modal

* compact number for pie label

* Add final reasoning message tracking and workflow start time to chunk processor and related steps

- Introduced `finalReasoningMessage` to schemas in `analyst-step`, `mark-message-complete-step`, and `create-todos-step`.
- Updated `ChunkProcessor` to calculate and store the final reasoning message based on workflow duration.
- Enhanced various steps to utilize the new `workflowStartTime` for better tracking of execution duration.
- Improved database update logic to include `finalReasoningMessage` when applicable.

* 9 digit cutoff for pie

* trigger update

* test on mastra braintrust

* test deployment

* testing

* pnpm install

* pnpm

* node 22

* pnpm version

* trigger main

* get initial chat file

* hono main deploymenbt

* clear timeouts

* Remove console logs

* migration test to staging

* db url

* try again

* k get rid of tls var

* hmmm lets try this

* mark migrations

* fix migration file?

* drizzle-kit upgrade

* tweaks to the github actions

* clean up workflows db migration

* cert location copy moved

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>

* deploy to staging (#393)

* Mastra braintrust (#394)

* Update tsup.config.ts

* Update pnpm-lock.yaml

* Add @buster/server2 Hono API app with Vercel deployment configuration

* slack oauth integration

* mainly some clean up and biome formatting

* slack oauth

* slack migration + snapshot

* remove unused files

* finalized docker image for porter

* Create porter_app_buster-server_3155.yml file

* Add integration tests for Slack handler and refactor Slack OAuth service

- Introduced integration tests for the Slack handler, covering OAuth initiation, callback handling, and integration status retrieval.
- Refactored Slack OAuth service to improve error handling and ensure proper integration state management.
- Updated token storage implementation to use a database vault instead of Supabase.
- Enhanced existing tests for better coverage and reliability, including cleanup of test data.
- Added new utility functions for managing vault secrets in the database.

* docker image update

* new prompts

* individual tests and a schema fix

* server build

* final working dockerfile

* Update Dockerfile

* new messages to slack messages (#369)

* Update dockerfile

* Update validate-env.js

* update build pipeline

* Update the dockerfile flow

* finalize logging for pino

* stable base

* Update cors middleware logger

* Update cors.ts

* update docker to be more imformative

* Update index.ts

* Update auth.ts

* Update cors.ts

* Update cors.ts

* Update logger.ts

* remove logs

* more cors updates

* build server shared

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies. (#370)

* tons of file parsing errors (#371)

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies.

* tons of file parsing errors

* Dev mode updates

* more stable electric handler

* Dal/agent-self-healing-fixes (#372)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* Dal/agent-self-healing-fixes (#373)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* change porter staging deploy to mastra-braintrust.

* new path for porter deploy

* deploy to staging fix

* Create porter_app_mastra-braintrust-api_3155.yml file (#375)

Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>

* Update sizing and opacity

* supe up the instance for mastra

* environment staging

* ssl script

* copy path

* Update list padding

* no throttle and the anthropic cached

* move select to the top

* Update margin inline start

* shrink reasoning vertical space to 2px

* semi bold font for headers

* update animation timing

* haiku

* Add createTodoList tool and integrate into create-todos-step

* chat helper on post chat

* only trigger cicd when change made

* Start created streaming text components

* Refactor analyst agent task to initialize Braintrust logging asynchronously and parallelize database queries for improved performance. Adjusted cleanup timeout for Braintrust traces to reduce delays.

* fixed reasoned for X, so that it rounds down to the minute

* Update users page

* update build pipeline for new web

* document title update

* Named chats for page

* Datasets titles

* Refactor visualization tools and enhance error handling in retryable agent stream. Removed unused metricValueLabel from metrics file tool, updated metric configuration schemas, and improved healing mechanism for tool errors during streaming.

* analyst

* document title updates

* Update useDocumentTitle.tsx

* Refactor tool choice configuration in create-todos-step to use structured object. Remove exponential backoff logic from retryable agent stream for healable errors. Introduce new test for real-world healing scenarios in retryable agent stream.

* Refactor SQL validation logic in modify-metrics-file-tool to skip unnecessary checks when SQL has not changed. Enhance error handling and update validation messages. Clean up code formatting for improved readability.

* update collapse for filecard

* chevron collapse

* Jacob prompt changes (#376)

* prompt changes to improve filtering logic and handle priv/sec errors

* prompt changes to make aggregation better and improved filter best practices

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/agents/think-and-prep-agent/think-and-prep-instructions.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: dal <dallin@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* think and prep

* change header and strong fonts weights

* Update get collection

* combo chart x axis update

* Create a chart schemas as types

* schema types

* simple unit tests for line chart props

* fix the response file ordering iwth active selection.

* copy around reasoning messages taken care of

* fix nullable user message and file processing and such.

* update ticks for chart config

* fix todo parsing.

* app markdown update

* Update splitter to use border instead of width

* change ml

* If no file is found we should auto redirect

* Refactor database connection handling to support SSL modes. Introduced functions to extract SSL parameters and manage connections based on SSL requirements, including a custom verifier for unverified connections.

* black box message update

* chat title updates

* optimizations for trigger.

* some keepalive logic on the anthropic cached

* keep title empty until new one

* no duplicate messages

* null user message on asset pull

* posthog error handling

* 20 sec idle timeout on anthropic

* null req message

* fixed modificiation names missing

* Refactor tool call handling to support new content array format in asset messages and context loaders

* cache most recent file from workflow

* Enhance date and number detection in createDataMetadata function to improve data type handling for metrics files

* group hover effect for message

* logging for chat

* Add messageId handling and file association tracking in dashboard and metrics tools

- Updated runtime context to include messageId in create and modify dashboard and metrics file tools.
- Implemented file association tracking based on messageId in create and modify functions for both dashboards and metrics.
- Ensured type consistency by using AnalystRuntimeContext in runtime context parameters.

* logging for chat

* message type update

* Route to first file instead

* trigger moved to catalog

* Enhance file selection logic to support YAML parsing and improve logging

- Updated `extractMetricIdsFromDashboard` to first attempt JSON parsing, falling back to a regex-based YAML parsing for metric IDs.
- Added detailed debug logging in `selectFilesForResponse` to track file selection process, including metrics and dashboards involved.
- Introduced tests for various scenarios in `file-selection.test.ts` to ensure correct behavior with dashboard context and edge cases.

* trigger dev v4-beta

* Retry + Self Healing (#381)

* Refactor retry logic in analyst and think-and-prep steps

Co-authored-by: dallin <dallin@buster.so>

* some fixes

* console log error

* self healing

* todos retry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* remove lots of logs

* Remove chat streaming

* Remove chat streaming

* timeout

* Change to updated at field

* link to home

* Update timeout settings for HTTP and HTTPS agents from 20 seconds to 10 seconds for improved responsiveness.

* Add utils module and integrate message conversion in post_chat_handler

* Implement error handling for extract values (#382)

* Remove chat streaming

* Improve error handling and logging in extract values and chat title steps

Co-authored-by: dallin <dallin@buster.so>

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* loading icon for buster avatar

* finalize tooltip cache

* upgrade mastra

* increase retries

* Add redo functionality for chat messages

- Introduced `redoFromMessageId` parameter in `handleExistingChat` to allow users to specify a message to redo from.
- Implemented validation to ensure the specified message belongs to the current chat.
- Added `softDeleteMessagesFromPoint` function to soft delete a message and all subsequent messages in the same chat, facilitating the redo feature.

* fix electric potential memory leak

* tooltip cache and chart cleanup

* Update bullet to be more indented

* latest version number

* add support endpoint to new server

* Fix jank in combo bar charts

* index check for dashboard

* Collapse only if there are metrics

* Is finished reasoing back

* Update dependencies and enhance chat message handling

- Upgraded `@mastra/core` to version 0.10.8 and added `node-sql-parser` at version 5.3.10 in the lock file.
- Improved integration tests for chat message redo functionality, ensuring correct behavior when deriving `chat_id` from `message_id`.
- Enhanced error handling and validation in the `initializeChat` function to manage cases where `chat_id` is not provided.

* Update pnpm-lock and enhance chat message integration tests

- Added `node-sql-parser` version 5.3.10 to dependencies and updated the lock file.
- Improved integration tests for chat message redo functionality, ensuring accurate deletion and retrieval of messages.
- Enhanced the `initializeChat` function to derive `chat_id` from `message_id` when not provided, improving error handling and validation.

* remove .env import breaking build

* add updated at to the get chat handler

* zmall runtime error fix

* permission tests passing

* return updated at on the get chat handler now

* slq parser fixes

* Implement chat access control logic and add comprehensive tests

- Developed the `canUserAccessChat` function to determine user access to chats based on direct permissions, collection permissions, creator status, and organizational roles.
- Introduced helper functions for checking permissions and retrieving chat information.
- Added integration tests to validate access control logic, covering various scenarios including direct permissions, collection permissions, and user roles.
- Created unit tests to ensure the correctness of the access control function with mocked database interactions.
- Included simple integration tests to verify functionality with existing database data.

* sql parser and int tests working.

* fix test and lint issues

* comment to kick off deployment lo

* access controls on datasets

* electric context bug fix with sql helpers.

* permission and read only

* Add lru-cache dependency and export cache management functions

- Added `lru-cache` as a dependency in the access-controls package.
- Exported new cache management functions from `chats-cached` module, including `canUserAccessChatCached`, `getCacheStats`, `resetCacheStats`, `clearCache`, `invalidateAccess`, `invalidateUserAccess`, and `invalidateChatAccess`.

* packages deploy as well

* wrong workflow lol

* Update AppVerticalCodeSplitter.tsx

* Add error handling for query run and SQL save operations

Co-authored-by: natemkelley <natemkelley@gmail.com>

* Trim whitespace from input values before sending chat prompts

Co-authored-by: natemkelley <natemkelley@gmail.com>

* type in think-and-prep

* use the cached access chat

* update package version

* new asset import message

* Error fallback for login

* Update BusterChart.BarChart.stories.tsx

* Staging changes to fix number card titles, combo chart axis, and using dynamic filters (#386)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* db init command pass through

* combo chart fixes (#387)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* clarifying question and connection logic

* pino pretty error fix

* clarifying is a finishing tool

* change update latest version logic

* Update support endpoint

* fixes for horizontal bar charts and added the combo chart logic to update metrics (#388)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* permission fix on dashboard metric handlers for workspace and data admin

* Add more try catches

* Hide avatar is no more

* Horizontal bar fixes (#389)

* fixes for horizontal bar charts and added the combo chart logic to update metrics

* hopefully fixed horizontal bar charts

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* reasoning shimmer update

* Make the embed flow work with versions

* new account warning update

* Move support modal

* compact number for pie label

* Add final reasoning message tracking and workflow start time to chunk processor and related steps

- Introduced `finalReasoningMessage` to schemas in `analyst-step`, `mark-message-complete-step`, and `create-todos-step`.
- Updated `ChunkProcessor` to calculate and store the final reasoning message based on workflow duration.
- Enhanced various steps to utilize the new `workflowStartTime` for better tracking of execution duration.
- Improved database update logic to include `finalReasoningMessage` when applicable.

* 9 digit cutoff for pie

* trigger update

* test on mastra braintrust

* test deployment

* testing

* pnpm install

* pnpm

* node 22

* pnpm version

* trigger main

* get initial chat file

* hono main deploymenbt

* clear timeouts

* Remove console logs

* migration test to staging

* db url

* try again

* k get rid of tls var

* hmmm lets try this

* mark migrations

* fix migration file?

* drizzle-kit upgrade

* tweaks to the github actions

* clean up workflows db migration

* cert location copy moved

* Prism highlighting update

* Fix merge conflicts

---------

Co-authored-by: dal <dallin@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>

* Remove logs

* Remove logs (#395)

* Fix broken web unit tests

* Remove useless test

* Update helpers.test.ts

* Create tsconfig.json

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>
2025-07-03 06:32:18 -07:00
Nate Kelley 3ec9e775a3
Mastra braintrust (#394)
* Update tsup.config.ts

* Update pnpm-lock.yaml

* Add @buster/server2 Hono API app with Vercel deployment configuration

* slack oauth integration

* mainly some clean up and biome formatting

* slack oauth

* slack migration + snapshot

* remove unused files

* finalized docker image for porter

* Create porter_app_buster-server_3155.yml file

* Add integration tests for Slack handler and refactor Slack OAuth service

- Introduced integration tests for the Slack handler, covering OAuth initiation, callback handling, and integration status retrieval.
- Refactored Slack OAuth service to improve error handling and ensure proper integration state management.
- Updated token storage implementation to use a database vault instead of Supabase.
- Enhanced existing tests for better coverage and reliability, including cleanup of test data.
- Added new utility functions for managing vault secrets in the database.

* docker image update

* new prompts

* individual tests and a schema fix

* server build

* final working dockerfile

* Update Dockerfile

* new messages to slack messages (#369)

* Update dockerfile

* Update validate-env.js

* update build pipeline

* Update the dockerfile flow

* finalize logging for pino

* stable base

* Update cors middleware logger

* Update cors.ts

* update docker to be more imformative

* Update index.ts

* Update auth.ts

* Update cors.ts

* Update cors.ts

* Update logger.ts

* remove logs

* more cors updates

* build server shared

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies. (#370)

* tons of file parsing errors (#371)

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies.

* tons of file parsing errors

* Dev mode updates

* more stable electric handler

* Dal/agent-self-healing-fixes (#372)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* Dal/agent-self-healing-fixes (#373)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* change porter staging deploy to mastra-braintrust.

* new path for porter deploy

* deploy to staging fix

* Create porter_app_mastra-braintrust-api_3155.yml file (#375)

Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>

* Update sizing and opacity

* supe up the instance for mastra

* environment staging

* ssl script

* copy path

* Update list padding

* no throttle and the anthropic cached

* move select to the top

* Update margin inline start

* shrink reasoning vertical space to 2px

* semi bold font for headers

* update animation timing

* haiku

* Add createTodoList tool and integrate into create-todos-step

* chat helper on post chat

* only trigger cicd when change made

* Start created streaming text components

* Refactor analyst agent task to initialize Braintrust logging asynchronously and parallelize database queries for improved performance. Adjusted cleanup timeout for Braintrust traces to reduce delays.

* fixed reasoned for X, so that it rounds down to the minute

* Update users page

* update build pipeline for new web

* document title update

* Named chats for page

* Datasets titles

* Refactor visualization tools and enhance error handling in retryable agent stream. Removed unused metricValueLabel from metrics file tool, updated metric configuration schemas, and improved healing mechanism for tool errors during streaming.

* analyst

* document title updates

* Update useDocumentTitle.tsx

* Refactor tool choice configuration in create-todos-step to use structured object. Remove exponential backoff logic from retryable agent stream for healable errors. Introduce new test for real-world healing scenarios in retryable agent stream.

* Refactor SQL validation logic in modify-metrics-file-tool to skip unnecessary checks when SQL has not changed. Enhance error handling and update validation messages. Clean up code formatting for improved readability.

* update collapse for filecard

* chevron collapse

* Jacob prompt changes (#376)

* prompt changes to improve filtering logic and handle priv/sec errors

* prompt changes to make aggregation better and improved filter best practices

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/agents/think-and-prep-agent/think-and-prep-instructions.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: dal <dallin@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* think and prep

* change header and strong fonts weights

* Update get collection

* combo chart x axis update

* Create a chart schemas as types

* schema types

* simple unit tests for line chart props

* fix the response file ordering iwth active selection.

* copy around reasoning messages taken care of

* fix nullable user message and file processing and such.

* update ticks for chart config

* fix todo parsing.

* app markdown update

* Update splitter to use border instead of width

* change ml

* If no file is found we should auto redirect

* Refactor database connection handling to support SSL modes. Introduced functions to extract SSL parameters and manage connections based on SSL requirements, including a custom verifier for unverified connections.

* black box message update

* chat title updates

* optimizations for trigger.

* some keepalive logic on the anthropic cached

* keep title empty until new one

* no duplicate messages

* null user message on asset pull

* posthog error handling

* 20 sec idle timeout on anthropic

* null req message

* fixed modificiation names missing

* Refactor tool call handling to support new content array format in asset messages and context loaders

* cache most recent file from workflow

* Enhance date and number detection in createDataMetadata function to improve data type handling for metrics files

* group hover effect for message

* logging for chat

* Add messageId handling and file association tracking in dashboard and metrics tools

- Updated runtime context to include messageId in create and modify dashboard and metrics file tools.
- Implemented file association tracking based on messageId in create and modify functions for both dashboards and metrics.
- Ensured type consistency by using AnalystRuntimeContext in runtime context parameters.

* logging for chat

* message type update

* Route to first file instead

* trigger moved to catalog

* Enhance file selection logic to support YAML parsing and improve logging

- Updated `extractMetricIdsFromDashboard` to first attempt JSON parsing, falling back to a regex-based YAML parsing for metric IDs.
- Added detailed debug logging in `selectFilesForResponse` to track file selection process, including metrics and dashboards involved.
- Introduced tests for various scenarios in `file-selection.test.ts` to ensure correct behavior with dashboard context and edge cases.

* trigger dev v4-beta

* Retry + Self Healing (#381)

* Refactor retry logic in analyst and think-and-prep steps

Co-authored-by: dallin <dallin@buster.so>

* some fixes

* console log error

* self healing

* todos retry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* remove lots of logs

* Remove chat streaming

* Remove chat streaming

* timeout

* Change to updated at field

* link to home

* Update timeout settings for HTTP and HTTPS agents from 20 seconds to 10 seconds for improved responsiveness.

* Add utils module and integrate message conversion in post_chat_handler

* Implement error handling for extract values (#382)

* Remove chat streaming

* Improve error handling and logging in extract values and chat title steps

Co-authored-by: dallin <dallin@buster.so>

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* loading icon for buster avatar

* finalize tooltip cache

* upgrade mastra

* increase retries

* Add redo functionality for chat messages

- Introduced `redoFromMessageId` parameter in `handleExistingChat` to allow users to specify a message to redo from.
- Implemented validation to ensure the specified message belongs to the current chat.
- Added `softDeleteMessagesFromPoint` function to soft delete a message and all subsequent messages in the same chat, facilitating the redo feature.

* fix electric potential memory leak

* tooltip cache and chart cleanup

* Update bullet to be more indented

* latest version number

* add support endpoint to new server

* Fix jank in combo bar charts

* index check for dashboard

* Collapse only if there are metrics

* Is finished reasoing back

* Update dependencies and enhance chat message handling

- Upgraded `@mastra/core` to version 0.10.8 and added `node-sql-parser` at version 5.3.10 in the lock file.
- Improved integration tests for chat message redo functionality, ensuring correct behavior when deriving `chat_id` from `message_id`.
- Enhanced error handling and validation in the `initializeChat` function to manage cases where `chat_id` is not provided.

* Update pnpm-lock and enhance chat message integration tests

- Added `node-sql-parser` version 5.3.10 to dependencies and updated the lock file.
- Improved integration tests for chat message redo functionality, ensuring accurate deletion and retrieval of messages.
- Enhanced the `initializeChat` function to derive `chat_id` from `message_id` when not provided, improving error handling and validation.

* remove .env import breaking build

* add updated at to the get chat handler

* zmall runtime error fix

* permission tests passing

* return updated at on the get chat handler now

* slq parser fixes

* Implement chat access control logic and add comprehensive tests

- Developed the `canUserAccessChat` function to determine user access to chats based on direct permissions, collection permissions, creator status, and organizational roles.
- Introduced helper functions for checking permissions and retrieving chat information.
- Added integration tests to validate access control logic, covering various scenarios including direct permissions, collection permissions, and user roles.
- Created unit tests to ensure the correctness of the access control function with mocked database interactions.
- Included simple integration tests to verify functionality with existing database data.

* sql parser and int tests working.

* fix test and lint issues

* comment to kick off deployment lo

* access controls on datasets

* electric context bug fix with sql helpers.

* permission and read only

* Add lru-cache dependency and export cache management functions

- Added `lru-cache` as a dependency in the access-controls package.
- Exported new cache management functions from `chats-cached` module, including `canUserAccessChatCached`, `getCacheStats`, `resetCacheStats`, `clearCache`, `invalidateAccess`, `invalidateUserAccess`, and `invalidateChatAccess`.

* packages deploy as well

* wrong workflow lol

* Update AppVerticalCodeSplitter.tsx

* Add error handling for query run and SQL save operations

Co-authored-by: natemkelley <natemkelley@gmail.com>

* Trim whitespace from input values before sending chat prompts

Co-authored-by: natemkelley <natemkelley@gmail.com>

* type in think-and-prep

* use the cached access chat

* update package version

* new asset import message

* Error fallback for login

* Update BusterChart.BarChart.stories.tsx

* Staging changes to fix number card titles, combo chart axis, and using dynamic filters (#386)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* db init command pass through

* combo chart fixes (#387)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* clarifying question and connection logic

* pino pretty error fix

* clarifying is a finishing tool

* change update latest version logic

* Update support endpoint

* fixes for horizontal bar charts and added the combo chart logic to update metrics (#388)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* permission fix on dashboard metric handlers for workspace and data admin

* Add more try catches

* Hide avatar is no more

* Horizontal bar fixes (#389)

* fixes for horizontal bar charts and added the combo chart logic to update metrics

* hopefully fixed horizontal bar charts

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* reasoning shimmer update

* Make the embed flow work with versions

* new account warning update

* Move support modal

* compact number for pie label

* Add final reasoning message tracking and workflow start time to chunk processor and related steps

- Introduced `finalReasoningMessage` to schemas in `analyst-step`, `mark-message-complete-step`, and `create-todos-step`.
- Updated `ChunkProcessor` to calculate and store the final reasoning message based on workflow duration.
- Enhanced various steps to utilize the new `workflowStartTime` for better tracking of execution duration.
- Improved database update logic to include `finalReasoningMessage` when applicable.

* 9 digit cutoff for pie

* trigger update

* test on mastra braintrust

* test deployment

* testing

* pnpm install

* pnpm

* node 22

* pnpm version

* trigger main

* get initial chat file

* hono main deploymenbt

* clear timeouts

* Remove console logs

* migration test to staging

* db url

* try again

* k get rid of tls var

* hmmm lets try this

* mark migrations

* fix migration file?

* drizzle-kit upgrade

* tweaks to the github actions

* clean up workflows db migration

* cert location copy moved

* Prism highlighting update

* Fix merge conflicts

---------

Co-authored-by: dal <dallin@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>
2025-07-02 16:31:20 -06:00
dal 27008f1539
deploy to staging (#393) 2025-07-02 14:50:45 -07:00
dal bed288f4e1
cert location copy moved (#392)
* biome clean on ai

* add user to flag chat

* attempt to get vercel deployed

* Update tsup.config.ts

* Update pnpm-lock.yaml

* Add @buster/server2 Hono API app with Vercel deployment configuration

* slack oauth integration

* mainly some clean up and biome formatting

* slack oauth

* slack migration + snapshot

* remove unused files

* finalized docker image for porter

* Create porter_app_buster-server_3155.yml file

* Add integration tests for Slack handler and refactor Slack OAuth service

- Introduced integration tests for the Slack handler, covering OAuth initiation, callback handling, and integration status retrieval.
- Refactored Slack OAuth service to improve error handling and ensure proper integration state management.
- Updated token storage implementation to use a database vault instead of Supabase.
- Enhanced existing tests for better coverage and reliability, including cleanup of test data.
- Added new utility functions for managing vault secrets in the database.

* docker image update

* new prompts

* individual tests and a schema fix

* server build

* final working dockerfile

* Update Dockerfile

* new messages to slack messages (#369)

* Update dockerfile

* Update validate-env.js

* update build pipeline

* Update the dockerfile flow

* finalize logging for pino

* stable base

* Update cors middleware logger

* Update cors.ts

* update docker to be more imformative

* Update index.ts

* Update auth.ts

* Update cors.ts

* Update cors.ts

* Update logger.ts

* remove logs

* more cors updates

* build server shared

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies. (#370)

* tons of file parsing errors (#371)

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies.

* tons of file parsing errors

* Dev mode updates

* more stable electric handler

* Dal/agent-self-healing-fixes (#372)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* Dal/agent-self-healing-fixes (#373)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* change porter staging deploy to mastra-braintrust.

* new path for porter deploy

* deploy to staging fix

* Create porter_app_mastra-braintrust-api_3155.yml file (#375)

Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>

* Update sizing and opacity

* supe up the instance for mastra

* environment staging

* ssl script

* copy path

* Update list padding

* no throttle and the anthropic cached

* move select to the top

* Update margin inline start

* shrink reasoning vertical space to 2px

* semi bold font for headers

* update animation timing

* haiku

* Add createTodoList tool and integrate into create-todos-step

* chat helper on post chat

* only trigger cicd when change made

* Start created streaming text components

* Refactor analyst agent task to initialize Braintrust logging asynchronously and parallelize database queries for improved performance. Adjusted cleanup timeout for Braintrust traces to reduce delays.

* fixed reasoned for X, so that it rounds down to the minute

* Update users page

* update build pipeline for new web

* document title update

* Named chats for page

* Datasets titles

* Refactor visualization tools and enhance error handling in retryable agent stream. Removed unused metricValueLabel from metrics file tool, updated metric configuration schemas, and improved healing mechanism for tool errors during streaming.

* analyst

* document title updates

* Update useDocumentTitle.tsx

* Refactor tool choice configuration in create-todos-step to use structured object. Remove exponential backoff logic from retryable agent stream for healable errors. Introduce new test for real-world healing scenarios in retryable agent stream.

* Refactor SQL validation logic in modify-metrics-file-tool to skip unnecessary checks when SQL has not changed. Enhance error handling and update validation messages. Clean up code formatting for improved readability.

* update collapse for filecard

* chevron collapse

* Jacob prompt changes (#376)

* prompt changes to improve filtering logic and handle priv/sec errors

* prompt changes to make aggregation better and improved filter best practices

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/agents/think-and-prep-agent/think-and-prep-instructions.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: dal <dallin@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* think and prep

* change header and strong fonts weights

* Update get collection

* combo chart x axis update

* Create a chart schemas as types

* schema types

* simple unit tests for line chart props

* fix the response file ordering iwth active selection.

* copy around reasoning messages taken care of

* fix nullable user message and file processing and such.

* update ticks for chart config

* fix todo parsing.

* app markdown update

* Update splitter to use border instead of width

* change ml

* If no file is found we should auto redirect

* Refactor database connection handling to support SSL modes. Introduced functions to extract SSL parameters and manage connections based on SSL requirements, including a custom verifier for unverified connections.

* black box message update

* chat title updates

* optimizations for trigger.

* some keepalive logic on the anthropic cached

* keep title empty until new one

* no duplicate messages

* null user message on asset pull

* posthog error handling

* 20 sec idle timeout on anthropic

* null req message

* fixed modificiation names missing

* Refactor tool call handling to support new content array format in asset messages and context loaders

* cache most recent file from workflow

* Enhance date and number detection in createDataMetadata function to improve data type handling for metrics files

* group hover effect for message

* logging for chat

* Add messageId handling and file association tracking in dashboard and metrics tools

- Updated runtime context to include messageId in create and modify dashboard and metrics file tools.
- Implemented file association tracking based on messageId in create and modify functions for both dashboards and metrics.
- Ensured type consistency by using AnalystRuntimeContext in runtime context parameters.

* logging for chat

* message type update

* Route to first file instead

* trigger moved to catalog

* Enhance file selection logic to support YAML parsing and improve logging

- Updated `extractMetricIdsFromDashboard` to first attempt JSON parsing, falling back to a regex-based YAML parsing for metric IDs.
- Added detailed debug logging in `selectFilesForResponse` to track file selection process, including metrics and dashboards involved.
- Introduced tests for various scenarios in `file-selection.test.ts` to ensure correct behavior with dashboard context and edge cases.

* trigger dev v4-beta

* Retry + Self Healing (#381)

* Refactor retry logic in analyst and think-and-prep steps

Co-authored-by: dallin <dallin@buster.so>

* some fixes

* console log error

* self healing

* todos retry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* remove lots of logs

* Remove chat streaming

* Remove chat streaming

* timeout

* Change to updated at field

* link to home

* Update timeout settings for HTTP and HTTPS agents from 20 seconds to 10 seconds for improved responsiveness.

* Add utils module and integrate message conversion in post_chat_handler

* Implement error handling for extract values (#382)

* Remove chat streaming

* Improve error handling and logging in extract values and chat title steps

Co-authored-by: dallin <dallin@buster.so>

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* loading icon for buster avatar

* finalize tooltip cache

* upgrade mastra

* increase retries

* Add redo functionality for chat messages

- Introduced `redoFromMessageId` parameter in `handleExistingChat` to allow users to specify a message to redo from.
- Implemented validation to ensure the specified message belongs to the current chat.
- Added `softDeleteMessagesFromPoint` function to soft delete a message and all subsequent messages in the same chat, facilitating the redo feature.

* fix electric potential memory leak

* tooltip cache and chart cleanup

* Update bullet to be more indented

* latest version number

* add support endpoint to new server

* Fix jank in combo bar charts

* index check for dashboard

* Collapse only if there are metrics

* Is finished reasoing back

* Update dependencies and enhance chat message handling

- Upgraded `@mastra/core` to version 0.10.8 and added `node-sql-parser` at version 5.3.10 in the lock file.
- Improved integration tests for chat message redo functionality, ensuring correct behavior when deriving `chat_id` from `message_id`.
- Enhanced error handling and validation in the `initializeChat` function to manage cases where `chat_id` is not provided.

* Update pnpm-lock and enhance chat message integration tests

- Added `node-sql-parser` version 5.3.10 to dependencies and updated the lock file.
- Improved integration tests for chat message redo functionality, ensuring accurate deletion and retrieval of messages.
- Enhanced the `initializeChat` function to derive `chat_id` from `message_id` when not provided, improving error handling and validation.

* remove .env import breaking build

* add updated at to the get chat handler

* zmall runtime error fix

* permission tests passing

* return updated at on the get chat handler now

* slq parser fixes

* Implement chat access control logic and add comprehensive tests

- Developed the `canUserAccessChat` function to determine user access to chats based on direct permissions, collection permissions, creator status, and organizational roles.
- Introduced helper functions for checking permissions and retrieving chat information.
- Added integration tests to validate access control logic, covering various scenarios including direct permissions, collection permissions, and user roles.
- Created unit tests to ensure the correctness of the access control function with mocked database interactions.
- Included simple integration tests to verify functionality with existing database data.

* sql parser and int tests working.

* fix test and lint issues

* comment to kick off deployment lo

* access controls on datasets

* electric context bug fix with sql helpers.

* permission and read only

* Add lru-cache dependency and export cache management functions

- Added `lru-cache` as a dependency in the access-controls package.
- Exported new cache management functions from `chats-cached` module, including `canUserAccessChatCached`, `getCacheStats`, `resetCacheStats`, `clearCache`, `invalidateAccess`, `invalidateUserAccess`, and `invalidateChatAccess`.

* packages deploy as well

* wrong workflow lol

* Update AppVerticalCodeSplitter.tsx

* Add error handling for query run and SQL save operations

Co-authored-by: natemkelley <natemkelley@gmail.com>

* Trim whitespace from input values before sending chat prompts

Co-authored-by: natemkelley <natemkelley@gmail.com>

* type in think-and-prep

* use the cached access chat

* update package version

* new asset import message

* Error fallback for login

* Update BusterChart.BarChart.stories.tsx

* Staging changes to fix number card titles, combo chart axis, and using dynamic filters (#386)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* db init command pass through

* combo chart fixes (#387)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* clarifying question and connection logic

* pino pretty error fix

* clarifying is a finishing tool

* change update latest version logic

* Update support endpoint

* fixes for horizontal bar charts and added the combo chart logic to update metrics (#388)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* permission fix on dashboard metric handlers for workspace and data admin

* Add more try catches

* Hide avatar is no more

* Horizontal bar fixes (#389)

* fixes for horizontal bar charts and added the combo chart logic to update metrics

* hopefully fixed horizontal bar charts

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* reasoning shimmer update

* Make the embed flow work with versions

* new account warning update

* Move support modal

* compact number for pie label

* Add final reasoning message tracking and workflow start time to chunk processor and related steps

- Introduced `finalReasoningMessage` to schemas in `analyst-step`, `mark-message-complete-step`, and `create-todos-step`.
- Updated `ChunkProcessor` to calculate and store the final reasoning message based on workflow duration.
- Enhanced various steps to utilize the new `workflowStartTime` for better tracking of execution duration.
- Improved database update logic to include `finalReasoningMessage` when applicable.

* 9 digit cutoff for pie

* trigger update

* test on mastra braintrust

* test deployment

* testing

* pnpm install

* pnpm

* node 22

* pnpm version

* trigger main

* get initial chat file

* hono main deploymenbt

* clear timeouts

* Remove console logs

* migration test to staging

* db url

* try again

* k get rid of tls var

* hmmm lets try this

* mark migrations

* fix migration file?

* drizzle-kit upgrade

* tweaks to the github actions

* clean up workflows db migration

* cert location copy moved

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>
2025-07-02 14:47:44 -07:00
dal 58f9d261e6
Mastra braintrust (#391)
* type fixes

* biome clean on ai

* add user to flag chat

* attempt to get vercel deployed

* Update tsup.config.ts

* Update pnpm-lock.yaml

* Add @buster/server2 Hono API app with Vercel deployment configuration

* slack oauth integration

* mainly some clean up and biome formatting

* slack oauth

* slack migration + snapshot

* remove unused files

* finalized docker image for porter

* Create porter_app_buster-server_3155.yml file

* Add integration tests for Slack handler and refactor Slack OAuth service

- Introduced integration tests for the Slack handler, covering OAuth initiation, callback handling, and integration status retrieval.
- Refactored Slack OAuth service to improve error handling and ensure proper integration state management.
- Updated token storage implementation to use a database vault instead of Supabase.
- Enhanced existing tests for better coverage and reliability, including cleanup of test data.
- Added new utility functions for managing vault secrets in the database.

* docker image update

* new prompts

* individual tests and a schema fix

* server build

* final working dockerfile

* Update Dockerfile

* new messages to slack messages (#369)

* Update dockerfile

* Update validate-env.js

* update build pipeline

* Update the dockerfile flow

* finalize logging for pino

* stable base

* Update cors middleware logger

* Update cors.ts

* update docker to be more imformative

* Update index.ts

* Update auth.ts

* Update cors.ts

* Update cors.ts

* Update logger.ts

* remove logs

* more cors updates

* build server shared

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies. (#370)

* tons of file parsing errors (#371)

* Refactor PostgreSQL credentials handling and remove unused memory storage. Update package dependencies.

* tons of file parsing errors

* Dev mode updates

* more stable electric handler

* Dal/agent-self-healing-fixes (#372)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* Dal/agent-self-healing-fixes (#373)

* change to 6 min

* optmizations around saving and non-blocking actions.

* stream optimizations

* change porter staging deploy to mastra-braintrust.

* new path for porter deploy

* deploy to staging fix

* Create porter_app_mastra-braintrust-api_3155.yml file (#375)

Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>

* Update sizing and opacity

* supe up the instance for mastra

* environment staging

* ssl script

* copy path

* Update list padding

* no throttle and the anthropic cached

* move select to the top

* Update margin inline start

* shrink reasoning vertical space to 2px

* semi bold font for headers

* update animation timing

* haiku

* Add createTodoList tool and integrate into create-todos-step

* chat helper on post chat

* only trigger cicd when change made

* Start created streaming text components

* Refactor analyst agent task to initialize Braintrust logging asynchronously and parallelize database queries for improved performance. Adjusted cleanup timeout for Braintrust traces to reduce delays.

* fixed reasoned for X, so that it rounds down to the minute

* Update users page

* update build pipeline for new web

* document title update

* Named chats for page

* Datasets titles

* Refactor visualization tools and enhance error handling in retryable agent stream. Removed unused metricValueLabel from metrics file tool, updated metric configuration schemas, and improved healing mechanism for tool errors during streaming.

* analyst

* document title updates

* Update useDocumentTitle.tsx

* Refactor tool choice configuration in create-todos-step to use structured object. Remove exponential backoff logic from retryable agent stream for healable errors. Introduce new test for real-world healing scenarios in retryable agent stream.

* Refactor SQL validation logic in modify-metrics-file-tool to skip unnecessary checks when SQL has not changed. Enhance error handling and update validation messages. Clean up code formatting for improved readability.

* update collapse for filecard

* chevron collapse

* Jacob prompt changes (#376)

* prompt changes to improve filtering logic and handle priv/sec errors

* prompt changes to make aggregation better and improved filter best practices

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/agents/think-and-prep-agent/think-and-prep-instructions.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* Update packages/ai/src/steps/create-todos-step.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: dal <dallin@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* think and prep

* change header and strong fonts weights

* Update get collection

* combo chart x axis update

* Create a chart schemas as types

* schema types

* simple unit tests for line chart props

* fix the response file ordering iwth active selection.

* copy around reasoning messages taken care of

* fix nullable user message and file processing and such.

* update ticks for chart config

* fix todo parsing.

* app markdown update

* Update splitter to use border instead of width

* change ml

* If no file is found we should auto redirect

* Refactor database connection handling to support SSL modes. Introduced functions to extract SSL parameters and manage connections based on SSL requirements, including a custom verifier for unverified connections.

* black box message update

* chat title updates

* optimizations for trigger.

* some keepalive logic on the anthropic cached

* keep title empty until new one

* no duplicate messages

* null user message on asset pull

* posthog error handling

* 20 sec idle timeout on anthropic

* null req message

* fixed modificiation names missing

* Refactor tool call handling to support new content array format in asset messages and context loaders

* cache most recent file from workflow

* Enhance date and number detection in createDataMetadata function to improve data type handling for metrics files

* group hover effect for message

* logging for chat

* Add messageId handling and file association tracking in dashboard and metrics tools

- Updated runtime context to include messageId in create and modify dashboard and metrics file tools.
- Implemented file association tracking based on messageId in create and modify functions for both dashboards and metrics.
- Ensured type consistency by using AnalystRuntimeContext in runtime context parameters.

* logging for chat

* message type update

* Route to first file instead

* trigger moved to catalog

* Enhance file selection logic to support YAML parsing and improve logging

- Updated `extractMetricIdsFromDashboard` to first attempt JSON parsing, falling back to a regex-based YAML parsing for metric IDs.
- Added detailed debug logging in `selectFilesForResponse` to track file selection process, including metrics and dashboards involved.
- Introduced tests for various scenarios in `file-selection.test.ts` to ensure correct behavior with dashboard context and edge cases.

* trigger dev v4-beta

* Retry + Self Healing (#381)

* Refactor retry logic in analyst and think-and-prep steps

Co-authored-by: dallin <dallin@buster.so>

* some fixes

* console log error

* self healing

* todos retry

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* remove lots of logs

* Remove chat streaming

* Remove chat streaming

* timeout

* Change to updated at field

* link to home

* Update timeout settings for HTTP and HTTPS agents from 20 seconds to 10 seconds for improved responsiveness.

* Add utils module and integrate message conversion in post_chat_handler

* Implement error handling for extract values (#382)

* Remove chat streaming

* Improve error handling and logging in extract values and chat title steps

Co-authored-by: dallin <dallin@buster.so>

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>

* loading icon for buster avatar

* finalize tooltip cache

* upgrade mastra

* increase retries

* Add redo functionality for chat messages

- Introduced `redoFromMessageId` parameter in `handleExistingChat` to allow users to specify a message to redo from.
- Implemented validation to ensure the specified message belongs to the current chat.
- Added `softDeleteMessagesFromPoint` function to soft delete a message and all subsequent messages in the same chat, facilitating the redo feature.

* fix electric potential memory leak

* tooltip cache and chart cleanup

* Update bullet to be more indented

* latest version number

* add support endpoint to new server

* Fix jank in combo bar charts

* index check for dashboard

* Collapse only if there are metrics

* Is finished reasoing back

* Update dependencies and enhance chat message handling

- Upgraded `@mastra/core` to version 0.10.8 and added `node-sql-parser` at version 5.3.10 in the lock file.
- Improved integration tests for chat message redo functionality, ensuring correct behavior when deriving `chat_id` from `message_id`.
- Enhanced error handling and validation in the `initializeChat` function to manage cases where `chat_id` is not provided.

* Update pnpm-lock and enhance chat message integration tests

- Added `node-sql-parser` version 5.3.10 to dependencies and updated the lock file.
- Improved integration tests for chat message redo functionality, ensuring accurate deletion and retrieval of messages.
- Enhanced the `initializeChat` function to derive `chat_id` from `message_id` when not provided, improving error handling and validation.

* remove .env import breaking build

* add updated at to the get chat handler

* zmall runtime error fix

* permission tests passing

* return updated at on the get chat handler now

* slq parser fixes

* Implement chat access control logic and add comprehensive tests

- Developed the `canUserAccessChat` function to determine user access to chats based on direct permissions, collection permissions, creator status, and organizational roles.
- Introduced helper functions for checking permissions and retrieving chat information.
- Added integration tests to validate access control logic, covering various scenarios including direct permissions, collection permissions, and user roles.
- Created unit tests to ensure the correctness of the access control function with mocked database interactions.
- Included simple integration tests to verify functionality with existing database data.

* sql parser and int tests working.

* fix test and lint issues

* comment to kick off deployment lo

* access controls on datasets

* electric context bug fix with sql helpers.

* permission and read only

* Add lru-cache dependency and export cache management functions

- Added `lru-cache` as a dependency in the access-controls package.
- Exported new cache management functions from `chats-cached` module, including `canUserAccessChatCached`, `getCacheStats`, `resetCacheStats`, `clearCache`, `invalidateAccess`, `invalidateUserAccess`, and `invalidateChatAccess`.

* packages deploy as well

* wrong workflow lol

* Update AppVerticalCodeSplitter.tsx

* Add error handling for query run and SQL save operations

Co-authored-by: natemkelley <natemkelley@gmail.com>

* Trim whitespace from input values before sending chat prompts

Co-authored-by: natemkelley <natemkelley@gmail.com>

* type in think-and-prep

* use the cached access chat

* update package version

* new asset import message

* Error fallback for login

* Update BusterChart.BarChart.stories.tsx

* Staging changes to fix number card titles, combo chart axis, and using dynamic filters (#386)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* db init command pass through

* combo chart fixes (#387)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* clarifying question and connection logic

* pino pretty error fix

* clarifying is a finishing tool

* change update latest version logic

* Update support endpoint

* fixes for horizontal bar charts and added the combo chart logic to update metrics (#388)

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* permission fix on dashboard metric handlers for workspace and data admin

* Add more try catches

* Hide avatar is no more

* Horizontal bar fixes (#389)

* fixes for horizontal bar charts and added the combo chart logic to update metrics

* hopefully fixed horizontal bar charts

---------

Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>

* reasoning shimmer update

* Make the embed flow work with versions

* new account warning update

* Move support modal

* compact number for pie label

* Add final reasoning message tracking and workflow start time to chunk processor and related steps

- Introduced `finalReasoningMessage` to schemas in `analyst-step`, `mark-message-complete-step`, and `create-todos-step`.
- Updated `ChunkProcessor` to calculate and store the final reasoning message based on workflow duration.
- Enhanced various steps to utilize the new `workflowStartTime` for better tracking of execution duration.
- Improved database update logic to include `finalReasoningMessage` when applicable.

* 9 digit cutoff for pie

* trigger update

* test on mastra braintrust

* test deployment

* testing

* pnpm install

* pnpm

* node 22

* pnpm version

* trigger main

* get initial chat file

* hono main deploymenbt

* clear timeouts

* Remove console logs

* migration test to staging

* db url

* try again

* k get rid of tls var

* hmmm lets try this

* mark migrations

* fix migration file?

* drizzle-kit upgrade

* tweaks to the github actions

---------

Co-authored-by: Nate Kelley <nate@buster.so>
Co-authored-by: porter-deployment-app[bot] <87230664+porter-deployment-app[bot]@users.noreply.github.com>
Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com>
Co-authored-by: Jacob Anderson <jacobanderson@Jacobs-MacBook-Air.local>
Co-authored-by: jacob-buster <jacob@buster.so>
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: Cursor Agent <cursoragent@cursor.com>
Co-authored-by: natemkelley <natemkelley@gmail.com>
2025-07-02 14:33:40 -07:00
dal 152b901f8f
windows update for github actions 2025-06-12 08:07:45 -06:00
dal e47ce0d842
feat: enhance Windows build process in CLI release workflow
- Added support for building and compressing the Windows CLI artifact, including the installation of PostgreSQL binaries for libpq.
- Updated the workflow to include the necessary environment variables for PostgreSQL and adjusted artifact upload paths.
- Improved the overall structure and clarity of the Windows-specific build steps.
2025-06-12 07:39:02 -06:00
Nate Kelley 2c866f14e9
Create collapse sidebar
- Created a nice little sidebar for the navigation
- Ripped out old AppSplitter in favor a new pixel based on (with better animations)
- Preserved the chat layout on page refresh
2025-06-02 22:15:59 -06:00
Nate Kelley f98e23f15f
Adjust linting (#338)
* eslint step 1

* update additional packages

* fix some linting errors

* fix some problems

* Fix some linting things
2025-05-30 13:22:56 -06:00
Nate Kelley d4567d52cb
Biome linter test (#336)
* update files for use with biom

* fix prettier formats

* minor biome fixes

* fix additional files

* update linting rules

* fix additional linting errors

* fix linting errors

* update lib

* run check in the lib direcotyr

* update some linting errors

* fix problems

* move addtional files to stricter linting

* fix imports and linting errors

* update some biome settings

* fix query parser

* quick wins

* update files

* fix addtional things

* fix context files

* update additional biome files

* fix additional files

* biome fixes

* fixin files

* fix broken

* fix additional files and problems

* fix biome buster stuff

* fix helpers and

* stricter linting

* fixed

* fix missing types

* fix linting

* remove missing imports

* update more biome test stuff

* update fixes

* update imports

* more updates

* fix some linting errors

* fix broken children as prop errors

* lint ci

* update lint check

* Update package.json

* Update next.config.mjs

* fix some broken tests

* update some tests

* chore(versions): bump api to v0.1.16; bump web to v0.1.16; bump cli to v0.1.16 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-unit-tests.yml

* fix some tests

* chore(versions): bump api to v0.1.17; bump web to v0.1.17; bump cli to v0.1.17 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* update ignore files

* chore(versions): bump api to v0.1.18; bump web to v0.1.18; bump cli to v0.1.18 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-lint.yml

* chore(versions): bump api to v0.1.19; bump web to v0.1.19; bump cli to v0.1.19 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* update boime package settings

* chore(versions): bump api to v0.1.20; bump web to v0.1.20; bump cli to v0.1.20 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* save to dropdown bug fixes

* update console logs

* update chart animation

* chore(versions): bump api to v0.1.21; bump web to v0.1.21; bump cli to v0.1.21 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* update build

* chore(versions): bump api to v0.1.22; bump web to v0.1.22; bump cli to v0.1.22 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* additinal updates for biome 2

* update imports

* chore(versions): bump api to v0.1.23; bump web to v0.1.23; bump cli to v0.1.23 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* final updates

* chore(versions): bump api to v0.1.24; bump web to v0.1.24; bump cli to v0.1.24 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update dashboard-updates.test.ts

* chore(versions): bump api to v0.1.25; bump web to v0.1.25; bump cli to v0.1.25 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* crazy fix

* chore(versions): bump api to v0.1.26; bump web to v0.1.26; bump cli to v0.1.26 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* more attempts for stable tests

* chore(versions): bump api to v0.1.27; bump web to v0.1.27; bump cli to v0.1.27 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* update tests

* split the test off

* chore(versions): bump api to v0.1.28; bump web to v0.1.28; bump cli to v0.1.28 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update pie-styling-updates.spec.ts

* Create web-e2e-tests-optimized.yml

* chore(versions): bump api to v0.1.29; bump web to v0.1.29; bump cli to v0.1.29 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* update tests

* chore(versions): bump api to v0.1.30; bump web to v0.1.30; bump cli to v0.1.30 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-e2e-tests-optimized.yml

* chore(versions): bump api to v0.1.31; bump web to v0.1.31; bump cli to v0.1.31 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-e2e-tests-optimized.yml

* chore(versions): bump api to v0.1.32; bump web to v0.1.32; bump cli to v0.1.32 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-e2e-tests-optimized.yml

* chore(versions): bump api to v0.1.33; bump web to v0.1.33; bump cli to v0.1.33 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-e2e-tests-optimized.yml

* chore(versions): bump api to v0.1.34; bump web to v0.1.34; bump cli to v0.1.34 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-e2e-tests-optimized.yml

* Update web-e2e-tests-optimized.yml

* Revert "Update web-e2e-tests-optimized.yml"

This reverts commit dfc9263a26.

* Update web-e2e-tests-optimized.yml

* chore(versions): bump api to v0.1.35; bump web to v0.1.35; bump cli to v0.1.35 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

* Update web-e2e-tests-optimized.yml

* chore(versions): bump api to v0.1.36; bump web to v0.1.36; bump cli to v0.1.36 [skip ci]

* chore: update tag_info.json with potential release versions [skip ci]

---------

Co-authored-by: github-actions[bot] <github-actions[bot]@users.noreply.github.com>
2025-05-30 11:16:48 -06:00
dal 3269d634c0
Merge pull request #309 from buster-so/dal/local-deployment
Dal/local-deployment
2025-05-12 10:18:19 -07:00
dal 6257cd7a51
wrong file for buster.rb 2025-05-09 17:04:26 -06:00
dal c30b6b893b
workflow dispatch on brew 2025-05-09 16:18:41 -06:00
dal 6c8ae281bb
update brew workflow 2025-05-09 16:06:45 -06:00
dal 69faf4faf9
makefile fix, test env fixes, auth 2025-05-09 12:13:27 -06:00
Nate Kelley 8ecea82a88
Merge pull request #289 from buster-so/staging
Staging
2025-05-08 13:35:35 -06:00
dal 0a3bb5db01
envs on docker and cli tweak 2025-05-08 04:00:42 -06:00
dal b441c32ae0
cli test 2025-05-08 03:45:34 -06:00
dal 56acfcab1b
fix on release and run 2025-05-08 03:25:59 -06:00
dal ef8ba36e6d
Merge pull request #287 from buster-so/staging
ok some quick changes for bugs
2025-05-08 02:13:18 -07:00
dal abf09eed6b
ok some quick changes for bugs 2025-05-08 03:12:44 -06:00
dal 44f8406131
ok release both rn with old file 2025-05-08 02:05:34 -06:00
dal f08ef35270
cli release for homebrew deploy, backwards compatibiltiy on model types 2025-05-08 01:55:58 -06:00
dal 415b964930
ok remove windows and test 2025-05-08 01:34:55 -06:00
dal 9a8297c747
cli with config commands 2025-05-08 01:24:46 -06:00
dal 3d19d8eab0
cli release and locla oai setup 2025-05-08 01:12:57 -06:00
dal 2d1ded6643
lets test the cli release 2025-05-08 00:03:15 -06:00
dal fb0077c583
docker release update to merge to latest. run updates, migrate rerank 2025-05-07 18:25:34 -06:00
dal d3f637e0e8
next public web socket url 2025-05-07 12:47:07 -06:00
dal a901ee382e
main env on local build 2025-05-07 12:16:25 -06:00
dal 1b16398413
docker release with arm64 2025-05-07 08:47:08 -06:00
dal a24033fb63
window error 2025-05-07 08:37:11 -06:00
dal 771bc3b7d2
cli release with libpq error resolve 2025-05-07 08:33:17 -06:00
dal 256bbbb934
threw in old workflow with the github tap.... 2025-05-07 08:19:46 -06:00
dal 4751cf13f0
remove linux arm v7 2025-05-07 08:15:46 -06:00
dal 13c12be36e
platform support for docker images 2025-05-07 08:07:15 -06:00
dal 8d28993bb4
ok libpq error 2025-05-07 08:00:52 -06:00
dal 3410475c2d
docker release public 2025-05-07 07:56:09 -06:00
dal a96a422b63
new docker image locations and cli fix 2025-05-07 07:43:24 -06:00
dal f9d644d030
caching and cli fix 2025-05-07 07:29:51 -06:00
dal 4adca4ecc5
docker release fix 2025-05-07 07:23:51 -06:00
dal 732984206b
version bump 2025-05-07 07:16:15 -06:00
dal b16a6d8b17
on pr to main 2025-05-06 18:37:15 -06:00
dal 771256d182
releases 2025-05-06 18:33:44 -06:00
dal 5ab8c65da4
ok tag triggers 2025-05-06 18:23:43 -06:00
dal d3949cbd71
one more try 2025-05-06 18:09:28 -06:00
dal c06c5bac94
ok try again 2025-05-06 18:00:24 -06:00
dal 13f1ddeba5
try again 2025-05-06 17:52:21 -06:00
dal 5fe0864027
test cli release 2025-05-06 17:42:53 -06:00
dal 60ab1c87a6
k try again with pat 2025-05-06 17:37:00 -06:00
dal 067627141b
another attempt 2025-05-06 17:24:41 -06:00
dal 39e78a5fc9
version bump 2025-05-06 17:19:44 -06:00
dal 3088ef501b
try differrent bump method 2025-05-06 17:14:45 -06:00
dal 76b301e38d
rust 2025-05-06 17:08:31 -06:00
dal 91e308ee9b
fix on gh action for release 2025-05-06 17:05:45 -06:00
dal 254fc4420f
manage versions 2025-05-06 17:01:16 -06:00
dal 1bd94819b1
testing actions 2025-05-06 17:00:12 -06:00
dal f8377d7168
test the releasess 2025-05-06 16:57:57 -06:00
dal 06730ca837
workflow changes and version 0.1.0 2025-05-06 16:23:50 -06:00
dal 881cd5913a
docker release points to gh 2025-05-06 10:45:58 -06:00
dal ff348745c3
Implement CLI configuration and model discovery for projects
- Updated BusterConfig structure to better support project contexts
- Added get_context_for_path method to determine project context for a file
- Enhanced resolve_model_paths to respect the project hierarchy
- Updated deploy.rs to use the revised model discovery and configuration inheritance
- Added unit tests and integration tests for project config functionality

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>
2025-05-06 08:50:56 -06:00
dal e66843b1f1
prds, docker release, models 2025-05-06 08:09:31 -06:00
dal e322b6cff9
setup.sql in the env setup 2025-05-02 16:31:09 -06:00
Nate Kelley c048bb30c6
additional tests 2025-05-02 15:15:17 -06:00
Nate Kelley 13f720637a
Playwright (Codegen) Smoke tests (#257)
* smoke test for loading pages

* write additional tests

* headless options are passed in

* added some additional checks

* added list clicking tests

* Update web/playwright-tests/ask-question-navigate-pages.spec.ts

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>

* added comments

* fix broken unit tests

* update to auth.json

* Update web-testing.yml

* update playwright to use 1 worker

* Update web-testing.yml

* Update web-testing.yml

* Update web-testing.yml

* Update web-testing.yml

* add a few options for ci

* seed script update

* anonymous sign ins

* additional unit tests

* Skip some broken tests

---------

Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
Co-authored-by: dal <dallin@buster.so>
2025-05-02 11:27:46 -06:00
dal 892fba069a
Merge branch 'staging' into evals 2025-05-01 15:19:41 -06:00
Nate Kelley ebe4f91ed0
Nate/new data matrix (#253)
* updates for efficiency

* working on a single function

* update

* working

* update

* optimized

* update

* trendline operations pt 1

* update datasets to include ticks

* dataset creator

* udate

* trendlines

* move to one single dataset options hook

* started adding in options

* fix some broken functions

* line series builder

* threadshold updates

* pie sort by

* additional fixes to data structures

* update

* update animations

* update to y axis combos

* more stable asset prop passing

* Update ChartJSTheme.ts

* update check

* update problematic dataset

* with additional tests for size

* size data key

* update unit tests

* add a virtualizer for the rows

* add a few scatter tests

* update

* add additionla unit tests

* add additionla unit tests

* update

* add a couple more tests

* many updates

* update builder to not use deprcated functinos

* added additional unit tests

* add a few more unit tests

* make a few new unit tests

* make tooltip a little more efficient

* builders have some unit tests

* make additional tests

* added a bunch of unit tests

* move mount logic closer to buster chart

* fix link click bug

* adjust the icons

* fix broken time stamps

* additional config for tooltips

* add one additional unit test

* update package versions

* update the tests to run on the branch

* Update web-tests.yml
2025-05-01 13:47:34 -06:00
dal 09eae9eacb
api testing had the rust tool chain removed 2025-04-29 13:41:40 -06:00
dal 9905f16ed1
a few optimizations 2025-04-29 11:24:34 -06:00
dal 6aa1442b06
dockerfile added to test env 2025-04-29 11:02:08 -06:00
dal 5d4b4e8124
always run tests 2025-04-29 10:57:04 -06:00
dal 19eea365a4
api, cli, web, tsting env and some rust fixes 2025-04-29 10:37:37 -06:00
dal 0c5c9eafff
ok trying again with env setup 2025-04-29 10:12:27 -06:00
dal a735c8bc57
Update .github/workflows/web-testing.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-29 09:00:43 -07:00
dal 6d0b7b4f20
Update .github/actions/setup-test-environment/action.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-29 09:00:36 -07:00
dal 5385d76d11
Update .github/actions/stop-supabase/action.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-29 09:00:12 -07:00
dal 39199e2d5f
modularize the testing a bit... 2025-04-29 09:57:48 -06:00
dal e0da15c05f
api testing env 2025-04-29 09:51:10 -06:00
dal 9d6675a643
Update .github/workflows/web-testing.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-29 08:46:53 -07:00
dal 9945d4aa4e
Update .github/workflows/web-testing.yml
Co-authored-by: greptile-apps[bot] <165735046+greptile-apps[bot]@users.noreply.github.com>
2025-04-29 08:46:36 -07:00
dal 4cc3c014ee
web testin 2025-04-29 09:43:42 -06:00
dal d39a34dd68
speed up main 2025-04-21 14:21:47 -06:00
dal 6113ff6b92
fix evals not to run on nate push 2025-04-04 16:54:49 -06:00
Nate Kelley 813b05c18a
Merge branch 'staging' into big-nate/bus-939-create-new-structure-for-chats 2025-03-04 10:53:16 -07:00
dal e582f89e0b
get rid of preview 2025-02-26 08:06:02 -07:00
dal 89e92cf81f
ok better version handling 2025-02-26 07:51:04 -07:00
dal a70389b4e8
bigquery support 2025-02-26 07:45:22 -07:00
dal 9c7e217077
some good logging and config tracking 2025-02-25 20:11:35 -07:00
dal 013af2be71
version 0.0.5 for release. 2025-02-25 12:17:30 -07:00
dal 116677c0ab
bugfix - cli missing types and deploy 2025-02-18 13:35:07 -07:00
dal 04bc32cc6f
get rid of aws bc no need 2025-02-17 07:43:52 -07:00
dal faf64df107
change vm? 2025-02-17 07:41:42 -07:00
dal a99712358e
hardcode the region? 2025-02-17 07:37:08 -07:00
dal d853233c4f
indentation is the problem lol? 2025-02-17 07:33:35 -07:00
dal 029b9bbcd4
add correct actions to evals branch 2025-02-17 07:25:11 -07:00
porter-deployment-app[bot] 6ad029bc7d
Create porter_app_evals_3155.yml file 2025-02-17 13:11:08 +00:00
porter-deployment-app[bot] 2a3000fb59
Create porter_preview_staging_3155.yml file 2025-02-14 23:16:27 +00:00
dal 1a4e43b0d9
lol manual version upgrade 2025-02-12 14:20:34 -07:00
dal ffbcedb8d5
lol upgrade version 2025-02-12 13:32:23 -07:00
dal caec707450
Add macOS ARM64 support and enhance CLI authentication and version management 2025-02-12 13:00:29 -07:00
dal 5fe8d54d0c
shasum command on action 2025-02-12 09:32:04 -07:00
dal 8d78b97725
Add cross-platform build support for CLI release workflow 2025-02-12 09:28:06 -07:00
dal ef4fcdbd00
my fault for not double checking the static version placeholder for now 2025-02-12 09:14:07 -07:00
dal 3cf320075b
version on cli 2025-02-12 09:10:52 -07:00
dal 4a6898fb09
update package name 2025-02-12 09:07:11 -07:00
dal f6e8422b10
update versions on cli release 2025-02-12 09:04:36 -07:00
dal 09ab45bbb5
cli release code 2025-02-12 09:00:02 -07:00
dal fe375a72e8
fix: deployment db migration 2025-01-22 12:59:55 -07:00
Nate Kelley 84387396f0
feat: added permission pages
* chore: add release-please configuration

* create virtua list component

* only debounce if there is text

* prefetch on demand

* add a popup for permissions

* update package versions

* Make users page (#39)

* create users pages

* abstract more components to correct folders

* carve out expection if we are hiding the select all

* user query

* Add ability to change default access

* Update package-lock.json

* Update package-lock.json

* Update web/src/components/list/BusterList/BusterListReactWindow.tsx

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Big nate/bus 924 make additional permissions pages (#50)

* create virtua list component

* only debounce if there is text

* prefetch on demand

* add a popup for permissions

* Update web/src/components/list/BusterList/BusterListReactWindow.tsx

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* finalize permission popup

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* more elegant infinite list component

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* add additional bulk popup menus

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* make pages unique

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* change how padding is applied to list

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* infinite list component

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* start user dataset lineage

Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>

* Refactor SQL query in list_assets_handler to use a Common Table Expression (CTE) for improved readability and maintainability. The CTE, `distinct_assets`, simplifies the selection of distinct asset records before applying the final ordering and limiting.

* add cursor rules for web directory + jest (#52)

* on change update for segments

* PRevent clicking passthrough on users click

* feat: Add assets module and nest routes in user router

- Introduced a new `assets` module to handle asset-related routes.
- Updated the user router to nest the `assets` routes under the user ID path, enhancing the organization of API endpoints.
- This change improves the structure and maintainability of the user-related routes in the API.

* feat: Enhance user attribute listing with authorization checks

- Updated the `list_attributes_handler` to include authorization checks for user roles and organization IDs.
- Implemented error handling for unauthorized access to user attributes.
- Refactored the SQL query to retrieve user attributes based on the authenticated user's organization, improving security and data integrity.
- This change ensures that only authorized users can list attributes, enhancing the overall security of the API.

* pass through dataset overview

* feat: Update dataset group listing to include permissions

- Enhanced the `list_dataset_groups` function to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset group.
- Modified the `DatasetGroupInfo` struct to include `permission_id` and `assigned` fields, reflecting the new data structure.
- Refactored the SQL query to group by necessary fields and ensure accurate permission data is returned, improving the functionality and security of dataset group listings.

* feat: Add DatasetGroupPermission model and schema

- Introduced a new `DatasetGroupPermission` struct in `models.rs` to represent permissions associated with dataset groups.
- Updated the database schema in `schema.rs` to include the `dataset_groups_permissions` table, defining its structure and relationships.
- Modified the `is_user_workspace_admin_or_data_admin` function in `checks.rs` to correctly reference the user's organization role, enhancing role validation logic.

* add component for users inputs

* feat: Update dataset group listing to include dataset group permissions

- Modified the `list_dataset_groups` function to accept an additional `id` parameter for filtering dataset groups based on user permissions.
- Updated the SQL query to join with the `dataset_groups_permissions` table, allowing retrieval of permission counts for each dataset group.
- Refactored the `DatasetGroupInfo` struct to replace `permission_id` with `permission_count`, enhancing clarity and accuracy in the data representation.
- Ensured that the query groups by the new permission structure, improving the functionality and security of dataset group listings.

* feat: Refactor dataset listing to include user-specific permissions

- Updated the `list_datasets` function to accept an additional `id` parameter for filtering datasets based on user permissions.
- Enhanced the SQL query to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset.
- Refactored the `DatasetInfo` struct to include an `assigned` field, improving clarity in the dataset representation.
- Improved error handling for dataset retrieval, ensuring robust logging and response management.

* fix: Correct user role attribute and enhance read-only logic in list_attributes_handler

- Updated the user role attribute key from "role" to "organization_role" for accurate role retrieval.
- Introduced a read-only flag for specific user attributes, improving data integrity by clearly indicating which attributes should not be modified.
- Enhanced error handling for user role retrieval, ensuring robust responses for missing or incorrect attributes.

* normalize header for list

* new line

* popup conatiner for users

* feat: Enhance user authorization checks and refactor related functions

- Added user authorization checks in `list_attributes`, `list_dataset_groups`, `list_datasets`, `list_permission_groups`, and `list_teams` functions to ensure only users with appropriate roles can access these resources.
- Refactored the `list_teams_handler` to accept `user_id` as a parameter, improving clarity and consistency across user-related functions.
- Updated SQL queries to utilize the new authorization checks, enhancing security and data integrity.
- Removed redundant column allowances in `list_teams` permissions, streamlining the codebase.

* pass last child as index

* feat: Enhance permission group handling and streamline SQL queries

- Expanded the `allow_columns_to_appear_in_same_group_by_clause!` macro in `models.rs` to include additional columns for datasets and users, improving query flexibility.
- Refactored the `list_permission_groups` function to include dataset count and assigned status, enhancing the information returned for each permission group.
- Updated SQL queries in `list_permission_groups` to utilize left joins for better data retrieval and to ensure accurate permission checks.
- Removed redundant column allowances in various files, streamlining the codebase and improving maintainability.

* feat: Add PUT route for updating teams in user assets

- Introduced a new module `put_teams` to handle updates for teams.
- Added a PUT route for `/teams` in the user assets router, allowing for team modifications.
- Enhanced the routing capabilities of the user assets API to support both GET and PUT requests for teams.

* onchagne appsegmetned update

* create permission user endpoints

* move files to match new page structure

* refactor: Clean up routing and improve PUT teams handler

- Reformatted imports in `mod.rs` for better readability.
- Commented out the PUT route for `/teams` in the user assets router, indicating a potential future change or deprecation.
- Updated the `put_teams` handler to return a `NoContent` response upon successful execution, enhancing clarity in API responses.
- Improved error handling in the `put_teams` function for better logging and response management.

* refactor: Standardize user ID parameter naming across user-related routes

- Updated all user-related route handlers to use `user_id` instead of `id` for better clarity and consistency.
- Modified the routing definitions in `mod.rs` to reflect the new parameter naming convention.
- Enhanced the `list_permission_groups` function to accept `user_id` as a parameter, improving clarity in the handler's signature.
- Ensured all relevant functions now consistently handle the `user_id` parameter, streamlining the codebase and improving maintainability.

* feat: Enhance team management with role-based assignments

- Introduced a new `TeamInfoRole` enum to represent user roles within teams, replacing the previous boolean `assigned` field.
- Updated the `list_teams` handler to return team roles instead of assignment status, improving clarity on user roles.
- Refactored the `put_teams` handler to support role-based assignments, allowing for more granular control over team memberships.
- Added new PUT routes for dataset groups and permission groups in the user assets router, enhancing API capabilities.
- Improved SQL queries for team assignments to utilize role information, streamlining database interactions.

* feat: Add organization_id to DatasetGroupPermission and update dataset group handler

- Introduced a new `organization_id` field in the `DatasetGroupPermission` struct to associate permissions with specific organizations.
- Updated the `put_dataset_groups_handler` to include `organization_id` when creating or updating dataset group permissions, enhancing the API's capability to manage permissions at the organizational level.
- Improved SQL query formatting for better readability in the handler.

* add list components for permission settings

* feat: Introduce assets module and update routing for permission groups

- Added a new `assets` module to organize related routes.
- Updated the routing in `mod.rs` to nest the `assets` router under the `/:permission_group_id` path, enhancing the structure and clarity of the API.
- Maintained existing routes for managing permission groups while improving modularity.

* create permission group users

* refactor: Update list_permission_groups_handler to use user_id and improve SQL queries

- Changed the parameter in the SQL query from `user.id` to `user_id` for consistency with the updated user ID parameter naming convention.
- Enhanced the SQL query to count distinct dataset permissions and utilize `bool_or` for identity checks, improving accuracy and performance.
- Cleaned up the grouping in the SQL query by removing unnecessary fields, streamlining the data retrieval process.

* add listing for dataset groups

* feat: Add PUT routes for user and dataset management in assets module

- Introduced new PUT routes for managing users and dataset groups in the assets module.
- Updated the router to support PUT requests for `/users`, `/dataset_groups`, and `/datasets`, enhancing the API's functionality for resource updates.
- Improved modularity by organizing related routes within the assets module.

* add listing for dataset groups

* add list for datasets

* assigned permissions

* Add attributes and teams

* feat: Add DatasetToDatasetGroup model and update schema

- Introduced a new `DatasetToDatasetGroup` struct to represent the relationship between datasets and dataset groups, including fields for timestamps and optional deletion.
- Updated the database schema to include `updated_at` and `deleted_at` fields for the `datasets_to_dataset_groups` table, enhancing data tracking capabilities.
- Refactored the routing in `mod.rs` to include a nested router for assets, improving the organization of dataset group routes.

* invalidate query if user id is present

* create a dataset modal added to user page

* add team modal to teams

* assigned popup

* feat: Enhance user retrieval with dataset information

- Added new structs `DatasetLineage` and `DatasetInfo` to represent dataset details and lineage.
- Updated `UserResponse` to include a list of datasets associated with the user.
- Refactored `get_user_information` function to concurrently fetch user info, direct datasets, permission group datasets, and organization datasets using `tokio::spawn` for improved performance.
- Implemented logic to compile datasets based on direct access and permission group access, including lineage tracking for better data representation.
- Enhanced error handling during database queries to ensure robust user information retrieval.

* feat: Enhance dataset access retrieval in user and dataset overview

- Updated `get_dataset_overview` to include dataset group access and permission group to dataset group access, improving the granularity of dataset permissions.
- Introduced new queries to fetch dataset groups and their associated permissions, enhancing the dataset overview for users.
- Refactored `get_user_information` to concurrently retrieve dataset groups and permission group datasets, optimizing performance with `tokio::spawn`.
- Enhanced lineage tracking for datasets, allowing for better representation of user permissions across dataset groups and permission groups.
- Improved error handling during database queries to ensure robust data retrieval.

* lineage props passed

* refactor: Streamline dataset access logic in get_user_information

- Simplified access control logic for datasets based on user roles, consolidating conditions for WorkspaceAdmin, DataAdmin, Querier, Viewer, and RestrictedQuerier.
- Enhanced dataset lineage tracking to provide clearer representation of user permissions across various dataset access types.
- Removed redundant code related to dataset processing, improving readability and maintainability of the `get_user_information` function.
- Ensured that datasets are correctly categorized based on direct access, permission group access, and organization datasets, optimizing the overall data retrieval process.

* feat: Refactor dataset overview access lineage in get_dataset_overview

- Introduced a default access lineage for users, ensuring consistent representation of user permissions.
- Simplified the addition of user roles to the lineage, consolidating logic for WorkspaceAdmin, DataAdmin, Querier, and Viewer roles.
- Enhanced lineage tracking for RestrictedQuerier role to include direct dataset access and permission group lineage, improving granularity of dataset permissions.
- Removed redundant code related to dataset and permission group lineage, optimizing readability and maintainability of the `get_dataset_overview` function.

* add datasetgroup handler

* feat: Improve dataset access control and lineage tracking

- Enhanced the `get_dataset_overview` function to refine access control for the `RestrictedQuerier` role, allowing for more granular permission checks based on various access paths.
- Updated the `get_user_information` function to streamline dataset processing, ensuring that datasets are categorized correctly based on direct access and permission group access.
- Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability of both functions.
- Improved lineage tracking for datasets, providing a clearer representation of user permissions across different access types.

* clickable pills

* fix build errors

* add endpoints for dataset groups, and permission groups

* refactor: Enhance dataset access control and lineage tracking

- Streamlined the `get_dataset_overview` function to improve access control for the `RestrictedQuerier` role, ensuring more precise permission checks.
- Updated the `get_user_information` function to optimize dataset processing, categorizing datasets based on direct access and permission group access.
- Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability.
- Enhanced lineage tracking for datasets, providing a clearer representation of user permissions across different access types.

* better handling for create a permission group

* Refetch on team created for now

* add additional pages

* permission group modal update

* added pages for permission groups

* add user permission list

* add datasets to permission groups page

* Update dependencies and refactor Snowflake query handling

- Downgraded the `base64` crate version in `Cargo.toml` from `0.22.1` to `0.21`.
- Refactored the `snowflake_query` function in `snowflake_query.rs` to improve data type handling, including support for additional Arrow data types and enhanced null value checks.
- Updated the `route_to_query` function in `query_router.rs` to use mutable `snowflake_client` for better state management during query execution.
- Improved error handling for closing the Snowflake client session, ensuring proper logging of any issues encountered.

* add datasets to permission groups page

* Refactor Snowflake client connection to remove warehouse and database IDs

- Updated the `get_snowflake_client` function to no longer require `warehouse_id` and `database_id`, simplifying the connection process.
- This change enhances flexibility in client initialization and aligns with recent updates to Snowflake API handling.

* add permission groups lists

* add dataset group in permission group area

---------

Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com>
Co-authored-by: dal <dallin@buster.so>
2025-01-22 11:25:06 -08:00
Nate Kelley 5b0e221738
fix: stablize versioning 2025-01-16 15:08:47 -07:00
Nate Kelley 3016e16afb
fix: update the prerelease logic 2025-01-16 15:05:19 -07:00
Nate Kelley 6a67afe5cb
fix: use a node pipeline 2025-01-16 14:53:09 -07:00
Nate Kelley 20ba771a8d
fix: update README 2025-01-16 14:37:32 -07:00
Nate Kelley 7c26d63929
fix(main): this is a push to main to trigger a release 2025-01-16 14:23:48 -07:00
Nate Kelley 9732dfd6b7
fix(main): update release-please 2025-01-16 14:12:03 -07:00
Nate Kelley 95e991b10b
chore: add release-please configuration 2025-01-16 13:49:28 -07:00
Nate Kelley e8ee2ccac1
fix: release please update 2025-01-16 13:10:58 -07:00
Nate Kelley da7f2a64ee
chore: update invalid fields 2025-01-16 13:01:37 -07:00
Nate Kelley 05202a3ddb
feat: update path to version file 2025-01-16 12:56:55 -07:00
Nate Kelley da422c0a14
Merge branch 'main' into staging 2025-01-16 11:09:31 -08:00
Nate Kelley dcd1d51fc7
fix: Add release please syntax handler and github action (#40)
* fix(buster): Add release please syntax handler and github action

* chore: add version tracking setup

fix: update

update nate rulez
2025-01-16 12:08:21 -07:00
Nate Kelley f1ee35b890
chore: add semvar version update (#42)
* added empty state text

* update permission group in the dataset

* Enhance dataset asset listing with organization-specific filtering

- Updated the `list_assets` function to include organization ID filtering in dataset permissions queries.
- Removed redundant organization ID filters from the dataset permissions queries to streamline the logic.
- Ensured that only relevant dataset assets are returned based on the user's organization, improving data security and relevance.

These changes enhance the API's ability to serve organization-specific data, aligning with recent improvements in dataset asset APIs.

* containerized class should be white with no border at bottom

* clear query when signing out

* Use correct endpoint for dataset groups

* yaml syntax highligting

* create dataset endpoints

* update disable logic for deploying a dataset

* Refactor user routes to include new endpoint for retrieving user by ID

- Removed the public modifier from `get_user` and `update_user` modules to encapsulate them within the module.
- Added a new route to the user router for fetching a user by their ID, enhancing the API's functionality.
- This change improves the user management capabilities by allowing retrieval of specific user details based on their unique identifier.

* Add organizations module and integrate with user routes

* remove unused imports and abstract variables

* Refactor user update functionality to support role changes

- Enhanced the `update_user` endpoint to accept and process user role updates alongside name changes.
- Introduced a new `UserResponse` struct for improved response handling.
- Updated the `update_user_handler` to handle changes in both user name and organization role, improving the flexibility of user management.
- Adjusted response type to return no content upon successful updates, aligning with RESTful practices.

These changes enhance the user management capabilities by allowing for more comprehensive updates to user information.

* Update user route to use ID parameter for updates

- Changed the user update route to require a user ID in the URL, enhancing RESTful practices.
- Updated the `update_user` function to extract the user ID from the path, ensuring the correct user is updated based on the provided ID.

These changes improve the clarity and functionality of the user update endpoint, aligning it with standard REST conventions.

* simplify hooks imports

* Remove unused component

* restructure folders for layout

* update imports for gloabl components

* add additional routes

* Implement user permission checks in dataset deployment and user update routes

- Added permission validation to the `deploy_datasets` and `post_dataset` functions to ensure only users with workspace admin or data admin roles can execute these actions.
- Enhanced error handling for permission checks, returning appropriate HTTP status codes and messages for insufficient permissions and internal errors.
- Updated imports to include the new security checks module for consistency across routes.

These changes improve security by enforcing role-based access control in critical dataset operations.

* Refactor user update route to enhance RESTful practices

- Updated the user update route to require a user ID in the URL, ensuring the correct user is updated based on the provided ID.
- Improved clarity and functionality of the `update_user` function by extracting the user ID from the path.

These changes align the user update endpoint with standard REST conventions, enhancing overall API usability.

* Enhance dataset listing functionality with user organization roles

- Refactored dataset listing logic to incorporate user organization roles, allowing for more granular access control based on user permissions.
- Introduced new role checks for `WorkspaceAdmin`, `DataAdmin`, `Querier`, `RestrictedQuerier`, and `Viewer` to determine dataset visibility.
- Updated database queries to fetch datasets based on user roles and organization associations, improving data retrieval efficiency.
- Removed deprecated functions and streamlined the dataset fetching process, ensuring clarity and maintainability in the codebase.

These changes improve the API's security and usability by enforcing role-based access control for dataset operations.

* tweaked the post thread permissions handle.

* permission_group string fix

* remove package.json

* fix: Add release please syntax handler and github action (#40)

* fix(buster): Add release please syntax handler and github action

* chore: add version tracking setup

fix: update

update nate rulez

---------

Co-authored-by: dal <dallin@buster.so>
2025-01-16 11:02:12 -08:00
Nate Kelley 22e23d6d97
Create a pipeline to update versions
chore: bump version to 0.0.2
2025-01-14 13:21:12 -07:00
dal f7bb098ba9 Enhance GitHub Actions workflows and Dependabot configuration
- Added new Dependabot configurations for npm packages in the /api and /web directories, scheduled for weekly updates with a limit of 10 open pull requests.
- Updated the GitHub Actions workflows for both main and staging deployments to trigger on changes within the api directory, specifically for migrations, improving deployment reliability.

These changes streamline dependency management and enhance the deployment process for the application.
2025-01-06 12:10:42 -07:00
dal 488bc596dc Refactor GitHub Actions workflow for enhanced deployment process
- Added a new job for database deployment, including steps for installing Rust, configuring AWS credentials, downloading the Postgres certificate from S3, and running migrations.
- Updated the existing porter-deploy job to include AWS credentials configuration and SSL certificate download from S3.
- Improved organization and reliability of the deployment process by ensuring all steps are correctly defined and executed in the appropriate context.

These changes streamline the deployment workflow and enhance the overall reliability of the application deployment process.
2025-01-03 16:49:28 -07:00
porter-deployment-app[bot] a34312f64f
Create porter_app_main_3155.yml file 2025-01-03 23:46:52 +00:00
dal c23ca4c36e Update GitHub Actions workflow to set working directory for migrations
- Added a working-directory specification for the 'Run migrations' step in the GitHub Actions workflow, ensuring that Diesel migrations are executed in the correct context within the './api' directory. This change enhances the reliability of the migration process during deployment.
2025-01-03 16:39:26 -07:00
dal 501b8031a4 Update GitHub Actions workflow to use new S3 URL for Postgres SSL certificate
- Changed the S3 URL variable for downloading the Postgres SSL certificate in the GitHub Actions workflow from `PG_CERT_S3_URL` to `CERT_S3_URL`.
- This update ensures the workflow correctly references the updated secret for SSL certificate retrieval, improving the reliability of the deployment process.
2025-01-03 16:29:21 -07:00
dal 5f6792484a Enhance GitHub Actions workflow and Dockerfile for SSL certificate management
- Added steps to configure AWS credentials and download the Postgres SSL certificate from S3 in the GitHub Actions workflow.
- Updated the Dockerfile to reflect the new path for the SSL certificate, ensuring it is correctly copied to the container.

These changes improve the security and organization of SSL certificate handling during the deployment process.
2025-01-03 16:25:39 -07:00
dal 0ef11c1971 Update SSL certificate handling in GitHub Actions workflow and Dockerfile
- Changed the directory structure for SSL certificates in the GitHub Actions workflow, creating a new path `certs/cert` for better organization.
- Updated the Dockerfile to reference the new certificate path, ensuring the SSL certificate is correctly installed in the container.

These changes improve the clarity and maintainability of the deployment process.
2025-01-03 16:19:02 -07:00
dal 11ee658d39 Fix Ubuntu version format in GitHub Actions workflow for staging deployment 2025-01-03 16:18:07 -07:00
dal 78da6550d1 Fix GitHub Actions workflow configuration for staging deployment
- Updated the runner specification for the database-deploy job to correct the Ubuntu version format.
- Ensured consistency in the runner configuration for the porter-deploy job by reverting to the previous Ubuntu version.

These changes enhance the accuracy of the workflow configuration, ensuring proper execution of deployment jobs.
2025-01-03 16:17:24 -07:00
dal 31bfaaeecf Update GitHub Actions workflow and Dockerfile for SSL certificate handling
- Added steps to create a directory for SSL certificates and copy the downloaded certificate to the new location in the GitHub Actions workflow.
- Updated the Dockerfile to reference the new certificate path, ensuring proper installation of the SSL certificate in the container.

These changes improve the organization of SSL certificates and enhance the deployment process.
2025-01-03 16:12:31 -07:00
dal fe6cf9d945 Enhance GitHub Actions workflow for staging deployment
- Added support for pull requests targeting the staging branch to the existing workflow.
- This change improves the deployment process by ensuring that pull requests are also considered for staging deployments.
2025-01-03 15:59:25 -07:00
dal 03f522dddb Refactor GitHub Actions workflow for staging deployment
- Introduced a new job for database deployment, including steps to install Rust, cache dependencies, and run Diesel migrations.
- Updated the existing porter-deploy job to include AWS credentials configuration and SSL certificate retrieval from S3.
- Enhanced the overall structure and organization of the workflow for improved clarity and maintainability.
2025-01-03 15:56:44 -07:00