feat: implement documentation agent workflow with context initialization

- Added new steps for the documentation agent, including context initialization, todo creation, and main processing logic.
- Introduced schemas for input and output validation using Zod.
- Created a new workflow that orchestrates the steps for generating documentation based on user input.
- Enhanced the existing file structure to support the new documentation agent functionality.
This commit is contained in:
dal 2025-07-23 11:32:50 -06:00
parent b043bcd03d
commit 4f3f8f3810
No known key found for this signature in database
GPG Key ID: 16F4B0E1E9F61122
7 changed files with 677 additions and 10 deletions

View File

@ -1,8 +1,7 @@
--- ---
description:
globs:
alwaysApply: true alwaysApply: true
--- ---
# CLAUDE.md # CLAUDE.md
This file provides guidance to Claude Code when working with code in this monorepo. This file provides guidance to Claude Code when working with code in this monorepo.
@ -54,6 +53,58 @@ When writing code, follow this workflow to ensure code quality:
- Keep business logic separate from infrastructure concerns - Keep business logic separate from infrastructure concerns
- Use proper error handling at each level - Use proper error handling at each level
## Environment Variables
This project uses a centralized environment variable system:
1. **All environment variables are defined at the root level** in a single `.env` file
2. **Turbo passes these variables** to all packages via the `globalEnv` configuration in `turbo.json`
3. **Individual packages validate** their required environment variables using the shared `@buster/env-utils` package
### Setting Up Environment Variables
1. Copy `.env.example` to `.env` at the project root:
```bash
cp .env.example .env
```
2. Fill in the required values in `.env`
3. All packages will automatically have access to these variables through Turbo
### Adding New Environment Variables
When adding new environment variables:
1. Add the variable to `.env.example` with a descriptive comment
2. Add the variable name to the `globalEnv` array in `turbo.json`
3. Update the package's `validate-env.js` script to include the new variable
4. Update the package's `env.d.ts` file with the TypeScript type definition
### Migrating Packages to Centralized Env
To migrate an existing package to use the centralized environment system:
1. Remove any local `.env` files from the package
2. Add `@buster/env-utils` as a dependency:
```json
"@buster/env-utils": "workspace:*"
```
3. Update the package's `scripts/validate-env.js` to use the shared utilities:
```javascript
import { loadRootEnv, validateEnv } from '@buster/env-utils';
loadRootEnv();
const requiredEnv = {
DATABASE_URL: process.env.DATABASE_URL,
// ... other required variables
};
const { hasErrors } = validateEnv(requiredEnv);
if (hasErrors) process.exit(1);
```
### 3. Ensure Type Safety ### 3. Ensure Type Safety
```bash ```bash
# Build entire monorepo to check types # Build entire monorepo to check types
@ -202,6 +253,13 @@ export async function getWorkspaceSettingsHandler(
## Database Operations ## Database Operations
### Query Organization
- **All database queries must be created as helper functions** in `@packages/database/src/queries/`
- **Organize by table** - Each table should have its own subdirectory (e.g., `assets/`, `chats/`, `users/`)
- **Type all queries** - Every query function must have properly typed parameters and return types
- **Export from index** - Each subdirectory should have an `index.ts` that exports all queries for that table
- **Reusable and composable** - Write queries as small, focused functions that can be composed together
### Soft Delete and Upsert Practices ### Soft Delete and Upsert Practices
- In our database, we never hard delete, we always use soft deletes with the `deleted_at` field - In our database, we never hard delete, we always use soft deletes with the `deleted_at` field
- For update operations, we should almost always perform an upsert unless otherwise specified - For update operations, we should almost always perform an upsert unless otherwise specified
@ -211,9 +269,271 @@ export async function getWorkspaceSettingsHandler(
- When running tests, use the following Turbo commands: - When running tests, use the following Turbo commands:
- `turbo test:unit` for unit tests - `turbo test:unit` for unit tests
- `turbo test:integration` for integration tests - `turbo test:integration` for integration tests
- `turbo test` for running all tests - `turbo test` for running all tests# CLAUDE.md
# important-instruction-reminders
Do what has been asked; nothing more, nothing less. This file provides guidance to Claude Code when working with code in this monorepo.
NEVER create files unless they're absolutely necessary for achieving your goal.
ALWAYS prefer editing an existing file to creating a new one. **Note**: Many packages and apps have their own CLAUDE.md files with specific implementation details and patterns. Always check for a local CLAUDE.md when working in a specific directory.
NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.
## Monorepo Structure
This is a pnpm-based monorepo using Turborepo with the following structure:
### Apps (`@buster-app/*`)
- `apps/web` - Next.js frontend application
- `apps/server` - Node.js/Hono backend server
- `apps/trigger` - Background job processing with Trigger.dev v3
- `apps/electric-server` - Electric SQL sync server
- `apps/api` - Rust backend API (legacy)
- `apps/cli` - Command-line tools (Rust)
### Packages (`@buster/*`)
- `packages/ai` - AI agents, tools, and workflows using Mastra framework
- `packages/database` - Database schema, migrations, and utilities (Drizzle ORM)
- `packages/data-source` - Data source adapters (PostgreSQL, MySQL, BigQuery, Snowflake, etc.)
- `packages/access-controls` - Permission and access control logic
- `packages/stored-values` - Stored values management
- `packages/rerank` - Document reranking functionality
- `packages/server-shared` - Shared server types and utilities
- `packages/test-utils` - Shared testing utilities
- `packages/vitest-config` - Shared Vitest configuration
- `packages/typescript-config` - Shared TypeScript configuration
- `packages/web-tools` - Web scraping and research tools
- `packages/slack` - Standalone Slack integration (OAuth, messaging, channels)
- `packages/supabase` - Supabase setup and configuration
- `packages/sandbox` - Sandboxed code execution using Daytona SDK
## Development Workflow
When writing code, follow this workflow to ensure code quality:
### 1. Write Modular, Testable Functions
- Create small, focused functions with single responsibilities
- Design functions to be easily testable with clear inputs/outputs
- Use dependency injection for external dependencies
- **IMPORTANT: Write functional, composable code - avoid classes**
- All features should be composed of testable functions
- Follow existing patterns in the codebase
### 2. Build Features by Composing Functions
- Combine modular functions to create complete features
- Keep business logic separate from infrastructure concerns
- Use proper error handling at each level
## Environment Variables
This project uses a centralized environment variable system:
1. **All environment variables are defined at the root level** in a single `.env` file
2. **Turbo passes these variables** to all packages via the `globalEnv` configuration in `turbo.json`
3. **Individual packages validate** their required environment variables using the shared `@buster/env-utils` package
### Setting Up Environment Variables
1. Copy `.env.example` to `.env` at the project root:
```bash
cp .env.example .env
```
2. Fill in the required values in `.env`
3. All packages will automatically have access to these variables through Turbo
### Adding New Environment Variables
When adding new environment variables:
1. Add the variable to `.env.example` with a descriptive comment
2. Add the variable name to the `globalEnv` array in `turbo.json`
3. Update the package's `validate-env.js` script to include the new variable
4. Update the package's `env.d.ts` file with the TypeScript type definition
### Migrating Packages to Centralized Env
To migrate an existing package to use the centralized environment system:
1. Remove any local `.env` files from the package
2. Add `@buster/env-utils` as a dependency:
```json
"@buster/env-utils": "workspace:*"
```
3. Update the package's `scripts/validate-env.js` to use the shared utilities:
```javascript
import { loadRootEnv, validateEnv } from '@buster/env-utils';
loadRootEnv();
const requiredEnv = {
DATABASE_URL: process.env.DATABASE_URL,
// ... other required variables
};
const { hasErrors } = validateEnv(requiredEnv);
if (hasErrors) process.exit(1);
```
### 3. Ensure Type Safety
```bash
# Build entire monorepo to check types
turbo run build
# Build specific package/app
turbo run build --filter=@buster/ai
turbo run build --filter=@buster-app/web
# Type check without building
turbo run typecheck
turbo run typecheck --filter=@buster/database
```
### 4. Run Biome for Linting & Formatting
```bash
# Check files with Biome
pnpm run check path/to/file.ts
pnpm run check packages/ai
# Auto-fix linting, formatting, and import organization
pnpm run check:fix path/to/file.ts
pnpm run check:fix packages/ai
```
### 5. Run Tests with Vitest
**Important**: Always run unit tests before completing any task to ensure code changes don't break existing functionality.
```bash
# Run unit tests (always run these when working locally)
turbo run test:unit
# Run unit tests for specific package
turbo run test:unit --filter=@buster/ai
# Run integration tests ONLY for specific features/packages you're working on
turbo run test:integration --filter=@buster/database
# Run specific test file
pnpm run test path/to/file.test.ts
# Watch mode for development
pnpm run test:watch
```
### 6. Pre-Completion Checklist
**IMPORTANT: Before finishing any task or creating PRs, always run:**
```bash
# 1. Run unit tests for the entire monorepo
turbo run test:unit
# 2. Build the entire monorepo to ensure everything compiles
turbo run build
# 3. Run linting for the entire monorepo
turbo run lint
```
**Key Testing Guidelines:**
- **Always run unit tests, build, and lint** when working locally before considering a task complete
- **Unit tests** should be run for the entire monorepo to catch any breaking changes
- **Build** must pass for the entire monorepo to ensure type safety
- **Integration tests** should only be run for specific packages/features you're working on (NOT the entire monorepo)
- **Fix all failing tests, build errors, and lint errors** before completing any task
- **Heavily bias toward unit tests** - they are faster and cheaper to run
- **Mock everything you can** in unit tests for isolation and speed
## Code Quality Standards
### TypeScript Configuration
- **Strict mode enabled** - All strict checks are on
- **No implicit any** - Always use specific types
- **Strict null checks** - Handle null/undefined explicitly
- **No implicit returns** - All code paths must return
- **Consistent file casing** - Enforced by TypeScript
### Type Safety and Zod Best Practices
- We care deeply about type safety and we use Zod schemas and then export them as types
- We prefer using type abstractions over `.parse()` method calls
- Always export Zod schemas as TypeScript types to leverage static type checking
- Avoid runtime type checking when compile-time type checks are sufficient
### Biome Rules (Key Enforcements)
- **`useImportType: "warn"`** - Use type-only imports when possible
- **`noExplicitAny: "error"`** - Never use `any` type
- **`noUnusedVariables: "error"`** - Remove unused code
- **`noNonNullAssertion: "error"`** - No `!` assertions
- **`noConsoleLog: "warn"`** - Avoid console.log in production
- **`useNodejsImportProtocol: "error"`** - Use `node:` prefix for Node.js imports
### Logging Guidelines
- **Never use `console.log`**
- **Use appropriate console methods**:
- `console.info` for general information
- `console.warn` for warning messages
- `console.error` for error messages
## Error Handling and Logging Philosophy
- We care deeply about error handling and logging
- Key principles for error management:
- Catch errors effectively and thoughtfully
- Consider the state errors put the system into
- Implement comprehensive unit tests for error scenarios
- Log errors strategically for effective debugging
- Avoid over-logging while ensuring sufficient context for troubleshooting
## Hono API Development Guidelines
### API Structure and Organization
- **Version-based organization** - APIs are organized under `/api/v2/` directory
- **Feature-based folders** - Each feature gets its own folder (e.g., `chats/`, `security/`)
- **Separate handler files** - Each endpoint handler must be in its own file
- **Functional handlers** - All handlers should be pure functions that accept request data and return response data
### Request/Response Type Safety
- **Use shared types** - All request and response types must be defined in `@buster/server-shared`
- **Zod schemas** - Define schemas in server-shared and export both the schema and inferred types
- **zValidator middleware** - Always use `zValidator` from `@hono/zod-validator` for request validation
- **Type imports** - Import types from server-shared packages for consistency
### Handler Pattern
```typescript
// Handler file (e.g., get-workspace-settings.ts)
import type { GetWorkspaceSettingsResponse } from '@buster/server-shared/security';
import type { User } from '@buster/database';
export async function getWorkspaceSettingsHandler(
user: User
): Promise<GetWorkspaceSettingsResponse> {
// Implementation
}
// Route definition (index.ts)
.get('/workspace-settings', async (c) => {
const user = c.get('busterUser');
const response = await getWorkspaceSettingsHandler(user);
return c.json(response);
})
```
### Authentication and User Context
- **Use requireAuth middleware** - Apply to all protected routes
- **Extract user context** - Use `c.get('busterUser')` to get the authenticated user
- **Type as User** - Import `User` type from `@buster/database` for handler parameters
## Database Operations
### Query Organization
- **All database queries must be created as helper functions** in `@packages/database/src/queries/`
- **Organize by table** - Each table should have its own subdirectory (e.g., `assets/`, `chats/`, `users/`)
- **Type all queries** - Every query function must have properly typed parameters and return types
- **Export from index** - Each subdirectory should have an `index.ts` that exports all queries for that table
- **Reusable and composable** - Write queries as small, focused functions that can be composed together
### Soft Delete and Upsert Practices
- In our database, we never hard delete, we always use soft deletes with the `deleted_at` field
- For update operations, we should almost always perform an upsert unless otherwise specified
```
**Test Running Guidelines**:
- When running tests, use the following Turbo commands:
- `turbo test:unit` for unit tests
- `turbo test:integration` for integration tests
- `turbo test` for running all tests

View File

@ -0,0 +1,87 @@
import type { Sandbox } from '@buster/sandbox';
import { createStep } from '@mastra/core';
import type { RuntimeContext } from '@mastra/core/runtime-context';
import { z } from 'zod';
import { DocsAgentContextKey } from '../../context/docs-agent-context';
const createDocsTodosStepInputSchema = z.object({
message: z.string(),
organizationId: z.string(),
contextInitialized: z.boolean(),
context: z.object({
sandbox: z.any(),
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const createDocsTodosStepOutputSchema = z.object({
todos: z.array(z.string()),
todoList: z.string(),
// Pass through other fields
message: z.string(),
organizationId: z.string(),
context: z.object({
sandbox: z.any(),
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const createDocsTodosExecution = async ({
inputData,
runtimeContext,
}: {
inputData: z.infer<typeof createDocsTodosStepInputSchema>;
runtimeContext: RuntimeContext;
}): Promise<z.infer<typeof createDocsTodosStepOutputSchema>> => {
// Get the sandbox from runtime context (it was set by initializeContextStep)
const sandbox = runtimeContext.get(DocsAgentContextKey.Sandbox) as Sandbox;
const currentTodoList = runtimeContext.get(DocsAgentContextKey.TodoListFile) as string;
// TODO: Implement the logic to create documentation todos
// This step should analyze the message and create a list of documentation tasks
// Now you have access to the sandbox from runtime context!
// The sandbox and currentTodoList will be used in the actual implementation
// For now, log to show they're available
console.info('Sandbox available:', !!sandbox);
console.info('Current todo list:', currentTodoList);
const todos: string[] = [];
// Update the runtime context with the new todo list
const updatedTodoList = todos.join('\n');
runtimeContext.set(DocsAgentContextKey.TodoListFile, updatedTodoList);
// Return the data with todos
return {
...inputData,
todos,
todoList: updatedTodoList,
context: {
...inputData.context,
todoList: updatedTodoList,
},
};
};
export const createDocsTodosStep = createStep({
id: 'create-docs-todos',
description: 'Creates a list of documentation todos based on the user message',
inputSchema: createDocsTodosStepInputSchema,
outputSchema: createDocsTodosStepOutputSchema,
execute: createDocsTodosExecution,
});

View File

@ -0,0 +1,104 @@
import type { Sandbox } from '@buster/sandbox';
import { createStep } from '@mastra/core';
import type { RuntimeContext } from '@mastra/core/runtime-context';
import { z } from 'zod';
import { DocsAgentContextKey } from '../../context/docs-agent-context';
import type { MessageUserClarifyingQuestion } from '../../context/docs-agent-context';
const docsAgentStepInputSchema = z.object({
todos: z.array(z.string()),
todoList: z.string(),
message: z.string(),
organizationId: z.string(),
context: z.object({
sandbox: z.any(),
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const docsAgentStepOutputSchema = z.object({
todos: z.array(z.string()).optional(),
todoList: z.string().optional(),
documentationCreated: z.boolean().optional(),
clarificationNeeded: z.boolean().optional(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
finished: z.boolean().optional(),
metadata: z
.object({
filesCreated: z.number().optional(),
toolsUsed: z.array(z.string()).optional(),
})
.optional(),
});
const docsAgentExecution = async ({
inputData,
runtimeContext,
}: {
inputData: z.infer<typeof docsAgentStepInputSchema>;
runtimeContext: RuntimeContext;
}): Promise<z.infer<typeof docsAgentStepOutputSchema>> => {
// Access values from runtime context
const sandbox = runtimeContext.get(DocsAgentContextKey.Sandbox) as Sandbox;
const todoList = runtimeContext.get(DocsAgentContextKey.TodoListFile) as string;
const clarificationQuestion = runtimeContext.get(DocsAgentContextKey.ClarificationFile) as
| MessageUserClarifyingQuestion
| undefined;
// Also access standard workflow context
const organizationId = runtimeContext.get('organizationId') as string;
const workflowStartTime = runtimeContext.get('workflowStartTime') as number;
console.info('[DocsAgent] Runtime context values:', {
hasSandbox: !!sandbox,
todoList,
hasClarificationQuestion: !!clarificationQuestion,
organizationId,
workflowStartTime,
});
// TODO: Implement the docs agent logic
// This step should:
// 1. Process the todos using the sandbox
// 2. Create documentation files
// 3. Handle clarification questions if needed
// Example of how to use sandbox (when implemented):
// const result = await sandbox.execute('console.log("Hello from sandbox")');
// If you need to update clarification question:
// runtimeContext.set(DocsAgentContextKey.ClarificationFile, newClarificationQuestion);
return {
todos: inputData.todos,
todoList: inputData.todoList,
documentationCreated: false,
clarificationNeeded: false,
finished: true,
metadata: {
filesCreated: 0,
toolsUsed: [],
},
};
};
export const docsAgentStep = createStep({
id: 'docs-agent',
description: 'Main documentation agent that processes todos and creates documentation',
inputSchema: docsAgentStepInputSchema,
outputSchema: docsAgentStepOutputSchema,
execute: docsAgentExecution,
});

View File

@ -0,0 +1,87 @@
import type { Sandbox } from '@buster/sandbox';
import { createStep } from '@mastra/core';
import type { RuntimeContext } from '@mastra/core/runtime-context';
import { z } from 'zod';
import { ClarifyingQuestionSchema, DocsAgentContextKey } from '../../context/docs-agent-context';
// Input schema includes the context values we need to set
const initializeContextStepInputSchema = z.object({
message: z.string(),
organizationId: z.string(),
// Context values that will be set in runtime context
sandbox: z.custom<Sandbox>(
(val) => {
// Validate it's a sandbox instance with required methods
return (
val &&
typeof val === 'object' &&
typeof val.execute === 'function' &&
typeof val.cleanup === 'function'
);
},
{
message: 'Invalid Sandbox instance',
}
),
todoList: z.string().optional().default(''),
clarificationQuestion: ClarifyingQuestionSchema.optional(),
});
// Output schema passes through all input data plus confirms context is initialized
const initializeContextStepOutputSchema = z.object({
message: z.string(),
organizationId: z.string(),
contextInitialized: z.boolean(),
// Pass through for subsequent steps (but they'll use runtime context)
context: z.object({
sandbox: z.any(), // Will be in runtime context
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const initializeContextExecution = async ({
inputData,
runtimeContext,
}: {
inputData: z.infer<typeof initializeContextStepInputSchema>;
runtimeContext: RuntimeContext;
}): Promise<z.infer<typeof initializeContextStepOutputSchema>> => {
// Set runtime context values
runtimeContext.set(DocsAgentContextKey.Sandbox, inputData.sandbox);
runtimeContext.set(DocsAgentContextKey.TodoListFile, inputData.todoList);
if (inputData.clarificationQuestion) {
runtimeContext.set(DocsAgentContextKey.ClarificationFile, inputData.clarificationQuestion);
}
// Also set standard workflow context
runtimeContext.set('organizationId', inputData.organizationId);
runtimeContext.set('workflowStartTime', Date.now());
// Return data for next steps with context initialized
return {
message: inputData.message,
organizationId: inputData.organizationId,
contextInitialized: true,
context: {
sandbox: inputData.sandbox,
todoList: inputData.todoList,
clarificationQuestion: inputData.clarificationQuestion,
},
};
};
export const initializeContextStep = createStep({
id: 'initialize-context',
description: 'Initializes the runtime context with DocsAgent specific values',
inputSchema: initializeContextStepInputSchema,
outputSchema: initializeContextStepOutputSchema,
execute: initializeContextExecution,
});

View File

@ -34,7 +34,7 @@ describe('bash-execute-tool', () => {
describe('bashExecute tool', () => { describe('bashExecute tool', () => {
it('should have correct tool configuration', () => { it('should have correct tool configuration', () => {
expect(executeBash.id).toBe('bash_execute'); expect(executeBash.id).toBe('execute-bash');
expect(executeBash.description).toContain('Executes bash commands'); expect(executeBash.description).toContain('Executes bash commands');
expect(executeBash.inputSchema).toBeDefined(); expect(executeBash.inputSchema).toBeDefined();
expect(executeBash.outputSchema).toBeDefined(); expect(executeBash.outputSchema).toBeDefined();

View File

@ -51,7 +51,10 @@ describe('Permission Validator', () => {
}, },
] as any); ] as any);
const result = await validateSqlPermissions('SELECT id, user_id FROM public.orders', 'user123'); const result = await validateSqlPermissions(
'SELECT id, user_id FROM public.orders',
'user123'
);
expect(result).toEqual({ expect(result).toEqual({
isAuthorized: false, isAuthorized: false,

View File

@ -0,0 +1,66 @@
import type { Sandbox } from '@buster/sandbox';
import { createWorkflow } from '@mastra/core';
import { z } from 'zod';
import { ClarifyingQuestionSchema } from '../../context/docs-agent-context';
import { createDocsTodosStep } from '../../steps/docs-agent/create-docs-todos-step';
import { docsAgentStep } from '../../steps/docs-agent/docs-agent-step';
import { initializeContextStep } from '../../steps/docs-agent/initialize-context-step';
// Input schema for the workflow - now accepts context values directly
const docsAgentWorkflowInputSchema = z.object({
message: z.string(),
organizationId: z.string(),
// Direct context values instead of nested object
sandbox: z.custom<Sandbox>(
(val) => {
return (
val &&
typeof val === 'object' &&
typeof val.execute === 'function' &&
typeof val.cleanup === 'function'
);
},
{
message: 'Invalid Sandbox instance',
}
),
todoList: z.string().optional().default(''),
clarificationQuestion: ClarifyingQuestionSchema.optional(),
});
// Output schema for the workflow
const docsAgentWorkflowOutputSchema = z.object({
todos: z.array(z.string()).optional(),
todoList: z.string().optional(),
documentationCreated: z.boolean().optional(),
clarificationNeeded: z.boolean().optional(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
finished: z.boolean().optional(),
metadata: z
.object({
filesCreated: z.number().optional(),
toolsUsed: z.array(z.string()).optional(),
})
.optional(),
});
// Create the workflow with initialization step first
const docsAgentWorkflow = createWorkflow({
id: 'docs-agent-workflow',
inputSchema: docsAgentWorkflowInputSchema,
outputSchema: docsAgentWorkflowOutputSchema,
steps: [initializeContextStep, createDocsTodosStep, docsAgentStep],
})
.then(initializeContextStep) // First step: initialize runtime context
.then(createDocsTodosStep) // Then create todos
.then(docsAgentStep) // Finally run the agent
.commit();
export default docsAgentWorkflow;
export { docsAgentWorkflowInputSchema, docsAgentWorkflowOutputSchema };