feat: implement documentation agent workflow with context initialization

- Added new steps for the documentation agent, including context initialization, todo creation, and main processing logic.
- Introduced schemas for input and output validation using Zod.
- Created a new workflow that orchestrates the steps for generating documentation based on user input.
- Enhanced the existing file structure to support the new documentation agent functionality.
This commit is contained in:
dal 2025-07-23 11:32:50 -06:00
parent b043bcd03d
commit 4f3f8f3810
No known key found for this signature in database
GPG Key ID: 16F4B0E1E9F61122
7 changed files with 677 additions and 10 deletions

View File

@ -1,8 +1,7 @@
---
description:
globs:
alwaysApply: true
---
# CLAUDE.md
This file provides guidance to Claude Code when working with code in this monorepo.
@ -54,6 +53,58 @@ When writing code, follow this workflow to ensure code quality:
- Keep business logic separate from infrastructure concerns
- Use proper error handling at each level
## Environment Variables
This project uses a centralized environment variable system:
1. **All environment variables are defined at the root level** in a single `.env` file
2. **Turbo passes these variables** to all packages via the `globalEnv` configuration in `turbo.json`
3. **Individual packages validate** their required environment variables using the shared `@buster/env-utils` package
### Setting Up Environment Variables
1. Copy `.env.example` to `.env` at the project root:
```bash
cp .env.example .env
```
2. Fill in the required values in `.env`
3. All packages will automatically have access to these variables through Turbo
### Adding New Environment Variables
When adding new environment variables:
1. Add the variable to `.env.example` with a descriptive comment
2. Add the variable name to the `globalEnv` array in `turbo.json`
3. Update the package's `validate-env.js` script to include the new variable
4. Update the package's `env.d.ts` file with the TypeScript type definition
### Migrating Packages to Centralized Env
To migrate an existing package to use the centralized environment system:
1. Remove any local `.env` files from the package
2. Add `@buster/env-utils` as a dependency:
```json
"@buster/env-utils": "workspace:*"
```
3. Update the package's `scripts/validate-env.js` to use the shared utilities:
```javascript
import { loadRootEnv, validateEnv } from '@buster/env-utils';
loadRootEnv();
const requiredEnv = {
DATABASE_URL: process.env.DATABASE_URL,
// ... other required variables
};
const { hasErrors } = validateEnv(requiredEnv);
if (hasErrors) process.exit(1);
```
### 3. Ensure Type Safety
```bash
# Build entire monorepo to check types
@ -202,6 +253,13 @@ export async function getWorkspaceSettingsHandler(
## Database Operations
### Query Organization
- **All database queries must be created as helper functions** in `@packages/database/src/queries/`
- **Organize by table** - Each table should have its own subdirectory (e.g., `assets/`, `chats/`, `users/`)
- **Type all queries** - Every query function must have properly typed parameters and return types
- **Export from index** - Each subdirectory should have an `index.ts` that exports all queries for that table
- **Reusable and composable** - Write queries as small, focused functions that can be composed together
### Soft Delete and Upsert Practices
- In our database, we never hard delete, we always use soft deletes with the `deleted_at` field
- For update operations, we should almost always perform an upsert unless otherwise specified
@ -211,9 +269,271 @@ export async function getWorkspaceSettingsHandler(
- When running tests, use the following Turbo commands:
- `turbo test:unit` for unit tests
- `turbo test:integration` for integration tests
- `turbo test` for running all tests
# important-instruction-reminders
Do what has been asked; nothing more, nothing less.
NEVER create files unless they're absolutely necessary for achieving your goal.
ALWAYS prefer editing an existing file to creating a new one.
NEVER proactively create documentation files (*.md) or README files. Only create documentation files if explicitly requested by the User.
- `turbo test` for running all tests# CLAUDE.md
This file provides guidance to Claude Code when working with code in this monorepo.
**Note**: Many packages and apps have their own CLAUDE.md files with specific implementation details and patterns. Always check for a local CLAUDE.md when working in a specific directory.
## Monorepo Structure
This is a pnpm-based monorepo using Turborepo with the following structure:
### Apps (`@buster-app/*`)
- `apps/web` - Next.js frontend application
- `apps/server` - Node.js/Hono backend server
- `apps/trigger` - Background job processing with Trigger.dev v3
- `apps/electric-server` - Electric SQL sync server
- `apps/api` - Rust backend API (legacy)
- `apps/cli` - Command-line tools (Rust)
### Packages (`@buster/*`)
- `packages/ai` - AI agents, tools, and workflows using Mastra framework
- `packages/database` - Database schema, migrations, and utilities (Drizzle ORM)
- `packages/data-source` - Data source adapters (PostgreSQL, MySQL, BigQuery, Snowflake, etc.)
- `packages/access-controls` - Permission and access control logic
- `packages/stored-values` - Stored values management
- `packages/rerank` - Document reranking functionality
- `packages/server-shared` - Shared server types and utilities
- `packages/test-utils` - Shared testing utilities
- `packages/vitest-config` - Shared Vitest configuration
- `packages/typescript-config` - Shared TypeScript configuration
- `packages/web-tools` - Web scraping and research tools
- `packages/slack` - Standalone Slack integration (OAuth, messaging, channels)
- `packages/supabase` - Supabase setup and configuration
- `packages/sandbox` - Sandboxed code execution using Daytona SDK
## Development Workflow
When writing code, follow this workflow to ensure code quality:
### 1. Write Modular, Testable Functions
- Create small, focused functions with single responsibilities
- Design functions to be easily testable with clear inputs/outputs
- Use dependency injection for external dependencies
- **IMPORTANT: Write functional, composable code - avoid classes**
- All features should be composed of testable functions
- Follow existing patterns in the codebase
### 2. Build Features by Composing Functions
- Combine modular functions to create complete features
- Keep business logic separate from infrastructure concerns
- Use proper error handling at each level
## Environment Variables
This project uses a centralized environment variable system:
1. **All environment variables are defined at the root level** in a single `.env` file
2. **Turbo passes these variables** to all packages via the `globalEnv` configuration in `turbo.json`
3. **Individual packages validate** their required environment variables using the shared `@buster/env-utils` package
### Setting Up Environment Variables
1. Copy `.env.example` to `.env` at the project root:
```bash
cp .env.example .env
```
2. Fill in the required values in `.env`
3. All packages will automatically have access to these variables through Turbo
### Adding New Environment Variables
When adding new environment variables:
1. Add the variable to `.env.example` with a descriptive comment
2. Add the variable name to the `globalEnv` array in `turbo.json`
3. Update the package's `validate-env.js` script to include the new variable
4. Update the package's `env.d.ts` file with the TypeScript type definition
### Migrating Packages to Centralized Env
To migrate an existing package to use the centralized environment system:
1. Remove any local `.env` files from the package
2. Add `@buster/env-utils` as a dependency:
```json
"@buster/env-utils": "workspace:*"
```
3. Update the package's `scripts/validate-env.js` to use the shared utilities:
```javascript
import { loadRootEnv, validateEnv } from '@buster/env-utils';
loadRootEnv();
const requiredEnv = {
DATABASE_URL: process.env.DATABASE_URL,
// ... other required variables
};
const { hasErrors } = validateEnv(requiredEnv);
if (hasErrors) process.exit(1);
```
### 3. Ensure Type Safety
```bash
# Build entire monorepo to check types
turbo run build
# Build specific package/app
turbo run build --filter=@buster/ai
turbo run build --filter=@buster-app/web
# Type check without building
turbo run typecheck
turbo run typecheck --filter=@buster/database
```
### 4. Run Biome for Linting & Formatting
```bash
# Check files with Biome
pnpm run check path/to/file.ts
pnpm run check packages/ai
# Auto-fix linting, formatting, and import organization
pnpm run check:fix path/to/file.ts
pnpm run check:fix packages/ai
```
### 5. Run Tests with Vitest
**Important**: Always run unit tests before completing any task to ensure code changes don't break existing functionality.
```bash
# Run unit tests (always run these when working locally)
turbo run test:unit
# Run unit tests for specific package
turbo run test:unit --filter=@buster/ai
# Run integration tests ONLY for specific features/packages you're working on
turbo run test:integration --filter=@buster/database
# Run specific test file
pnpm run test path/to/file.test.ts
# Watch mode for development
pnpm run test:watch
```
### 6. Pre-Completion Checklist
**IMPORTANT: Before finishing any task or creating PRs, always run:**
```bash
# 1. Run unit tests for the entire monorepo
turbo run test:unit
# 2. Build the entire monorepo to ensure everything compiles
turbo run build
# 3. Run linting for the entire monorepo
turbo run lint
```
**Key Testing Guidelines:**
- **Always run unit tests, build, and lint** when working locally before considering a task complete
- **Unit tests** should be run for the entire monorepo to catch any breaking changes
- **Build** must pass for the entire monorepo to ensure type safety
- **Integration tests** should only be run for specific packages/features you're working on (NOT the entire monorepo)
- **Fix all failing tests, build errors, and lint errors** before completing any task
- **Heavily bias toward unit tests** - they are faster and cheaper to run
- **Mock everything you can** in unit tests for isolation and speed
## Code Quality Standards
### TypeScript Configuration
- **Strict mode enabled** - All strict checks are on
- **No implicit any** - Always use specific types
- **Strict null checks** - Handle null/undefined explicitly
- **No implicit returns** - All code paths must return
- **Consistent file casing** - Enforced by TypeScript
### Type Safety and Zod Best Practices
- We care deeply about type safety and we use Zod schemas and then export them as types
- We prefer using type abstractions over `.parse()` method calls
- Always export Zod schemas as TypeScript types to leverage static type checking
- Avoid runtime type checking when compile-time type checks are sufficient
### Biome Rules (Key Enforcements)
- **`useImportType: "warn"`** - Use type-only imports when possible
- **`noExplicitAny: "error"`** - Never use `any` type
- **`noUnusedVariables: "error"`** - Remove unused code
- **`noNonNullAssertion: "error"`** - No `!` assertions
- **`noConsoleLog: "warn"`** - Avoid console.log in production
- **`useNodejsImportProtocol: "error"`** - Use `node:` prefix for Node.js imports
### Logging Guidelines
- **Never use `console.log`**
- **Use appropriate console methods**:
- `console.info` for general information
- `console.warn` for warning messages
- `console.error` for error messages
## Error Handling and Logging Philosophy
- We care deeply about error handling and logging
- Key principles for error management:
- Catch errors effectively and thoughtfully
- Consider the state errors put the system into
- Implement comprehensive unit tests for error scenarios
- Log errors strategically for effective debugging
- Avoid over-logging while ensuring sufficient context for troubleshooting
## Hono API Development Guidelines
### API Structure and Organization
- **Version-based organization** - APIs are organized under `/api/v2/` directory
- **Feature-based folders** - Each feature gets its own folder (e.g., `chats/`, `security/`)
- **Separate handler files** - Each endpoint handler must be in its own file
- **Functional handlers** - All handlers should be pure functions that accept request data and return response data
### Request/Response Type Safety
- **Use shared types** - All request and response types must be defined in `@buster/server-shared`
- **Zod schemas** - Define schemas in server-shared and export both the schema and inferred types
- **zValidator middleware** - Always use `zValidator` from `@hono/zod-validator` for request validation
- **Type imports** - Import types from server-shared packages for consistency
### Handler Pattern
```typescript
// Handler file (e.g., get-workspace-settings.ts)
import type { GetWorkspaceSettingsResponse } from '@buster/server-shared/security';
import type { User } from '@buster/database';
export async function getWorkspaceSettingsHandler(
user: User
): Promise<GetWorkspaceSettingsResponse> {
// Implementation
}
// Route definition (index.ts)
.get('/workspace-settings', async (c) => {
const user = c.get('busterUser');
const response = await getWorkspaceSettingsHandler(user);
return c.json(response);
})
```
### Authentication and User Context
- **Use requireAuth middleware** - Apply to all protected routes
- **Extract user context** - Use `c.get('busterUser')` to get the authenticated user
- **Type as User** - Import `User` type from `@buster/database` for handler parameters
## Database Operations
### Query Organization
- **All database queries must be created as helper functions** in `@packages/database/src/queries/`
- **Organize by table** - Each table should have its own subdirectory (e.g., `assets/`, `chats/`, `users/`)
- **Type all queries** - Every query function must have properly typed parameters and return types
- **Export from index** - Each subdirectory should have an `index.ts` that exports all queries for that table
- **Reusable and composable** - Write queries as small, focused functions that can be composed together
### Soft Delete and Upsert Practices
- In our database, we never hard delete, we always use soft deletes with the `deleted_at` field
- For update operations, we should almost always perform an upsert unless otherwise specified
```
**Test Running Guidelines**:
- When running tests, use the following Turbo commands:
- `turbo test:unit` for unit tests
- `turbo test:integration` for integration tests
- `turbo test` for running all tests

View File

@ -0,0 +1,87 @@
import type { Sandbox } from '@buster/sandbox';
import { createStep } from '@mastra/core';
import type { RuntimeContext } from '@mastra/core/runtime-context';
import { z } from 'zod';
import { DocsAgentContextKey } from '../../context/docs-agent-context';
const createDocsTodosStepInputSchema = z.object({
message: z.string(),
organizationId: z.string(),
contextInitialized: z.boolean(),
context: z.object({
sandbox: z.any(),
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const createDocsTodosStepOutputSchema = z.object({
todos: z.array(z.string()),
todoList: z.string(),
// Pass through other fields
message: z.string(),
organizationId: z.string(),
context: z.object({
sandbox: z.any(),
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const createDocsTodosExecution = async ({
inputData,
runtimeContext,
}: {
inputData: z.infer<typeof createDocsTodosStepInputSchema>;
runtimeContext: RuntimeContext;
}): Promise<z.infer<typeof createDocsTodosStepOutputSchema>> => {
// Get the sandbox from runtime context (it was set by initializeContextStep)
const sandbox = runtimeContext.get(DocsAgentContextKey.Sandbox) as Sandbox;
const currentTodoList = runtimeContext.get(DocsAgentContextKey.TodoListFile) as string;
// TODO: Implement the logic to create documentation todos
// This step should analyze the message and create a list of documentation tasks
// Now you have access to the sandbox from runtime context!
// The sandbox and currentTodoList will be used in the actual implementation
// For now, log to show they're available
console.info('Sandbox available:', !!sandbox);
console.info('Current todo list:', currentTodoList);
const todos: string[] = [];
// Update the runtime context with the new todo list
const updatedTodoList = todos.join('\n');
runtimeContext.set(DocsAgentContextKey.TodoListFile, updatedTodoList);
// Return the data with todos
return {
...inputData,
todos,
todoList: updatedTodoList,
context: {
...inputData.context,
todoList: updatedTodoList,
},
};
};
export const createDocsTodosStep = createStep({
id: 'create-docs-todos',
description: 'Creates a list of documentation todos based on the user message',
inputSchema: createDocsTodosStepInputSchema,
outputSchema: createDocsTodosStepOutputSchema,
execute: createDocsTodosExecution,
});

View File

@ -0,0 +1,104 @@
import type { Sandbox } from '@buster/sandbox';
import { createStep } from '@mastra/core';
import type { RuntimeContext } from '@mastra/core/runtime-context';
import { z } from 'zod';
import { DocsAgentContextKey } from '../../context/docs-agent-context';
import type { MessageUserClarifyingQuestion } from '../../context/docs-agent-context';
const docsAgentStepInputSchema = z.object({
todos: z.array(z.string()),
todoList: z.string(),
message: z.string(),
organizationId: z.string(),
context: z.object({
sandbox: z.any(),
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const docsAgentStepOutputSchema = z.object({
todos: z.array(z.string()).optional(),
todoList: z.string().optional(),
documentationCreated: z.boolean().optional(),
clarificationNeeded: z.boolean().optional(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
finished: z.boolean().optional(),
metadata: z
.object({
filesCreated: z.number().optional(),
toolsUsed: z.array(z.string()).optional(),
})
.optional(),
});
const docsAgentExecution = async ({
inputData,
runtimeContext,
}: {
inputData: z.infer<typeof docsAgentStepInputSchema>;
runtimeContext: RuntimeContext;
}): Promise<z.infer<typeof docsAgentStepOutputSchema>> => {
// Access values from runtime context
const sandbox = runtimeContext.get(DocsAgentContextKey.Sandbox) as Sandbox;
const todoList = runtimeContext.get(DocsAgentContextKey.TodoListFile) as string;
const clarificationQuestion = runtimeContext.get(DocsAgentContextKey.ClarificationFile) as
| MessageUserClarifyingQuestion
| undefined;
// Also access standard workflow context
const organizationId = runtimeContext.get('organizationId') as string;
const workflowStartTime = runtimeContext.get('workflowStartTime') as number;
console.info('[DocsAgent] Runtime context values:', {
hasSandbox: !!sandbox,
todoList,
hasClarificationQuestion: !!clarificationQuestion,
organizationId,
workflowStartTime,
});
// TODO: Implement the docs agent logic
// This step should:
// 1. Process the todos using the sandbox
// 2. Create documentation files
// 3. Handle clarification questions if needed
// Example of how to use sandbox (when implemented):
// const result = await sandbox.execute('console.log("Hello from sandbox")');
// If you need to update clarification question:
// runtimeContext.set(DocsAgentContextKey.ClarificationFile, newClarificationQuestion);
return {
todos: inputData.todos,
todoList: inputData.todoList,
documentationCreated: false,
clarificationNeeded: false,
finished: true,
metadata: {
filesCreated: 0,
toolsUsed: [],
},
};
};
export const docsAgentStep = createStep({
id: 'docs-agent',
description: 'Main documentation agent that processes todos and creates documentation',
inputSchema: docsAgentStepInputSchema,
outputSchema: docsAgentStepOutputSchema,
execute: docsAgentExecution,
});

View File

@ -0,0 +1,87 @@
import type { Sandbox } from '@buster/sandbox';
import { createStep } from '@mastra/core';
import type { RuntimeContext } from '@mastra/core/runtime-context';
import { z } from 'zod';
import { ClarifyingQuestionSchema, DocsAgentContextKey } from '../../context/docs-agent-context';
// Input schema includes the context values we need to set
const initializeContextStepInputSchema = z.object({
message: z.string(),
organizationId: z.string(),
// Context values that will be set in runtime context
sandbox: z.custom<Sandbox>(
(val) => {
// Validate it's a sandbox instance with required methods
return (
val &&
typeof val === 'object' &&
typeof val.execute === 'function' &&
typeof val.cleanup === 'function'
);
},
{
message: 'Invalid Sandbox instance',
}
),
todoList: z.string().optional().default(''),
clarificationQuestion: ClarifyingQuestionSchema.optional(),
});
// Output schema passes through all input data plus confirms context is initialized
const initializeContextStepOutputSchema = z.object({
message: z.string(),
organizationId: z.string(),
contextInitialized: z.boolean(),
// Pass through for subsequent steps (but they'll use runtime context)
context: z.object({
sandbox: z.any(), // Will be in runtime context
todoList: z.string(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
}),
});
const initializeContextExecution = async ({
inputData,
runtimeContext,
}: {
inputData: z.infer<typeof initializeContextStepInputSchema>;
runtimeContext: RuntimeContext;
}): Promise<z.infer<typeof initializeContextStepOutputSchema>> => {
// Set runtime context values
runtimeContext.set(DocsAgentContextKey.Sandbox, inputData.sandbox);
runtimeContext.set(DocsAgentContextKey.TodoListFile, inputData.todoList);
if (inputData.clarificationQuestion) {
runtimeContext.set(DocsAgentContextKey.ClarificationFile, inputData.clarificationQuestion);
}
// Also set standard workflow context
runtimeContext.set('organizationId', inputData.organizationId);
runtimeContext.set('workflowStartTime', Date.now());
// Return data for next steps with context initialized
return {
message: inputData.message,
organizationId: inputData.organizationId,
contextInitialized: true,
context: {
sandbox: inputData.sandbox,
todoList: inputData.todoList,
clarificationQuestion: inputData.clarificationQuestion,
},
};
};
export const initializeContextStep = createStep({
id: 'initialize-context',
description: 'Initializes the runtime context with DocsAgent specific values',
inputSchema: initializeContextStepInputSchema,
outputSchema: initializeContextStepOutputSchema,
execute: initializeContextExecution,
});

View File

@ -34,7 +34,7 @@ describe('bash-execute-tool', () => {
describe('bashExecute tool', () => {
it('should have correct tool configuration', () => {
expect(executeBash.id).toBe('bash_execute');
expect(executeBash.id).toBe('execute-bash');
expect(executeBash.description).toContain('Executes bash commands');
expect(executeBash.inputSchema).toBeDefined();
expect(executeBash.outputSchema).toBeDefined();

View File

@ -51,7 +51,10 @@ describe('Permission Validator', () => {
},
] as any);
const result = await validateSqlPermissions('SELECT id, user_id FROM public.orders', 'user123');
const result = await validateSqlPermissions(
'SELECT id, user_id FROM public.orders',
'user123'
);
expect(result).toEqual({
isAuthorized: false,

View File

@ -0,0 +1,66 @@
import type { Sandbox } from '@buster/sandbox';
import { createWorkflow } from '@mastra/core';
import { z } from 'zod';
import { ClarifyingQuestionSchema } from '../../context/docs-agent-context';
import { createDocsTodosStep } from '../../steps/docs-agent/create-docs-todos-step';
import { docsAgentStep } from '../../steps/docs-agent/docs-agent-step';
import { initializeContextStep } from '../../steps/docs-agent/initialize-context-step';
// Input schema for the workflow - now accepts context values directly
const docsAgentWorkflowInputSchema = z.object({
message: z.string(),
organizationId: z.string(),
// Direct context values instead of nested object
sandbox: z.custom<Sandbox>(
(val) => {
return (
val &&
typeof val === 'object' &&
typeof val.execute === 'function' &&
typeof val.cleanup === 'function'
);
},
{
message: 'Invalid Sandbox instance',
}
),
todoList: z.string().optional().default(''),
clarificationQuestion: ClarifyingQuestionSchema.optional(),
});
// Output schema for the workflow
const docsAgentWorkflowOutputSchema = z.object({
todos: z.array(z.string()).optional(),
todoList: z.string().optional(),
documentationCreated: z.boolean().optional(),
clarificationNeeded: z.boolean().optional(),
clarificationQuestion: z
.object({
issue: z.string(),
context: z.string(),
clarificationQuestion: z.string(),
})
.optional(),
finished: z.boolean().optional(),
metadata: z
.object({
filesCreated: z.number().optional(),
toolsUsed: z.array(z.string()).optional(),
})
.optional(),
});
// Create the workflow with initialization step first
const docsAgentWorkflow = createWorkflow({
id: 'docs-agent-workflow',
inputSchema: docsAgentWorkflowInputSchema,
outputSchema: docsAgentWorkflowOutputSchema,
steps: [initializeContextStep, createDocsTodosStep, docsAgentStep],
})
.then(initializeContextStep) // First step: initialize runtime context
.then(createDocsTodosStep) // Then create todos
.then(docsAgentStep) // Finally run the agent
.commit();
export default docsAgentWorkflow;
export { docsAgentWorkflowInputSchema, docsAgentWorkflowOutputSchema };