mirror of https://github.com/buster-so/buster.git
Refactor CLI to TypeScript and remove Rust components
- Added TypeScript-based CLI commands for local development and execution. - Updated package.json with new CLI scripts for development and execution. - Enhanced .gitignore to exclude TypeScript build artifacts and logs. - Removed Rust-related files and configurations, including Cargo.toml, Makefile, and various source files. - Updated README.md to reflect the new TypeScript implementation and technologies used. - Cleaned up unused files and directories related to the previous Rust implementation.
This commit is contained in:
parent
7159e7e8ea
commit
71230bdaed
|
@ -0,0 +1,19 @@
|
|||
# Generated by Cargo
|
||||
# will have compiled files and executables
|
||||
debug/
|
||||
target/
|
||||
|
||||
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
|
||||
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
|
||||
Cargo.lock
|
||||
|
||||
# These are backup files generated by rustfmt
|
||||
**/*.rs.bk
|
||||
|
||||
# MSVC Windows builds of rustc generate these, which store debugging information
|
||||
*.pdb
|
||||
|
||||
.vscode/
|
||||
|
||||
/prds/completed
|
||||
|
|
@ -0,0 +1,370 @@
|
|||
# Buster CLI
|
||||
|
||||
A powerful command-line interface for managing semantic models in Buster. Deploy and manage your data models with ease, whether they're standalone or part of a dbt project.
|
||||
|
||||
## Installation
|
||||
|
||||
Choose the installation command for your operating system:
|
||||
|
||||
### macOS (x86_64)
|
||||
```bash
|
||||
mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-darwin-x86_64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster
|
||||
```
|
||||
|
||||
### macOS (ARM/Apple Silicon)
|
||||
```bash
|
||||
mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-darwin-arm64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster
|
||||
```
|
||||
|
||||
### Linux (x86_64)
|
||||
```bash
|
||||
mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-linux-x86_64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster
|
||||
```
|
||||
|
||||
> **Note**: After installation, make sure `~/.local/bin` is in your PATH. Add this to your shell's config file (`.bashrc`, `.zshrc`, etc.):
|
||||
> ```bash
|
||||
> export PATH="$HOME/.local/bin:$PATH"
|
||||
> ```
|
||||
|
||||
### Windows (x86_64)
|
||||
1. Download the Windows binary:
|
||||
```powershell
|
||||
Invoke-WebRequest -Uri https://github.com/buster-so/buster/releases/latest/download/buster-cli-windows-x86_64.zip -OutFile buster.zip
|
||||
```
|
||||
|
||||
2. Extract and install:
|
||||
```powershell
|
||||
Expand-Archive -Path buster.zip -DestinationPath $env:USERPROFILE\buster
|
||||
Move-Item -Path $env:USERPROFILE\buster\buster-cli.exe -Destination $env:LOCALAPPDATA\Microsoft\WindowsApps\buster.exe
|
||||
```
|
||||
|
||||
## Quick Start Guide
|
||||
|
||||
### 1. Authentication
|
||||
|
||||
First, authenticate with Buster using your API key:
|
||||
|
||||
```bash
|
||||
buster auth
|
||||
```
|
||||
|
||||
This will prompt you for:
|
||||
- API Key (required) - Get this from the Buster Platform
|
||||
- Host (optional) - Defaults to production if not specified
|
||||
|
||||
You can also configure authentication using environment variables:
|
||||
```bash
|
||||
# Set API key via environment variable
|
||||
export BUSTER_API_KEY=your_api_key_here
|
||||
|
||||
# Optional: Set custom host. For self-hosted instances.
|
||||
export BUSTER_HOST=your_custom_host
|
||||
```
|
||||
|
||||
The CLI will check for these environment variables in the following order:
|
||||
1. Command line arguments
|
||||
2. Environment variables
|
||||
3. Interactive prompt
|
||||
|
||||
This is particularly useful for:
|
||||
- CI/CD environments
|
||||
- Automated scripts
|
||||
- Development workflows where you don't want to enter credentials repeatedly
|
||||
|
||||
### 2. Generate Models
|
||||
|
||||
Generate Buster YAML models from your existing SQL files:
|
||||
|
||||
```bash
|
||||
buster generate
|
||||
```
|
||||
|
||||
Key flags for generation:
|
||||
- `--source-path`: Directory containing your SQL files (defaults to current directory)
|
||||
- `--destination-path`: Where to output the generated YAML files (defaults to current directory)
|
||||
- `--data-source-name`: Name of the data source to use in the models
|
||||
- `--schema`: Database schema name
|
||||
- `--database`: Database name
|
||||
- `--flat-structure`: Output YML files in a flat structure instead of maintaining directory hierarchy
|
||||
|
||||
The generate command will:
|
||||
- Scan the source directory for SQL files
|
||||
- Create corresponding YAML model files
|
||||
- Create a `buster.yml` configuration file if it doesn't exist
|
||||
- Preserve any existing model customizations
|
||||
|
||||
Example with all options:
|
||||
```bash
|
||||
buster generate \
|
||||
--source-path ./sql \
|
||||
--destination-path ./models \
|
||||
--data-source-name my_warehouse \
|
||||
--schema analytics \
|
||||
--database prod
|
||||
```
|
||||
|
||||
### 3. Deploy Models
|
||||
|
||||
Deploy your models to Buster:
|
||||
|
||||
```bash
|
||||
buster deploy
|
||||
```
|
||||
|
||||
Deploy options:
|
||||
- `--path`: Specific path to deploy (defaults to current directory)
|
||||
- `--dry-run`: Validate the deployment without actually deploying (defaults to false)
|
||||
- `--recursive`: Recursively search for model files in subdirectories (defaults to true)
|
||||
|
||||
Examples:
|
||||
```bash
|
||||
# Deploy all models in current directory
|
||||
buster deploy
|
||||
|
||||
# Deploy a specific model or directory
|
||||
buster deploy --path ./models/customers.yml
|
||||
|
||||
# Validate deployment without applying changes
|
||||
buster deploy --dry-run
|
||||
|
||||
# Deploy only models in the specified directory (not recursively)
|
||||
buster deploy --path ./models --recursive=false
|
||||
```
|
||||
|
||||
The deploy command will:
|
||||
1. Discover all YAML model files in the specified path
|
||||
2. Load and validate the models
|
||||
3. Check for excluded models based on tags
|
||||
4. Validate cross-project references
|
||||
5. Deploy the models to Buster
|
||||
6. Provide detailed validation feedback and error messages
|
||||
|
||||
## Project Structure
|
||||
|
||||
A typical Buster project structure:
|
||||
|
||||
```
|
||||
your-project/
|
||||
├── buster.yml # Global configuration
|
||||
├── models/ # Your semantic model definitions
|
||||
│ ├── customers.yml
|
||||
│ ├── orders.yml
|
||||
│ └── products.yml
|
||||
└── sql/ # SQL definitions
|
||||
├── customers.sql
|
||||
├── orders.sql
|
||||
└── products.sql
|
||||
```
|
||||
|
||||
### Configuration (buster.yml)
|
||||
|
||||
```yaml
|
||||
# buster.yml
|
||||
data_source_name: "my_warehouse" # Your default data source
|
||||
schema: "analytics" # Default schema for models
|
||||
database: "prod" # Optional database name
|
||||
exclude_files: # Optional list of files to exclude from generation
|
||||
- "temp_*.sql" # Exclude all SQL files starting with temp_
|
||||
- "test/**/*.sql" # Exclude all SQL files in test directories
|
||||
- "customers.sql" # Exclude a specific file
|
||||
exclude_tags: # Optional list of tags to exclude from deployment
|
||||
- "staging" # Exclude models with the 'staging' tag
|
||||
- "test" # Exclude models with the 'test' tag
|
||||
```
|
||||
|
||||
The configuration supports the following fields:
|
||||
- `data_source_name`: (Required) Default data source for your models
|
||||
- `schema`: (Required) Default schema for your models
|
||||
- `database`: (Optional) Default database name
|
||||
- `exclude_files`: (Optional) List of glob patterns for files to exclude from generation
|
||||
- Supports standard glob patterns (*, **, ?, etc.)
|
||||
- Matches against relative paths from source directory
|
||||
- Common use cases:
|
||||
- Excluding temporary files: `temp_*.sql`
|
||||
- Excluding test files: `test/**/*.sql`
|
||||
- Excluding specific files: `customers.sql`
|
||||
- Excluding files in directories: `archive/**/*.sql`
|
||||
- `exclude_tags`: (Optional) List of tags to exclude from deployment
|
||||
- Looks for tags in SQL files in dbt format: `{{ config(tags=['tag1', 'tag2']) }}`
|
||||
- Useful for excluding staging models, test models, etc.
|
||||
- Case-insensitive matching
|
||||
|
||||
### Model Definition Example
|
||||
|
||||
```yaml
|
||||
# models/customers.yml
|
||||
version: 1
|
||||
models:
|
||||
- name: customers
|
||||
description: "Core customer data model"
|
||||
data_source_name: "my_warehouse" # Overrides buster.yml
|
||||
schema: "analytics" # Overrides buster.yml
|
||||
database: "prod" # Overrides buster.yml
|
||||
|
||||
entities:
|
||||
- name: customer_id
|
||||
expr: "id"
|
||||
type: "primary"
|
||||
description: "Primary customer identifier"
|
||||
- name: order
|
||||
expr: "order_id"
|
||||
type: "foreign"
|
||||
description: "Reference to order model"
|
||||
# Optional: reference to another model in a different project
|
||||
project_path: "path/to/other/project"
|
||||
# Optional: specify a different name for the referenced model
|
||||
ref: "orders"
|
||||
|
||||
dimensions:
|
||||
- name: email
|
||||
expr: "email"
|
||||
type: "string"
|
||||
description: "Customer email address"
|
||||
searchable: true # Optional: make this dimension searchable
|
||||
|
||||
measures:
|
||||
- name: total_customers
|
||||
expr: "customer_id"
|
||||
agg: "count_distinct"
|
||||
description: "Total number of unique customers"
|
||||
type: "integer" # Optional: specify the data type
|
||||
```
|
||||
|
||||
## Cross-Project References
|
||||
|
||||
Buster CLI supports referencing models across different projects, enabling you to build complex data relationships:
|
||||
|
||||
```yaml
|
||||
entities:
|
||||
- name: user_model
|
||||
expr: "user_id"
|
||||
type: "foreign"
|
||||
description: "Reference to user model in another project"
|
||||
project_path: "path/to/user/project"
|
||||
ref: "users" # Optional: specify a different name for the referenced model
|
||||
```
|
||||
|
||||
When using cross-project references, the CLI will:
|
||||
1. Validate that the referenced project exists
|
||||
2. Check for a valid buster.yml in the referenced project
|
||||
3. Verify that the data sources match between projects
|
||||
4. Confirm that the referenced model exists in the target project
|
||||
|
||||
This enables you to organize your models into logical projects while maintaining relationships between them.
|
||||
|
||||
## Tag-Based Exclusion
|
||||
|
||||
You can exclude models from deployment based on tags in your SQL files. This is useful for excluding staging models, test models, or any other models you don't want to deploy.
|
||||
|
||||
In your SQL files, add tags using the dbt format:
|
||||
|
||||
```sql
|
||||
{{ config(
|
||||
tags=['staging', 'test']
|
||||
) }}
|
||||
|
||||
SELECT * FROM source_table
|
||||
```
|
||||
|
||||
Then in your buster.yml, specify which tags to exclude:
|
||||
|
||||
```yaml
|
||||
exclude_tags:
|
||||
- "staging"
|
||||
- "test"
|
||||
```
|
||||
|
||||
During deployment, any model with matching tags will be automatically excluded.
|
||||
|
||||
## File and Tag Exclusions
|
||||
|
||||
Buster CLI provides a unified way to exclude files from processing across all commands. You can specify exclusions in your `buster.yml` file:
|
||||
|
||||
```yaml
|
||||
data_source_name: "my_data_source"
|
||||
schema: "my_schema"
|
||||
database: "my_database"
|
||||
exclude_files:
|
||||
- "**/*_temp.sql"
|
||||
- "staging/**/*.sql"
|
||||
- "tests/**/*.yml"
|
||||
exclude_tags:
|
||||
- "test"
|
||||
- "deprecated"
|
||||
- "wip"
|
||||
```
|
||||
|
||||
### Exclude Files
|
||||
|
||||
The `exclude_files` section allows you to specify glob patterns for files that should be excluded from processing. This works for any command that processes files.
|
||||
|
||||
Common patterns:
|
||||
- `"**/*_temp.sql"` - Exclude any SQL file ending with _temp.sql in any directory
|
||||
- `"staging/**/*.sql"` - Exclude all SQL files in the staging directory and its subdirectories
|
||||
- `"test_*.yml"` - Exclude all YAML files starting with test_
|
||||
|
||||
### Exclude Tags
|
||||
|
||||
The `exclude_tags` section allows you to exclude files based on tags specified in the file content. This is useful for excluding files that are marked as test, deprecated, etc.
|
||||
|
||||
Tags are specified in the SQL files using the format: `-- tags = ['tag1', 'tag2']`
|
||||
|
||||
When a file contains any of the excluded tags, it will be skipped by all commands.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Organization**
|
||||
- Keep YAML files in `models/`
|
||||
- Keep SQL files in `sql/`
|
||||
- Use `buster.yml` for shared settings
|
||||
- Group related models into subdirectories
|
||||
|
||||
2. **Model Generation**
|
||||
- Start with clean SQL files
|
||||
- Generate models first before customizing
|
||||
- Review generated models before deployment
|
||||
- Use tags to organize and filter models
|
||||
|
||||
3. **Deployment**
|
||||
- Use `--dry-run` to validate changes
|
||||
- Deploy frequently to catch issues early
|
||||
- Keep model and SQL files in sync
|
||||
- Use cross-project references for complex relationships
|
||||
|
||||
4. **Validation**
|
||||
- Ensure all models have descriptions
|
||||
- Validate cross-project references before deployment
|
||||
- Check for missing dependencies
|
||||
- Review validation errors carefully
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Common issues and solutions:
|
||||
|
||||
1. **Authentication Issues**
|
||||
- Verify your API key is correct
|
||||
- Check if the host is properly specified (if using non-production)
|
||||
- Ensure network connectivity to Buster
|
||||
|
||||
2. **Generation Issues**
|
||||
- Verify SQL files are in the correct location
|
||||
- Check file permissions
|
||||
- Ensure SQL syntax is valid
|
||||
- Check for excluded files or tags
|
||||
|
||||
3. **Deployment Issues**
|
||||
- Validate YAML syntax
|
||||
- Check for missing dependencies
|
||||
- Verify data source connectivity
|
||||
- Look for cross-project reference errors
|
||||
- Check for tag-based exclusions
|
||||
|
||||
4. **Cross-Project Reference Issues**
|
||||
- Ensure the referenced project exists
|
||||
- Verify the referenced project has a valid buster.yml
|
||||
- Check that data sources match between projects
|
||||
- Confirm the referenced model exists in the target project
|
||||
|
||||
## License
|
||||
|
||||
MIT License - see [LICENSE](LICENSE) for details.
|
|
@ -1,19 +1,46 @@
|
|||
# Generated by Cargo
|
||||
# will have compiled files and executables
|
||||
debug/
|
||||
target/
|
||||
# TypeScript build artifacts
|
||||
dist/
|
||||
build/
|
||||
*.tsbuildinfo
|
||||
|
||||
# Remove Cargo.lock from gitignore if creating an executable, leave it for libraries
|
||||
# More information here https://doc.rust-lang.org/cargo/guide/cargo-toml-vs-cargo-lock.html
|
||||
Cargo.lock
|
||||
# Logs
|
||||
*.log
|
||||
npm-debug.log*
|
||||
yarn-debug.log*
|
||||
yarn-error.log*
|
||||
lerna-debug.log*
|
||||
|
||||
# These are backup files generated by rustfmt
|
||||
**/*.rs.bk
|
||||
# Coverage
|
||||
coverage/
|
||||
*.lcov
|
||||
.nyc_output
|
||||
|
||||
# MSVC Windows builds of rustc generate these, which store debugging information
|
||||
*.pdb
|
||||
# Node modules
|
||||
node_modules/
|
||||
|
||||
# Temporary files
|
||||
*.tmp
|
||||
*.temp
|
||||
.DS_Store
|
||||
|
||||
# Environment files
|
||||
.env.local
|
||||
.env.*.local
|
||||
|
||||
# Test artifacts
|
||||
junit.xml
|
||||
test-results/
|
||||
|
||||
# IDE
|
||||
.idea/
|
||||
.vscode/
|
||||
*.swp
|
||||
*.swo
|
||||
|
||||
/prds/completed
|
||||
|
||||
# CLI specific
|
||||
buster-cli
|
||||
*.exe
|
||||
*.dmg
|
||||
*.pkg
|
||||
*.deb
|
||||
*.rpm
|
|
@ -1,370 +1,100 @@
|
|||
# Buster CLI
|
||||
# Buster CLI (TypeScript)
|
||||
|
||||
A powerful command-line interface for managing semantic models in Buster. Deploy and manage your data models with ease, whether they're standalone or part of a dbt project.
|
||||
A modern TypeScript-based CLI built with Bun, Commander.js, and React Ink for interactive terminal UIs.
|
||||
|
||||
## Installation
|
||||
## Quick Start - Local Testing
|
||||
|
||||
Choose the installation command for your operating system:
|
||||
|
||||
### macOS (x86_64)
|
||||
### 1. Install Dependencies
|
||||
```bash
|
||||
mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-darwin-x86_64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster
|
||||
cd apps/cli
|
||||
bun install
|
||||
```
|
||||
|
||||
### macOS (ARM/Apple Silicon)
|
||||
### 2. Run in Development Mode (with hot reload)
|
||||
```bash
|
||||
mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-darwin-arm64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster
|
||||
# Watch mode - automatically restarts on changes
|
||||
bun run dev hello
|
||||
bun run dev hello Claude --uppercase
|
||||
bun run dev interactive
|
||||
```
|
||||
|
||||
### Linux (x86_64)
|
||||
### 3. Build and Test the CLI
|
||||
```bash
|
||||
mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-linux-x86_64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster
|
||||
# Build the CLI
|
||||
bun run build
|
||||
|
||||
# Test the built version
|
||||
bun run start hello
|
||||
bun run start interactive
|
||||
```
|
||||
|
||||
> **Note**: After installation, make sure `~/.local/bin` is in your PATH. Add this to your shell's config file (`.bashrc`, `.zshrc`, etc.):
|
||||
> ```bash
|
||||
> export PATH="$HOME/.local/bin:$PATH"
|
||||
> ```
|
||||
|
||||
### Windows (x86_64)
|
||||
1. Download the Windows binary:
|
||||
```powershell
|
||||
Invoke-WebRequest -Uri https://github.com/buster-so/buster/releases/latest/download/buster-cli-windows-x86_64.zip -OutFile buster.zip
|
||||
```
|
||||
|
||||
2. Extract and install:
|
||||
```powershell
|
||||
Expand-Archive -Path buster.zip -DestinationPath $env:USERPROFILE\buster
|
||||
Move-Item -Path $env:USERPROFILE\buster\buster-cli.exe -Destination $env:LOCALAPPDATA\Microsoft\WindowsApps\buster.exe
|
||||
```
|
||||
|
||||
## Quick Start Guide
|
||||
|
||||
### 1. Authentication
|
||||
|
||||
First, authenticate with Buster using your API key:
|
||||
|
||||
### 4. Create Standalone Executable
|
||||
```bash
|
||||
buster auth
|
||||
# Build a standalone binary (no Bun required on target machine)
|
||||
bun run build:standalone
|
||||
|
||||
# Test the standalone executable
|
||||
./dist/buster hello
|
||||
./dist/buster interactive
|
||||
```
|
||||
|
||||
This will prompt you for:
|
||||
- API Key (required) - Get this from the Buster Platform
|
||||
- Host (optional) - Defaults to production if not specified
|
||||
## Available Commands
|
||||
|
||||
You can also configure authentication using environment variables:
|
||||
- `hello [name]` - Simple greeting command
|
||||
- Options: `-u, --uppercase` - Output in uppercase
|
||||
- Example: `bun run dev hello Claude --uppercase`
|
||||
|
||||
- `interactive` - Interactive menu demo using React Ink
|
||||
- Use arrow keys to navigate
|
||||
- Press Enter to select
|
||||
- Press Q or Escape to quit
|
||||
|
||||
## Development Workflow
|
||||
|
||||
### Hot Reload Development
|
||||
```bash
|
||||
# Set API key via environment variable
|
||||
export BUSTER_API_KEY=your_api_key_here
|
||||
|
||||
# Optional: Set custom host. For self-hosted instances.
|
||||
export BUSTER_HOST=your_custom_host
|
||||
# This watches for changes and auto-restarts
|
||||
bun run --watch src/index.tsx hello
|
||||
```
|
||||
|
||||
The CLI will check for these environment variables in the following order:
|
||||
1. Command line arguments
|
||||
2. Environment variables
|
||||
3. Interactive prompt
|
||||
|
||||
This is particularly useful for:
|
||||
- CI/CD environments
|
||||
- Automated scripts
|
||||
- Development workflows where you don't want to enter credentials repeatedly
|
||||
|
||||
### 2. Generate Models
|
||||
|
||||
Generate Buster YAML models from your existing SQL files:
|
||||
|
||||
### Direct Execution (fastest for development)
|
||||
```bash
|
||||
buster generate
|
||||
# Run directly with Bun
|
||||
bun src/index.tsx hello
|
||||
bun src/index.tsx interactive
|
||||
```
|
||||
|
||||
Key flags for generation:
|
||||
- `--source-path`: Directory containing your SQL files (defaults to current directory)
|
||||
- `--destination-path`: Where to output the generated YAML files (defaults to current directory)
|
||||
- `--data-source-name`: Name of the data source to use in the models
|
||||
- `--schema`: Database schema name
|
||||
- `--database`: Database name
|
||||
- `--flat-structure`: Output YML files in a flat structure instead of maintaining directory hierarchy
|
||||
|
||||
The generate command will:
|
||||
- Scan the source directory for SQL files
|
||||
- Create corresponding YAML model files
|
||||
- Create a `buster.yml` configuration file if it doesn't exist
|
||||
- Preserve any existing model customizations
|
||||
|
||||
Example with all options:
|
||||
### Testing Different Build Outputs
|
||||
```bash
|
||||
buster generate \
|
||||
--source-path ./sql \
|
||||
--destination-path ./models \
|
||||
--data-source-name my_warehouse \
|
||||
--schema analytics \
|
||||
--database prod
|
||||
# Test as Node.js script
|
||||
bun run build
|
||||
node dist/index.js hello
|
||||
|
||||
# Test as standalone executable
|
||||
bun run build:standalone
|
||||
./dist/buster hello
|
||||
```
|
||||
|
||||
### 3. Deploy Models
|
||||
|
||||
Deploy your models to Buster:
|
||||
|
||||
```bash
|
||||
buster deploy
|
||||
```
|
||||
|
||||
Deploy options:
|
||||
- `--path`: Specific path to deploy (defaults to current directory)
|
||||
- `--dry-run`: Validate the deployment without actually deploying (defaults to false)
|
||||
- `--recursive`: Recursively search for model files in subdirectories (defaults to true)
|
||||
|
||||
Examples:
|
||||
```bash
|
||||
# Deploy all models in current directory
|
||||
buster deploy
|
||||
|
||||
# Deploy a specific model or directory
|
||||
buster deploy --path ./models/customers.yml
|
||||
|
||||
# Validate deployment without applying changes
|
||||
buster deploy --dry-run
|
||||
|
||||
# Deploy only models in the specified directory (not recursively)
|
||||
buster deploy --path ./models --recursive=false
|
||||
```
|
||||
|
||||
The deploy command will:
|
||||
1. Discover all YAML model files in the specified path
|
||||
2. Load and validate the models
|
||||
3. Check for excluded models based on tags
|
||||
4. Validate cross-project references
|
||||
5. Deploy the models to Buster
|
||||
6. Provide detailed validation feedback and error messages
|
||||
|
||||
## Project Structure
|
||||
|
||||
A typical Buster project structure:
|
||||
|
||||
```
|
||||
your-project/
|
||||
├── buster.yml # Global configuration
|
||||
├── models/ # Your semantic model definitions
|
||||
│ ├── customers.yml
|
||||
│ ├── orders.yml
|
||||
│ └── products.yml
|
||||
└── sql/ # SQL definitions
|
||||
├── customers.sql
|
||||
├── orders.sql
|
||||
└── products.sql
|
||||
apps/cli/
|
||||
├── src/
|
||||
│ ├── index.tsx # Entry point with Commander setup
|
||||
│ ├── commands/ # Command implementations
|
||||
│ │ ├── hello.tsx # Basic command with Ink
|
||||
│ │ └── interactive.tsx # Interactive UI demo
|
||||
│ ├── components/ # Reusable Ink components
|
||||
│ └── utils/ # Helper utilities
|
||||
├── dist/ # Build output
|
||||
├── package.json
|
||||
├── tsconfig.json
|
||||
└── README.md
|
||||
```
|
||||
|
||||
### Configuration (buster.yml)
|
||||
## Technologies
|
||||
|
||||
```yaml
|
||||
# buster.yml
|
||||
data_source_name: "my_warehouse" # Your default data source
|
||||
schema: "analytics" # Default schema for models
|
||||
database: "prod" # Optional database name
|
||||
exclude_files: # Optional list of files to exclude from generation
|
||||
- "temp_*.sql" # Exclude all SQL files starting with temp_
|
||||
- "test/**/*.sql" # Exclude all SQL files in test directories
|
||||
- "customers.sql" # Exclude a specific file
|
||||
exclude_tags: # Optional list of tags to exclude from deployment
|
||||
- "staging" # Exclude models with the 'staging' tag
|
||||
- "test" # Exclude models with the 'test' tag
|
||||
```
|
||||
|
||||
The configuration supports the following fields:
|
||||
- `data_source_name`: (Required) Default data source for your models
|
||||
- `schema`: (Required) Default schema for your models
|
||||
- `database`: (Optional) Default database name
|
||||
- `exclude_files`: (Optional) List of glob patterns for files to exclude from generation
|
||||
- Supports standard glob patterns (*, **, ?, etc.)
|
||||
- Matches against relative paths from source directory
|
||||
- Common use cases:
|
||||
- Excluding temporary files: `temp_*.sql`
|
||||
- Excluding test files: `test/**/*.sql`
|
||||
- Excluding specific files: `customers.sql`
|
||||
- Excluding files in directories: `archive/**/*.sql`
|
||||
- `exclude_tags`: (Optional) List of tags to exclude from deployment
|
||||
- Looks for tags in SQL files in dbt format: `{{ config(tags=['tag1', 'tag2']) }}`
|
||||
- Useful for excluding staging models, test models, etc.
|
||||
- Case-insensitive matching
|
||||
|
||||
### Model Definition Example
|
||||
|
||||
```yaml
|
||||
# models/customers.yml
|
||||
version: 1
|
||||
models:
|
||||
- name: customers
|
||||
description: "Core customer data model"
|
||||
data_source_name: "my_warehouse" # Overrides buster.yml
|
||||
schema: "analytics" # Overrides buster.yml
|
||||
database: "prod" # Overrides buster.yml
|
||||
|
||||
entities:
|
||||
- name: customer_id
|
||||
expr: "id"
|
||||
type: "primary"
|
||||
description: "Primary customer identifier"
|
||||
- name: order
|
||||
expr: "order_id"
|
||||
type: "foreign"
|
||||
description: "Reference to order model"
|
||||
# Optional: reference to another model in a different project
|
||||
project_path: "path/to/other/project"
|
||||
# Optional: specify a different name for the referenced model
|
||||
ref: "orders"
|
||||
|
||||
dimensions:
|
||||
- name: email
|
||||
expr: "email"
|
||||
type: "string"
|
||||
description: "Customer email address"
|
||||
searchable: true # Optional: make this dimension searchable
|
||||
|
||||
measures:
|
||||
- name: total_customers
|
||||
expr: "customer_id"
|
||||
agg: "count_distinct"
|
||||
description: "Total number of unique customers"
|
||||
type: "integer" # Optional: specify the data type
|
||||
```
|
||||
|
||||
## Cross-Project References
|
||||
|
||||
Buster CLI supports referencing models across different projects, enabling you to build complex data relationships:
|
||||
|
||||
```yaml
|
||||
entities:
|
||||
- name: user_model
|
||||
expr: "user_id"
|
||||
type: "foreign"
|
||||
description: "Reference to user model in another project"
|
||||
project_path: "path/to/user/project"
|
||||
ref: "users" # Optional: specify a different name for the referenced model
|
||||
```
|
||||
|
||||
When using cross-project references, the CLI will:
|
||||
1. Validate that the referenced project exists
|
||||
2. Check for a valid buster.yml in the referenced project
|
||||
3. Verify that the data sources match between projects
|
||||
4. Confirm that the referenced model exists in the target project
|
||||
|
||||
This enables you to organize your models into logical projects while maintaining relationships between them.
|
||||
|
||||
## Tag-Based Exclusion
|
||||
|
||||
You can exclude models from deployment based on tags in your SQL files. This is useful for excluding staging models, test models, or any other models you don't want to deploy.
|
||||
|
||||
In your SQL files, add tags using the dbt format:
|
||||
|
||||
```sql
|
||||
{{ config(
|
||||
tags=['staging', 'test']
|
||||
) }}
|
||||
|
||||
SELECT * FROM source_table
|
||||
```
|
||||
|
||||
Then in your buster.yml, specify which tags to exclude:
|
||||
|
||||
```yaml
|
||||
exclude_tags:
|
||||
- "staging"
|
||||
- "test"
|
||||
```
|
||||
|
||||
During deployment, any model with matching tags will be automatically excluded.
|
||||
|
||||
## File and Tag Exclusions
|
||||
|
||||
Buster CLI provides a unified way to exclude files from processing across all commands. You can specify exclusions in your `buster.yml` file:
|
||||
|
||||
```yaml
|
||||
data_source_name: "my_data_source"
|
||||
schema: "my_schema"
|
||||
database: "my_database"
|
||||
exclude_files:
|
||||
- "**/*_temp.sql"
|
||||
- "staging/**/*.sql"
|
||||
- "tests/**/*.yml"
|
||||
exclude_tags:
|
||||
- "test"
|
||||
- "deprecated"
|
||||
- "wip"
|
||||
```
|
||||
|
||||
### Exclude Files
|
||||
|
||||
The `exclude_files` section allows you to specify glob patterns for files that should be excluded from processing. This works for any command that processes files.
|
||||
|
||||
Common patterns:
|
||||
- `"**/*_temp.sql"` - Exclude any SQL file ending with _temp.sql in any directory
|
||||
- `"staging/**/*.sql"` - Exclude all SQL files in the staging directory and its subdirectories
|
||||
- `"test_*.yml"` - Exclude all YAML files starting with test_
|
||||
|
||||
### Exclude Tags
|
||||
|
||||
The `exclude_tags` section allows you to exclude files based on tags specified in the file content. This is useful for excluding files that are marked as test, deprecated, etc.
|
||||
|
||||
Tags are specified in the SQL files using the format: `-- tags = ['tag1', 'tag2']`
|
||||
|
||||
When a file contains any of the excluded tags, it will be skipped by all commands.
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Organization**
|
||||
- Keep YAML files in `models/`
|
||||
- Keep SQL files in `sql/`
|
||||
- Use `buster.yml` for shared settings
|
||||
- Group related models into subdirectories
|
||||
|
||||
2. **Model Generation**
|
||||
- Start with clean SQL files
|
||||
- Generate models first before customizing
|
||||
- Review generated models before deployment
|
||||
- Use tags to organize and filter models
|
||||
|
||||
3. **Deployment**
|
||||
- Use `--dry-run` to validate changes
|
||||
- Deploy frequently to catch issues early
|
||||
- Keep model and SQL files in sync
|
||||
- Use cross-project references for complex relationships
|
||||
|
||||
4. **Validation**
|
||||
- Ensure all models have descriptions
|
||||
- Validate cross-project references before deployment
|
||||
- Check for missing dependencies
|
||||
- Review validation errors carefully
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
Common issues and solutions:
|
||||
|
||||
1. **Authentication Issues**
|
||||
- Verify your API key is correct
|
||||
- Check if the host is properly specified (if using non-production)
|
||||
- Ensure network connectivity to Buster
|
||||
|
||||
2. **Generation Issues**
|
||||
- Verify SQL files are in the correct location
|
||||
- Check file permissions
|
||||
- Ensure SQL syntax is valid
|
||||
- Check for excluded files or tags
|
||||
|
||||
3. **Deployment Issues**
|
||||
- Validate YAML syntax
|
||||
- Check for missing dependencies
|
||||
- Verify data source connectivity
|
||||
- Look for cross-project reference errors
|
||||
- Check for tag-based exclusions
|
||||
|
||||
4. **Cross-Project Reference Issues**
|
||||
- Ensure the referenced project exists
|
||||
- Verify the referenced project has a valid buster.yml
|
||||
- Check that data sources match between projects
|
||||
- Confirm the referenced model exists in the target project
|
||||
|
||||
## License
|
||||
|
||||
MIT License - see [LICENSE](LICENSE) for details.
|
||||
- **Bun** - JavaScript runtime and bundler
|
||||
- **Commander.js** - Command-line interface framework
|
||||
- **React Ink** - React for interactive terminal UIs
|
||||
- **TypeScript** - Type safety
|
||||
- **Chalk** - Terminal styling
|
|
@ -0,0 +1,7 @@
|
|||
{
|
||||
"$schema": "https://biomejs.dev/schemas/1.9.4/schema.json",
|
||||
"extends": ["../../biome.json"],
|
||||
"files": {
|
||||
"include": ["src/**/*", "scripts/**/*"]
|
||||
}
|
||||
}
|
|
@ -0,0 +1,12 @@
|
|||
declare global {
|
||||
namespace NodeJS {
|
||||
interface ProcessEnv {
|
||||
NODE_ENV?: 'development' | 'production' | 'test';
|
||||
// Add CLI-specific environment variables here
|
||||
API_URL?: string;
|
||||
AUTH_TOKEN?: string;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export {};
|
|
@ -0,0 +1,49 @@
|
|||
{
|
||||
"name": "@buster-app/cli",
|
||||
"version": "0.1.0",
|
||||
"description": "Buster CLI - TypeScript version",
|
||||
"type": "module",
|
||||
"bin": {
|
||||
"buster": "./dist/index.js"
|
||||
},
|
||||
"scripts": {
|
||||
"prebuild": "[ \"$SKIP_ENV_CHECK\" = \"true\" ] || tsx scripts/validate-env.ts",
|
||||
"dev": "bun run --watch src/index.tsx",
|
||||
"build": "bun build src/index.tsx --outdir dist --target node",
|
||||
"build:dry-run": "tsc --noEmit",
|
||||
"build:standalone": "bun build src/index.tsx --compile --outfile dist/buster",
|
||||
"start": "bun run dist/index.js",
|
||||
"typecheck": "tsc --noEmit",
|
||||
"lint": "biome check . --write",
|
||||
"test": "vitest run",
|
||||
"test:unit": "vitest run --exclude '**/*.int.test.ts' --exclude '**/*.integration.test.ts' --passWithNoTests",
|
||||
"test:integration": "vitest run **/*.int.test.ts **/*.integration.test.ts",
|
||||
"test:watch": "vitest watch",
|
||||
"test:coverage": "vitest run --coverage",
|
||||
"clean": "rm -rf dist"
|
||||
},
|
||||
"dependencies": {
|
||||
"commander": "^12.1.0",
|
||||
"ink": "^5.0.1",
|
||||
"react": "^18.3.1",
|
||||
"zod": "^3.24.1",
|
||||
"chalk": "^5.3.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@buster/typescript-config": "workspace:*",
|
||||
"@buster/vitest-config": "workspace:*",
|
||||
"@buster/env-utils": "workspace:*",
|
||||
"@types/bun": "^1.1.14",
|
||||
"@types/node": "^22.10.5",
|
||||
"@types/react": "^18.3.17",
|
||||
"@biomejs/biome": "^1.9.4",
|
||||
"typescript": "^5.7.3",
|
||||
"tsx": "^4.19.2",
|
||||
"vitest": "^2.1.8",
|
||||
"@vitest/coverage-v8": "^2.1.8",
|
||||
"ink-testing-library": "^4.0.0"
|
||||
},
|
||||
"engines": {
|
||||
"bun": ">=1.0.0"
|
||||
}
|
||||
}
|
|
@ -0,0 +1,22 @@
|
|||
#!/usr/bin/env node
|
||||
|
||||
// This script uses the shared env-utils to validate environment variables
|
||||
import { loadRootEnv, validateEnv } from '@buster/env-utils';
|
||||
|
||||
// Load environment variables from root .env file
|
||||
loadRootEnv();
|
||||
|
||||
// Define required environment variables for the CLI app
|
||||
const requiredEnv = {
|
||||
// NODE_ENV is optional - will default to 'development' if not set
|
||||
// Add any CLI-specific required environment variables here:
|
||||
// API_URL: process.env.API_URL,
|
||||
// AUTH_TOKEN: process.env.AUTH_TOKEN,
|
||||
};
|
||||
|
||||
// Validate environment variables
|
||||
const { hasErrors } = validateEnv(requiredEnv);
|
||||
|
||||
if (hasErrors) {
|
||||
process.exit(1);
|
||||
}
|
|
@ -0,0 +1,26 @@
|
|||
import { describe, it, expect } from 'vitest';
|
||||
import React from 'react';
|
||||
import { render } from 'ink-testing-library';
|
||||
import { HelloCommand } from './hello';
|
||||
|
||||
describe('HelloCommand', () => {
|
||||
it('should render greeting with default name', () => {
|
||||
const { lastFrame } = render(<HelloCommand name="World" />);
|
||||
|
||||
expect(lastFrame()).toContain('Hello, World!');
|
||||
expect(lastFrame()).toContain('Buster CLI');
|
||||
});
|
||||
|
||||
it('should render greeting in uppercase when flag is set', () => {
|
||||
const { lastFrame } = render(<HelloCommand name="Claude" uppercase={true} />);
|
||||
|
||||
expect(lastFrame()).toContain('HELLO, CLAUDE!');
|
||||
});
|
||||
|
||||
it('should render greeting in normal case when uppercase flag is false', () => {
|
||||
const { lastFrame } = render(<HelloCommand name="Claude" uppercase={false} />);
|
||||
|
||||
expect(lastFrame()).toContain('Hello, Claude!');
|
||||
expect(lastFrame()).not.toContain('HELLO, CLAUDE!');
|
||||
});
|
||||
});
|
|
@ -0,0 +1,31 @@
|
|||
import React, { useEffect } from 'react';
|
||||
import { Text, Box } from 'ink';
|
||||
import chalk from 'chalk';
|
||||
|
||||
interface HelloCommandProps {
|
||||
name: string;
|
||||
uppercase?: boolean;
|
||||
}
|
||||
|
||||
export const HelloCommand: React.FC<HelloCommandProps> = ({ name, uppercase }) => {
|
||||
const greeting = `Hello, ${name}!`;
|
||||
const displayText = uppercase ? greeting.toUpperCase() : greeting;
|
||||
|
||||
useEffect(() => {
|
||||
// Exit after rendering
|
||||
setTimeout(() => {
|
||||
process.exit(0);
|
||||
}, 100);
|
||||
}, []);
|
||||
|
||||
return (
|
||||
<Box flexDirection="column">
|
||||
<Text color="green">
|
||||
{chalk.bold('🚀 Buster CLI')}
|
||||
</Text>
|
||||
<Text>
|
||||
{displayText}
|
||||
</Text>
|
||||
</Box>
|
||||
);
|
||||
};
|
|
@ -0,0 +1,55 @@
|
|||
import React, { useState } from 'react';
|
||||
import { Text, Box, useInput, useApp } from 'ink';
|
||||
import chalk from 'chalk';
|
||||
|
||||
export const InteractiveCommand: React.FC = () => {
|
||||
const [selectedOption, setSelectedOption] = useState(0);
|
||||
const { exit } = useApp();
|
||||
|
||||
const options = [
|
||||
'Create a new project',
|
||||
'Deploy to production',
|
||||
'Run tests',
|
||||
'Exit'
|
||||
];
|
||||
|
||||
useInput((input, key) => {
|
||||
if (key.upArrow) {
|
||||
setSelectedOption(prev => Math.max(0, prev - 1));
|
||||
} else if (key.downArrow) {
|
||||
setSelectedOption(prev => Math.min(options.length - 1, prev + 1));
|
||||
} else if (key.return) {
|
||||
if (selectedOption === options.length - 1) {
|
||||
exit();
|
||||
} else {
|
||||
// Handle selection
|
||||
console.info(`\nYou selected: ${options[selectedOption]}`);
|
||||
exit();
|
||||
}
|
||||
} else if (input === 'q' || key.escape) {
|
||||
exit();
|
||||
}
|
||||
});
|
||||
|
||||
return (
|
||||
<Box flexDirection="column">
|
||||
<Box marginBottom={1}>
|
||||
<Text color="cyan" bold>
|
||||
🚀 Buster CLI - Interactive Mode
|
||||
</Text>
|
||||
</Box>
|
||||
|
||||
<Text dimColor>Use arrow keys to navigate, Enter to select, Q to quit</Text>
|
||||
<Box marginTop={1} flexDirection="column">
|
||||
{options.map((option, index) => (
|
||||
<Box key={option}>
|
||||
<Text color={selectedOption === index ? 'green' : undefined}>
|
||||
{selectedOption === index ? chalk.bold('▶ ') : ' '}
|
||||
{option}
|
||||
</Text>
|
||||
</Box>
|
||||
))}
|
||||
</Box>
|
||||
</Box>
|
||||
);
|
||||
};
|
|
@ -0,0 +1,38 @@
|
|||
#!/usr/bin/env bun
|
||||
import React from 'react';
|
||||
import { render } from 'ink';
|
||||
import { program } from 'commander';
|
||||
import { HelloCommand } from './commands/hello.js';
|
||||
import { InteractiveCommand } from './commands/interactive.js';
|
||||
|
||||
// CLI metadata
|
||||
program
|
||||
.name('buster')
|
||||
.description('Buster CLI - TypeScript version')
|
||||
.version('0.1.0');
|
||||
|
||||
// Hello command - basic example
|
||||
program
|
||||
.command('hello')
|
||||
.description('Say hello')
|
||||
.argument('[name]', 'Name to greet', 'World')
|
||||
.option('-u, --uppercase', 'Output in uppercase')
|
||||
.action(async (name: string, options: { uppercase?: boolean }) => {
|
||||
render(<HelloCommand name={name} uppercase={options.uppercase} />);
|
||||
});
|
||||
|
||||
// Interactive command - demonstrates Ink's capabilities
|
||||
program
|
||||
.command('interactive')
|
||||
.description('Run an interactive demo')
|
||||
.action(async () => {
|
||||
render(<InteractiveCommand />);
|
||||
});
|
||||
|
||||
// Parse command line arguments
|
||||
program.parse(process.argv);
|
||||
|
||||
// Show help if no command provided
|
||||
if (!process.argv.slice(2).length) {
|
||||
program.outputHelp();
|
||||
}
|
|
@ -0,0 +1,18 @@
|
|||
{
|
||||
"extends": "@buster/typescript-config/base.json",
|
||||
"compilerOptions": {
|
||||
"tsBuildInfoFile": "dist/.cache/tsbuildinfo.json",
|
||||
"outDir": "./dist",
|
||||
"rootDir": "./src",
|
||||
"jsx": "react",
|
||||
"jsxFactory": "React.createElement",
|
||||
"jsxFragmentFactory": "React.Fragment",
|
||||
"module": "ESNext",
|
||||
"target": "ES2022",
|
||||
"moduleResolution": "bundler",
|
||||
"allowImportingTsExtensions": true,
|
||||
"types": ["bun", "node"]
|
||||
},
|
||||
"include": ["src/**/*", "env.d.ts"],
|
||||
"exclude": ["node_modules", "dist", "tests", "**/*.test.ts", "**/*.spec.ts"]
|
||||
}
|
|
@ -0,0 +1,10 @@
|
|||
{
|
||||
"extends": ["//"],
|
||||
"tasks": {
|
||||
"build:standalone": {
|
||||
"dependsOn": ["build"],
|
||||
"outputs": ["dist/buster"],
|
||||
"cache": false
|
||||
}
|
||||
}
|
||||
}
|
|
@ -0,0 +1,3 @@
|
|||
import { baseConfig } from '@buster/vitest-config';
|
||||
|
||||
export default baseConfig;
|
|
@ -8,6 +8,8 @@
|
|||
"check": "biome check ${1:-.}",
|
||||
"check:fix": "biome check --write ${1:-.}",
|
||||
"ci:check": "pnpm run check && pnpm run typecheck",
|
||||
"cli": "cd apps/cli && bun src/index.tsx",
|
||||
"cli:dev": "cd apps/cli && bun run --watch src/index.tsx",
|
||||
"db:check": "pnpm --filter @buster/database run db:check",
|
||||
"db:generate": "pnpm --filter @buster/database run db:generate",
|
||||
"db:generate:custom": "pnpm --filter @buster/database run db:generate:custom",
|
||||
|
|
Loading…
Reference in New Issue