12 KiB

Raw Permalink Blame History

Buster CLI

A powerful command-line interface for managing semantic models in Buster. Deploy and manage your data models with ease, whether they're standalone or part of a dbt project.

Installation

Choose the installation command for your operating system:

macOS (x86_64)

mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-darwin-x86_64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster

macOS (ARM/Apple Silicon)

mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-darwin-arm64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster

Linux (x86_64)

mkdir -p ~/.local/bin && curl -L https://github.com/buster-so/buster/releases/latest/download/buster-cli-linux-x86_64.tar.gz | tar xz && mv buster-cli ~/.local/bin/buster && chmod +x ~/.local/bin/buster

Note: After installation, make sure ~/.local/bin is in your PATH. Add this to your shell's config file (.bashrc, .zshrc, etc.):
export PATH="$HOME/.local/bin:$PATH"

Windows (x86_64)

Download the Windows binary:

Invoke-WebRequest -Uri https://github.com/buster-so/buster/releases/latest/download/buster-cli-windows-x86_64.zip -OutFile buster.zip

Extract and install:

Expand-Archive -Path buster.zip -DestinationPath $env:USERPROFILE\buster
Move-Item -Path $env:USERPROFILE\buster\buster-cli.exe -Destination $env:LOCALAPPDATA\Microsoft\WindowsApps\buster.exe

Quick Start Guide

1. Authentication

First, authenticate with Buster using your API key:

buster auth

This will prompt you for:

API Key (required) - Get this from the Buster Platform
Host (optional) - Defaults to production if not specified

You can also configure authentication using environment variables:

# Set API key via environment variable
export BUSTER_API_KEY=your_api_key_here

# Optional: Set custom host. For self-hosted instances.
export BUSTER_HOST=your_custom_host

The CLI will check for these environment variables in the following order:

Command line arguments
Environment variables
Interactive prompt

This is particularly useful for:

CI/CD environments
Automated scripts
Development workflows where you don't want to enter credentials repeatedly

2. Generate Models

Generate Buster YAML models from your existing SQL files:

buster generate

Key flags for generation:

--source-path: Directory containing your SQL files (defaults to current directory)
--destination-path: Where to output the generated YAML files (defaults to current directory)
--data-source-name: Name of the data source to use in the models
--schema: Database schema name
--database: Database name
--flat-structure: Output YML files in a flat structure instead of maintaining directory hierarchy

The generate command will:

Scan the source directory for SQL files
Create corresponding YAML model files
Create a buster.yml configuration file if it doesn't exist
Preserve any existing model customizations

Example with all options:

buster generate \
  --source-path ./sql \
  --destination-path ./models \
  --data-source-name my_warehouse \
  --schema analytics \
  --database prod

3. Deploy Models

Deploy your models to Buster:

buster deploy

Deploy options:

--path: Specific path to deploy (defaults to current directory)
--dry-run: Validate the deployment without actually deploying (defaults to false)
--recursive: Recursively search for model files in subdirectories (defaults to true)

Examples:

# Deploy all models in current directory
buster deploy

# Deploy a specific model or directory
buster deploy --path ./models/customers.yml

# Validate deployment without applying changes
buster deploy --dry-run

# Deploy only models in the specified directory (not recursively)
buster deploy --path ./models --recursive=false

The deploy command will:

Discover all YAML model files in the specified path
Load and validate the models
Check for excluded models based on tags
Validate cross-project references
Deploy the models to Buster
Provide detailed validation feedback and error messages

Project Structure

A typical Buster project structure:

your-project/
├── buster.yml          # Global configuration
├── models/            # Your semantic model definitions
│   ├── customers.yml
│   ├── orders.yml
│   └── products.yml
└── sql/              # SQL definitions
    ├── customers.sql
    ├── orders.sql
    └── products.sql

Configuration (buster.yml)

# buster.yml
data_source_name: "my_warehouse"  # Your default data source
schema: "analytics"               # Default schema for models
database: "prod"                  # Optional database name
exclude_files:                    # Optional list of files to exclude from generation
  - "temp_*.sql"                 # Exclude all SQL files starting with temp_
  - "test/**/*.sql"             # Exclude all SQL files in test directories
  - "customers.sql"             # Exclude a specific file
exclude_tags:                     # Optional list of tags to exclude from deployment
  - "staging"                    # Exclude models with the 'staging' tag
  - "test"                       # Exclude models with the 'test' tag

The configuration supports the following fields:

data_source_name: (Required) Default data source for your models
schema: (Required) Default schema for your models
database: (Optional) Default database name
exclude_files: (Optional) List of glob patterns for files to exclude from generation
- Supports standard glob patterns (*, **, ?, etc.)
- Matches against relative paths from source directory
- Common use cases:
  - Excluding temporary files: temp_*.sql
  - Excluding test files: test/**/*.sql
  - Excluding specific files: customers.sql
  - Excluding files in directories: archive/**/*.sql
exclude_tags: (Optional) List of tags to exclude from deployment
- Looks for tags in SQL files in dbt format: {{ config(tags=['tag1', 'tag2']) }}
- Useful for excluding staging models, test models, etc.
- Case-insensitive matching

Model Definition Example

# models/customers.yml
version: 1
models:
  - name: customers
    description: "Core customer data model"
    data_source_name: "my_warehouse"  # Overrides buster.yml
    schema: "analytics"               # Overrides buster.yml
    database: "prod"                  # Overrides buster.yml
    
    entities:
      - name: customer_id
        expr: "id"
        type: "primary"
        description: "Primary customer identifier"
      - name: order
        expr: "order_id"
        type: "foreign"
        description: "Reference to order model"
        # Optional: reference to another model in a different project
        project_path: "path/to/other/project"
        # Optional: specify a different name for the referenced model
        ref: "orders"
    
    dimensions:
      - name: email
        expr: "email"
        type: "string"
        description: "Customer email address"
        searchable: true  # Optional: make this dimension searchable
    
    measures:
      - name: total_customers
        expr: "customer_id"
        agg: "count_distinct"
        description: "Total number of unique customers"
        type: "integer"  # Optional: specify the data type

Cross-Project References

Buster CLI supports referencing models across different projects, enabling you to build complex data relationships:

entities:
  - name: user_model
    expr: "user_id"
    type: "foreign"
    description: "Reference to user model in another project"
    project_path: "path/to/user/project"
    ref: "users"  # Optional: specify a different name for the referenced model

When using cross-project references, the CLI will:

Validate that the referenced project exists
Check for a valid buster.yml in the referenced project
Verify that the data sources match between projects
Confirm that the referenced model exists in the target project

This enables you to organize your models into logical projects while maintaining relationships between them.

Tag-Based Exclusion

You can exclude models from deployment based on tags in your SQL files. This is useful for excluding staging models, test models, or any other models you don't want to deploy.

In your SQL files, add tags using the dbt format:

{{ config(
    tags=['staging', 'test']
) }}

SELECT * FROM source_table

Then in your buster.yml, specify which tags to exclude:

exclude_tags:
  - "staging"
  - "test"

During deployment, any model with matching tags will be automatically excluded.

File and Tag Exclusions

Buster CLI provides a unified way to exclude files from processing across all commands. You can specify exclusions in your buster.yml file:

data_source_name: "my_data_source"
schema: "my_schema"
database: "my_database"
exclude_files:
  - "**/*_temp.sql"
  - "staging/**/*.sql"
  - "tests/**/*.yml"
exclude_tags:
  - "test"
  - "deprecated"
  - "wip"

Exclude Files

The exclude_files section allows you to specify glob patterns for files that should be excluded from processing. This works for any command that processes files.

Common patterns:

"**/*_temp.sql" - Exclude any SQL file ending with _temp.sql in any directory
"staging/**/*.sql" - Exclude all SQL files in the staging directory and its subdirectories
"test_*.yml" - Exclude all YAML files starting with test_

Exclude Tags

The exclude_tags section allows you to exclude files based on tags specified in the file content. This is useful for excluding files that are marked as test, deprecated, etc.

Tags are specified in the SQL files using the format: -- tags = ['tag1', 'tag2']

When a file contains any of the excluded tags, it will be skipped by all commands.

Best Practices

Organization
- Keep YAML files in models/
- Keep SQL files in sql/
- Use buster.yml for shared settings
- Group related models into subdirectories
Model Generation
- Start with clean SQL files
- Generate models first before customizing
- Review generated models before deployment
- Use tags to organize and filter models
Deployment
- Use --dry-run to validate changes
- Deploy frequently to catch issues early
- Keep model and SQL files in sync
- Use cross-project references for complex relationships
Validation
- Ensure all models have descriptions
- Validate cross-project references before deployment
- Check for missing dependencies
- Review validation errors carefully

Troubleshooting

Common issues and solutions:

Authentication Issues
- Verify your API key is correct
- Check if the host is properly specified (if using non-production)
- Ensure network connectivity to Buster
Generation Issues
- Verify SQL files are in the correct location
- Check file permissions
- Ensure SQL syntax is valid
- Check for excluded files or tags
Deployment Issues
- Validate YAML syntax
- Check for missing dependencies
- Verify data source connectivity
- Look for cross-project reference errors
- Check for tag-based exclusions
Cross-Project Reference Issues
- Ensure the referenced project exists
- Verify the referenced project has a valid buster.yml
- Check that data sources match between projects
- Confirm the referenced model exists in the target project

License

MIT License - see LICENSE for details.

12 KiB Raw Permalink Blame History