Buster Warehouse

A data warehouse built on Apache Iceberg and Starrocks

Sign up · Quickstart · Deployment

Buster Warehouse Overview

This project is a data warehouse built on Apache Iceberg and Starrocks. In working with our customers, we found that Snowflake, Bigquery, and other warehouse solutions were prohibitively expensive or slow in them being able to deploy AI-powered analytics at scale.

Additionaly, we found that having a close integration between the data warehouse and our AI-native BI tool allows for a better and more reliable data experience.

Key Features

• Built on Starrocks: We felt that Starrock was the best query engine by default for our use case. The main thing that pushed us towards it was that they perform predicate pushdown on iceberg tables, whereas Clickhouse and DuckDB do not. We were also impressed by the performance, caching system, and flexibility of Starrocks.
• Built on Apache Iceberg: Some of the top companies in the world use Apache Iceberg for storing and interacting with their data. We wanted a table format that not only brought tremendous benefits, but one that companies wouldn't outgrow.
• Bring Your Own Storage: We felt that customers should own their data and not be locked into a particular storage engine.

Quickstart

1. Dependencies:

   • Make sure that you have Docker Engine installed.
   • Install Python if you haven't already.
   • Install a MySQL client on your system.
   • An AWS account with S3 access.

2. Clone the repository:

git clone https://github.com/buster-so/warehouse.git

3. Run the warehouse:

docker compose up -d

4. Populate the .env file with AWS credentials provisioned for S3 access. Note: You can use any S3 compatible storage, you might just need to tweak some of the configs. Feel free to look at the Starrocks docs or PyIceberg docs for more information.

5. Connect to the warehouse with any MySQL client.

6. Create the external catalog:

CREATE EXTERNAL CATALOG 'public'
PROPERTIES
(
  "type"="iceberg",
  "iceberg.catalog.type"="rest",
  "iceberg.catalog.uri"="http://iceberg-rest:8181",
  "iceberg.catalog.warehouse"="<BUCKET_NAME>",
  "aws.s3.access_key"="<ACCESS_KEY>",
  "aws.s3.secret_key"="<SECRET_KEY>",
  "aws.s3.region" = "<REGION>",
  "aws.s3.enable_path_style_access"="true",
  "client.factory"="com.starrocks.connector.iceberg.IcebergAwsClientFactory"
);

7. Seed the data. If you want to populate a table with 75m records, you can run the notebook found here.

8. Set the catalog

SET CATALOG 'public';

9. Set the database

USE DATABASE 'public';

10. Run a query

SELECT COUNT(*) FROM public.nyc_taxi;

Optimizations

For data that you think will be accessed frequently, you can cache it on disk for faster access with:

CACHE SELECT * FROM public.nyc_taxi WHERE tpep_pickup_datetime > '2022-03-01';

Roadmap

Currently, we are in the process of open-sourcing the platform. This includes:

• Warehouse Product (This Repo) ✅
• BI platform (https://buster.so) ⏰

After that, we will release an official roadmap.

How We Plan to Make Money

Currently, we offer a few commercial products:

• Cloud-Hosted Version
  • Cluster
  • Serverless
• Managed Self-Hosted Version

Support and feedback

You can contact us through either:

• Github Discussions
• Email us at founders at buster dot com

License

This repository is MIT licensed, except for the ee folders. See LICENSE for more details.

Shoutouts

The documentation from the Starrocks, Iceberg, and PyIceberg team has been very helpful in building this project.