- Refactor stored values processing in dataset deployment to use background task
- Add `StoredValueColumn` struct to encapsulate column processing details
- Implement `process_stored_values_background` for parallel and resilient value storage
- Add logging for successful and failed stored value processing
- Update CLI to handle optional SQL definitions and improve file processing
* feat: enhance deploy command with skip_dbt option
- Updated the Deploy command to accept a `skip_dbt` boolean argument, allowing users to bypass the dbt run during deployment.
- Refactored the deploy function to conditionally execute the dbt command based on the `skip_dbt` flag, improving deployment flexibility.
* Refactor query engine and CLI commands for improved functionality and error handling
- Updated `get_bigquery_columns` and `get_snowflake_columns` functions to enhance column name handling and ensure proper error reporting.
- Modified `get_snowflake_client` to accept a database ID for better connection management.
- Enhanced the `deploy` command in the CLI to include additional parameters (`path`, `data_source_name`, `schema`, `env`) for more flexible deployments.
- Improved error handling and reporting in the `deploy` function, including detailed summaries of deployment errors and successful file processing.
- Updated `get_model_files` to accept a directory path and added checks for file existence, enhancing robustness.
- Adjusted model file structures to include schema information and refined the upload process to handle optional parameters more effectively.
These changes collectively improve the usability and reliability of the query engine and deployment process.
* Update dataset DDL generation to include optional YML file content
- Modified `generate_sql_agent` to append optional YML file content to dataset DDL
- Ensures more comprehensive dataset representation during SQL agent generation
- Handles cases where YML file might be present or absent gracefully
- Introduced a new `is_simple` flag in the `deploy_datasets` function to differentiate between full and simple dataset deployments.
- Updated the `deploy_datasets_handler` to accept the `is_simple` parameter, allowing for conditional processing of inserted datasets.
- Modified the `DeployDatasetsRequest` struct to include an optional `id` and `type_` field, enhancing the request's flexibility.
- Adjusted the handling of the `yml_file` field to be optional in the `DeployDatasetsRequest` struct.
- Updated the `process_batch` function to handle "USER-DEFINED" data types in addition to existing types.
These changes improve the dataset deployment process by allowing for more granular control and flexibility in handling different dataset types.
- Replaced the existing `post_datasets` endpoint with a new `deploy_datasets` endpoint to better reflect its purpose.
- Deleted the `post_datasets` module and its associated logic, streamlining the codebase.
- Updated the request and response structures to use `DeployDatasetsRequest` and related types, enhancing clarity and maintainability.
- Adjusted the BusterClient to utilize the new endpoint for deploying datasets, ensuring consistency across the API.
These changes improve the API's functionality by providing a clearer and more focused approach to dataset deployment, facilitating better data management.
- Introduced a new optional `yml_file` field in the Dataset model to store YAML file references.
- Updated the database schema to include the `yml_file` column in the datasets table.
- Modified various API request and response structures to accommodate the new `yml_file` field.
- Enhanced dataset handling functions to support the inclusion of `yml_file` in dataset operations.
These changes improve the dataset management capabilities by allowing the association of YAML files with datasets, facilitating better data organization and retrieval.
- Renamed `semantic_models` to `models` in YAML template and related structures for consistency.
- Changed `expr` field type in `PostDatasetsEntityRelationshipsRequest` from `Vec<String>` to `String` to simplify data handling.
- Updated `BusterModel` struct to reflect the new `models` naming and removed unused `ModelDefaults` struct.
- Adjusted tests to align with the updated model structure.
These changes enhance clarity and maintainability in the model representation and data handling within the buster-cli.
dal
dal