mirror of https://github.com/buster-so/buster.git
423 Commits
| Author | SHA1 | Message | Date |
|---|---|---|---|
dal
|
f081f3e16e
|
feat: enhance dataset validation and deployment error handling
- Add detailed validation error logging in CLI - Improve type compatibility checks in dataset validation - Modify deployment process to handle and report validation errors more comprehensively - Add Hash derive for Verification enum - Update API and CLI to support more informative validation results |
|
|
dal
|
3c82ac0774
|
feat: add dataset validation and improved deployment process | |
|
dal
|
66d9528619
|
refactor: deploy datasets much simpler now | |
|
dal
|
cace5d438a
|
Merge branch 'staging' into dal/cli-release | |
|
dal
|
2c7ef16956
|
fix the search value table | |
|
dal
|
8c853b7ac3
|
Merge pull request #98 from buster-so/dal/fix-dataset-delete-bug
bugfix(datasets): add delete dataset route |
|
|
dal
|
d9973a13dd
|
bugfix(datasets): add delete dataset route
Implement a new DELETE route for removing datasets by their ID |
|
|
dal
|
2be7383656
|
you learn something new every day... a schema in pg can't start with a number. | |
Nate Kelley
|
f437aea897
|
use colors in tooltip cache | |
|
Nate Kelley
|
255c1514ad
|
Update MetricTitle.tsx | |
|
dal
|
960c89ab84
|
fix: janky check for values | |
|
dal
|
87a6225f1d
|
remove the limit query bc mixing things up | |
|
dal
|
0fde90b848
|
refactor(snowflake_query): optimize Arrow data processing with explicit row collection (#92) | |
|
dal
|
05d613c9ad
|
chore: update .gitignore to exclude /prds directory | |
|
dal
|
05e75a5652
|
Merge branch 'staging' into dal/cli-release | |
|
dal
|
d0ff21e10d
|
Merge pull request #90 from buster-so/dal/stored_values_enum_push_to_description
feat(stored_values): enhance column value processing with enum detect… |
|
|
dal
|
d4825c0ffe
|
bugfix(snowflake_query): add data processing helpers for query results (#88)
- Introduce helper functions for processing string and JSON values - Implement case-insensitive string and JSON value transformations - Add robust timestamp parsing with error handling - Enhance Snowflake query result processing with consistent data normalization |
|
|
dal
|
59049b5604
|
refactor(stored_values): improve background processing and error handling for stored column values (#85)
- Refactor stored values processing in dataset deployment to use background task - Add `StoredValueColumn` struct to encapsulate column processing details - Implement `process_stored_values_background` for parallel and resilient value storage - Add logging for successful and failed stored value processing - Update CLI to handle optional SQL definitions and improve file processing |
|
|
dal
|
42c6080598
|
refactor(stored_values): improve background processing and error handling for stored column values
- Refactor stored values processing in dataset deployment to use background task - Add `StoredValueColumn` struct to encapsulate column processing details - Implement `process_stored_values_background` for parallel and resilient value storage - Add logging for successful and failed stored value processing - Update CLI to handle optional SQL definitions and improve file processing |
|
|
dal
|
daf4ec794f
|
Upgrade: Updated to o3-mini models
upgrade(ai): update OpenAI model configurations and add support for O… |
|
|
dal
|
158f5ba0a9
|
Dal/stored values fix (#81)
* fix: add stored values support for dataset columns This commit introduces stored values functionality for dataset columns, including: - Adding a `stored_values` flag to column deployment requests - Implementing a mechanism to store column values during dataset deployment - Updating data analyst and SQL generation agents to leverage stored values - Creating a new utility module for stored values search and management * refactor(stored_values): improve stored values implementation and schema management This commit enhances the stored values functionality with several key improvements: - Update schema and table creation to use organization ID as schema name - Modify stored values storage to include column ID - Improve value extraction and embedding generation process - Remove unnecessary distance calculation in search results - Clean up unused values_engine module |
|
|
dal
|
f11bfa9941
|
fix(invite_users): simplify and streamline user invitation process (#78) | |
|
dal
|
2a183ca711
|
fix(dashboard): improve dashboard access and permission handling (#76) | |
|
Nate Kelley
|
5b231e1efd
|
make dashboard header sizing more consistent | |
|
Nate Kelley
|
64971b7267
|
fix(posthog): update posthog config | |
|
Nate Kelley
|
4bade2114a
|
attempt to fix cursor bug | |
|
Nate Kelley
|
284dd43369
|
remove warning for show undefeine | |
|
Nate Kelley
|
01cbc70cd6
|
replace missing data with logic update | |
|
Nate Kelley
|
75e36b8e93
|
replace missing data with | |
|
Nate Kelley
|
cce49d7d95
|
Update createServerInstance.ts | |
|
dal
|
9ce7065789
|
Merge branch 'main' into staging | |
|
dal
|
27016df995
|
chore: cleanup limit insertion | |
github-actions[bot]
|
af38e68f19 | chore(release): update version to 0.0.13 | |
|
dal
|
81906d897a
|
chore: point snowflake warehouse id appropriately
* chore: add release-please configuration * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * update package versions * Make users page (#39) * create users pages * abstract more components to correct folders * carve out expection if we are hiding the select all * user query * Add ability to change default access * Update package-lock.json * Update package-lock.json * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Big nate/bus 924 make additional permissions pages (#50) * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * finalize permission popup Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * more elegant infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * add additional bulk popup menus Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * make pages unique Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * change how padding is applied to list Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * start user dataset lineage Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Refactor SQL query in list_assets_handler to use a Common Table Expression (CTE) for improved readability and maintainability. The CTE, `distinct_assets`, simplifies the selection of distinct asset records before applying the final ordering and limiting. * add cursor rules for web directory + jest (#52) * on change update for segments * PRevent clicking passthrough on users click * feat: Add assets module and nest routes in user router - Introduced a new `assets` module to handle asset-related routes. - Updated the user router to nest the `assets` routes under the user ID path, enhancing the organization of API endpoints. - This change improves the structure and maintainability of the user-related routes in the API. * feat: Enhance user attribute listing with authorization checks - Updated the `list_attributes_handler` to include authorization checks for user roles and organization IDs. - Implemented error handling for unauthorized access to user attributes. - Refactored the SQL query to retrieve user attributes based on the authenticated user's organization, improving security and data integrity. - This change ensures that only authorized users can list attributes, enhancing the overall security of the API. * pass through dataset overview * feat: Update dataset group listing to include permissions - Enhanced the `list_dataset_groups` function to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset group. - Modified the `DatasetGroupInfo` struct to include `permission_id` and `assigned` fields, reflecting the new data structure. - Refactored the SQL query to group by necessary fields and ensure accurate permission data is returned, improving the functionality and security of dataset group listings. * feat: Add DatasetGroupPermission model and schema - Introduced a new `DatasetGroupPermission` struct in `models.rs` to represent permissions associated with dataset groups. - Updated the database schema in `schema.rs` to include the `dataset_groups_permissions` table, defining its structure and relationships. - Modified the `is_user_workspace_admin_or_data_admin` function in `checks.rs` to correctly reference the user's organization role, enhancing role validation logic. * add component for users inputs * feat: Update dataset group listing to include dataset group permissions - Modified the `list_dataset_groups` function to accept an additional `id` parameter for filtering dataset groups based on user permissions. - Updated the SQL query to join with the `dataset_groups_permissions` table, allowing retrieval of permission counts for each dataset group. - Refactored the `DatasetGroupInfo` struct to replace `permission_id` with `permission_count`, enhancing clarity and accuracy in the data representation. - Ensured that the query groups by the new permission structure, improving the functionality and security of dataset group listings. * feat: Refactor dataset listing to include user-specific permissions - Updated the `list_datasets` function to accept an additional `id` parameter for filtering datasets based on user permissions. - Enhanced the SQL query to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset. - Refactored the `DatasetInfo` struct to include an `assigned` field, improving clarity in the dataset representation. - Improved error handling for dataset retrieval, ensuring robust logging and response management. * fix: Correct user role attribute and enhance read-only logic in list_attributes_handler - Updated the user role attribute key from "role" to "organization_role" for accurate role retrieval. - Introduced a read-only flag for specific user attributes, improving data integrity by clearly indicating which attributes should not be modified. - Enhanced error handling for user role retrieval, ensuring robust responses for missing or incorrect attributes. * normalize header for list * new line * popup conatiner for users * feat: Enhance user authorization checks and refactor related functions - Added user authorization checks in `list_attributes`, `list_dataset_groups`, `list_datasets`, `list_permission_groups`, and `list_teams` functions to ensure only users with appropriate roles can access these resources. - Refactored the `list_teams_handler` to accept `user_id` as a parameter, improving clarity and consistency across user-related functions. - Updated SQL queries to utilize the new authorization checks, enhancing security and data integrity. - Removed redundant column allowances in `list_teams` permissions, streamlining the codebase. * pass last child as index * feat: Enhance permission group handling and streamline SQL queries - Expanded the `allow_columns_to_appear_in_same_group_by_clause!` macro in `models.rs` to include additional columns for datasets and users, improving query flexibility. - Refactored the `list_permission_groups` function to include dataset count and assigned status, enhancing the information returned for each permission group. - Updated SQL queries in `list_permission_groups` to utilize left joins for better data retrieval and to ensure accurate permission checks. - Removed redundant column allowances in various files, streamlining the codebase and improving maintainability. * feat: Add PUT route for updating teams in user assets - Introduced a new module `put_teams` to handle updates for teams. - Added a PUT route for `/teams` in the user assets router, allowing for team modifications. - Enhanced the routing capabilities of the user assets API to support both GET and PUT requests for teams. * onchagne appsegmetned update * create permission user endpoints * move files to match new page structure * refactor: Clean up routing and improve PUT teams handler - Reformatted imports in `mod.rs` for better readability. - Commented out the PUT route for `/teams` in the user assets router, indicating a potential future change or deprecation. - Updated the `put_teams` handler to return a `NoContent` response upon successful execution, enhancing clarity in API responses. - Improved error handling in the `put_teams` function for better logging and response management. * refactor: Standardize user ID parameter naming across user-related routes - Updated all user-related route handlers to use `user_id` instead of `id` for better clarity and consistency. - Modified the routing definitions in `mod.rs` to reflect the new parameter naming convention. - Enhanced the `list_permission_groups` function to accept `user_id` as a parameter, improving clarity in the handler's signature. - Ensured all relevant functions now consistently handle the `user_id` parameter, streamlining the codebase and improving maintainability. * feat: Enhance team management with role-based assignments - Introduced a new `TeamInfoRole` enum to represent user roles within teams, replacing the previous boolean `assigned` field. - Updated the `list_teams` handler to return team roles instead of assignment status, improving clarity on user roles. - Refactored the `put_teams` handler to support role-based assignments, allowing for more granular control over team memberships. - Added new PUT routes for dataset groups and permission groups in the user assets router, enhancing API capabilities. - Improved SQL queries for team assignments to utilize role information, streamlining database interactions. * feat: Add organization_id to DatasetGroupPermission and update dataset group handler - Introduced a new `organization_id` field in the `DatasetGroupPermission` struct to associate permissions with specific organizations. - Updated the `put_dataset_groups_handler` to include `organization_id` when creating or updating dataset group permissions, enhancing the API's capability to manage permissions at the organizational level. - Improved SQL query formatting for better readability in the handler. * add list components for permission settings * feat: Introduce assets module and update routing for permission groups - Added a new `assets` module to organize related routes. - Updated the routing in `mod.rs` to nest the `assets` router under the `/:permission_group_id` path, enhancing the structure and clarity of the API. - Maintained existing routes for managing permission groups while improving modularity. * create permission group users * refactor: Update list_permission_groups_handler to use user_id and improve SQL queries - Changed the parameter in the SQL query from `user.id` to `user_id` for consistency with the updated user ID parameter naming convention. - Enhanced the SQL query to count distinct dataset permissions and utilize `bool_or` for identity checks, improving accuracy and performance. - Cleaned up the grouping in the SQL query by removing unnecessary fields, streamlining the data retrieval process. * add listing for dataset groups * feat: Add PUT routes for user and dataset management in assets module - Introduced new PUT routes for managing users and dataset groups in the assets module. - Updated the router to support PUT requests for `/users`, `/dataset_groups`, and `/datasets`, enhancing the API's functionality for resource updates. - Improved modularity by organizing related routes within the assets module. * add listing for dataset groups * add list for datasets * assigned permissions * Add attributes and teams * feat: Add DatasetToDatasetGroup model and update schema - Introduced a new `DatasetToDatasetGroup` struct to represent the relationship between datasets and dataset groups, including fields for timestamps and optional deletion. - Updated the database schema to include `updated_at` and `deleted_at` fields for the `datasets_to_dataset_groups` table, enhancing data tracking capabilities. - Refactored the routing in `mod.rs` to include a nested router for assets, improving the organization of dataset group routes. * invalidate query if user id is present * create a dataset modal added to user page * add team modal to teams * assigned popup * feat: Enhance user retrieval with dataset information - Added new structs `DatasetLineage` and `DatasetInfo` to represent dataset details and lineage. - Updated `UserResponse` to include a list of datasets associated with the user. - Refactored `get_user_information` function to concurrently fetch user info, direct datasets, permission group datasets, and organization datasets using `tokio::spawn` for improved performance. - Implemented logic to compile datasets based on direct access and permission group access, including lineage tracking for better data representation. - Enhanced error handling during database queries to ensure robust user information retrieval. * feat: Enhance dataset access retrieval in user and dataset overview - Updated `get_dataset_overview` to include dataset group access and permission group to dataset group access, improving the granularity of dataset permissions. - Introduced new queries to fetch dataset groups and their associated permissions, enhancing the dataset overview for users. - Refactored `get_user_information` to concurrently retrieve dataset groups and permission group datasets, optimizing performance with `tokio::spawn`. - Enhanced lineage tracking for datasets, allowing for better representation of user permissions across dataset groups and permission groups. - Improved error handling during database queries to ensure robust data retrieval. * lineage props passed * refactor: Streamline dataset access logic in get_user_information - Simplified access control logic for datasets based on user roles, consolidating conditions for WorkspaceAdmin, DataAdmin, Querier, Viewer, and RestrictedQuerier. - Enhanced dataset lineage tracking to provide clearer representation of user permissions across various dataset access types. - Removed redundant code related to dataset processing, improving readability and maintainability of the `get_user_information` function. - Ensured that datasets are correctly categorized based on direct access, permission group access, and organization datasets, optimizing the overall data retrieval process. * feat: Refactor dataset overview access lineage in get_dataset_overview - Introduced a default access lineage for users, ensuring consistent representation of user permissions. - Simplified the addition of user roles to the lineage, consolidating logic for WorkspaceAdmin, DataAdmin, Querier, and Viewer roles. - Enhanced lineage tracking for RestrictedQuerier role to include direct dataset access and permission group lineage, improving granularity of dataset permissions. - Removed redundant code related to dataset and permission group lineage, optimizing readability and maintainability of the `get_dataset_overview` function. * add datasetgroup handler * feat: Improve dataset access control and lineage tracking - Enhanced the `get_dataset_overview` function to refine access control for the `RestrictedQuerier` role, allowing for more granular permission checks based on various access paths. - Updated the `get_user_information` function to streamline dataset processing, ensuring that datasets are categorized correctly based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability of both functions. - Improved lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * clickable pills * fix build errors * add endpoints for dataset groups, and permission groups * refactor: Enhance dataset access control and lineage tracking - Streamlined the `get_dataset_overview` function to improve access control for the `RestrictedQuerier` role, ensuring more precise permission checks. - Updated the `get_user_information` function to optimize dataset processing, categorizing datasets based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability. - Enhanced lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * better handling for create a permission group * Refetch on team created for now * add additional pages * permission group modal update * added pages for permission groups * add user permission list * add datasets to permission groups page * Update dependencies and refactor Snowflake query handling - Downgraded the `base64` crate version in `Cargo.toml` from `0.22.1` to `0.21`. - Refactored the `snowflake_query` function in `snowflake_query.rs` to improve data type handling, including support for additional Arrow data types and enhanced null value checks. - Updated the `route_to_query` function in `query_router.rs` to use mutable `snowflake_client` for better state management during query execution. - Improved error handling for closing the Snowflake client session, ensuring proper logging of any issues encountered. * add datasets to permission groups page * Refactor Snowflake client connection to remove warehouse and database IDs - Updated the `get_snowflake_client` function to no longer require `warehouse_id` and `database_id`, simplifying the connection process. - This change enhances flexibility in client initialization and aligns with recent updates to Snowflake API handling. * add permission groups lists * add dataset group in permission group area * remove console logs * Big nate/bus 936 if a user is not an admin we should not see the permission (#57) * app sidebar settings should hide * rename api directories * list empty state * offset for row * refetch after user default access changed (#58) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * check if user has an org first * user organization can be null * get datasets only if they have an org * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * add hidden prop to row listing component * reroute to correct link for datasets * disable editing my own user * restrict certain routes to admins * small users page * restrict adding people to admins * update packages * Update shimmer text component * Update ShimmerText.tsx * update how no datasets is presented * fix permission check on post_dataset rest (#59) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * fix: Update SQL migration and seed data for user attributes - Modified the SQL migration to specify the schema for the `users` table, ensuring clarity in the update statement. - Adjusted the seed data for `users_to_organizations` to change the `organization_id` from 'public' to 'none', reflecting a more accurate state for user roles and organization associations. - Ensured consistency in the formatting of SQL insert statements for better readability. * fix: Prevent users from updating their own profiles - Added a check in the `update_user_handler` to prevent users from updating their own information, returning an error if they attempt to do so. - This change enhances security by ensuring that users cannot modify their own records, which could lead to unauthorized changes. * refactor: Simplify dashboard permission queries by removing team-based joins - Removed left joins with `teams_to_users` table in dashboard permission queries - Simplified permission checks to only filter by direct user ID - Updated queries in `get_user_dashboard_permission`, `get_bulk_user_dashboard_permission`, and `list_dashboards_handler` - Streamlined SQL query logic for more direct and efficient permission checks * add secure middleware checks * Update supabase server to secure and http only * dashboard permissions fix * set cookies * set cookies * Enhance email functionality by adding HTML escaping - Added `html-escape` crate to `Cargo.toml` for HTML escaping. - Updated email template processing to escape HTML in message and button text, preventing potential XSS vulnerabilities. - Modified test cases to include HTML content in email parameters, ensuring proper handling and escaping. This change improves security by sanitizing user input in email communications. * add csp policy middleware * Update cspPolicyMiddleware.ts * prevent email param injection * Dal/cli-updates-skip-dbt (#67) * feat: enhance deploy command with skip_dbt option - Updated the Deploy command to accept a `skip_dbt` boolean argument, allowing users to bypass the dbt run during deployment. - Refactored the deploy function to conditionally execute the dbt command based on the `skip_dbt` flag, improving deployment flexibility. * Refactor query engine and CLI commands for improved functionality and error handling - Updated `get_bigquery_columns` and `get_snowflake_columns` functions to enhance column name handling and ensure proper error reporting. - Modified `get_snowflake_client` to accept a database ID for better connection management. - Enhanced the `deploy` command in the CLI to include additional parameters (`path`, `data_source_name`, `schema`, `env`) for more flexible deployments. - Improved error handling and reporting in the `deploy` function, including detailed summaries of deployment errors and successful file processing. - Updated `get_model_files` to accept a directory path and added checks for file existence, enhancing robustness. - Adjusted model file structures to include schema information and refined the upload process to handle optional parameters more effectively. These changes collectively improve the usability and reliability of the query engine and deployment process. * Update dataset DDL generation to include optional YML file content - Modified `generate_sql_agent` to append optional YML file content to dataset DDL - Ensures more comprehensive dataset representation during SQL agent generation - Handles cases where YML file might be present or absent gracefully * chore: Enhance SQL agent generation with data source type context (#69) - Added support for retrieving and passing data source type during SQL agent generation - Updated SQL generation prompts to include data source type in system messages - Modified `generate_sql_agent` function to fetch and utilize data source type information - Improved SQL generation context by dynamically incorporating data source type details * Snowflake limit to prevent memory issues * chore: snowflake warehouse specification --------- Co-authored-by: Nate Kelley <nate@buster.so> Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> |
|
|
dal
|
d7087e8cd5
|
chore: snowflake warehouse specification | |
|
github-actions[bot]
|
15c3dfc8d1 | chore(release): update version to 0.0.12 | |
|
dal
|
5df2cb9526
|
chore: fix snowflake memory errors (#71)
* chore: add release-please configuration * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * update package versions * Make users page (#39) * create users pages * abstract more components to correct folders * carve out expection if we are hiding the select all * user query * Add ability to change default access * Update package-lock.json * Update package-lock.json * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Big nate/bus 924 make additional permissions pages (#50) * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * finalize permission popup Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * more elegant infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * add additional bulk popup menus Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * make pages unique Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * change how padding is applied to list Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * start user dataset lineage Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Refactor SQL query in list_assets_handler to use a Common Table Expression (CTE) for improved readability and maintainability. The CTE, `distinct_assets`, simplifies the selection of distinct asset records before applying the final ordering and limiting. * add cursor rules for web directory + jest (#52) * on change update for segments * PRevent clicking passthrough on users click * feat: Add assets module and nest routes in user router - Introduced a new `assets` module to handle asset-related routes. - Updated the user router to nest the `assets` routes under the user ID path, enhancing the organization of API endpoints. - This change improves the structure and maintainability of the user-related routes in the API. * feat: Enhance user attribute listing with authorization checks - Updated the `list_attributes_handler` to include authorization checks for user roles and organization IDs. - Implemented error handling for unauthorized access to user attributes. - Refactored the SQL query to retrieve user attributes based on the authenticated user's organization, improving security and data integrity. - This change ensures that only authorized users can list attributes, enhancing the overall security of the API. * pass through dataset overview * feat: Update dataset group listing to include permissions - Enhanced the `list_dataset_groups` function to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset group. - Modified the `DatasetGroupInfo` struct to include `permission_id` and `assigned` fields, reflecting the new data structure. - Refactored the SQL query to group by necessary fields and ensure accurate permission data is returned, improving the functionality and security of dataset group listings. * feat: Add DatasetGroupPermission model and schema - Introduced a new `DatasetGroupPermission` struct in `models.rs` to represent permissions associated with dataset groups. - Updated the database schema in `schema.rs` to include the `dataset_groups_permissions` table, defining its structure and relationships. - Modified the `is_user_workspace_admin_or_data_admin` function in `checks.rs` to correctly reference the user's organization role, enhancing role validation logic. * add component for users inputs * feat: Update dataset group listing to include dataset group permissions - Modified the `list_dataset_groups` function to accept an additional `id` parameter for filtering dataset groups based on user permissions. - Updated the SQL query to join with the `dataset_groups_permissions` table, allowing retrieval of permission counts for each dataset group. - Refactored the `DatasetGroupInfo` struct to replace `permission_id` with `permission_count`, enhancing clarity and accuracy in the data representation. - Ensured that the query groups by the new permission structure, improving the functionality and security of dataset group listings. * feat: Refactor dataset listing to include user-specific permissions - Updated the `list_datasets` function to accept an additional `id` parameter for filtering datasets based on user permissions. - Enhanced the SQL query to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset. - Refactored the `DatasetInfo` struct to include an `assigned` field, improving clarity in the dataset representation. - Improved error handling for dataset retrieval, ensuring robust logging and response management. * fix: Correct user role attribute and enhance read-only logic in list_attributes_handler - Updated the user role attribute key from "role" to "organization_role" for accurate role retrieval. - Introduced a read-only flag for specific user attributes, improving data integrity by clearly indicating which attributes should not be modified. - Enhanced error handling for user role retrieval, ensuring robust responses for missing or incorrect attributes. * normalize header for list * new line * popup conatiner for users * feat: Enhance user authorization checks and refactor related functions - Added user authorization checks in `list_attributes`, `list_dataset_groups`, `list_datasets`, `list_permission_groups`, and `list_teams` functions to ensure only users with appropriate roles can access these resources. - Refactored the `list_teams_handler` to accept `user_id` as a parameter, improving clarity and consistency across user-related functions. - Updated SQL queries to utilize the new authorization checks, enhancing security and data integrity. - Removed redundant column allowances in `list_teams` permissions, streamlining the codebase. * pass last child as index * feat: Enhance permission group handling and streamline SQL queries - Expanded the `allow_columns_to_appear_in_same_group_by_clause!` macro in `models.rs` to include additional columns for datasets and users, improving query flexibility. - Refactored the `list_permission_groups` function to include dataset count and assigned status, enhancing the information returned for each permission group. - Updated SQL queries in `list_permission_groups` to utilize left joins for better data retrieval and to ensure accurate permission checks. - Removed redundant column allowances in various files, streamlining the codebase and improving maintainability. * feat: Add PUT route for updating teams in user assets - Introduced a new module `put_teams` to handle updates for teams. - Added a PUT route for `/teams` in the user assets router, allowing for team modifications. - Enhanced the routing capabilities of the user assets API to support both GET and PUT requests for teams. * onchagne appsegmetned update * create permission user endpoints * move files to match new page structure * refactor: Clean up routing and improve PUT teams handler - Reformatted imports in `mod.rs` for better readability. - Commented out the PUT route for `/teams` in the user assets router, indicating a potential future change or deprecation. - Updated the `put_teams` handler to return a `NoContent` response upon successful execution, enhancing clarity in API responses. - Improved error handling in the `put_teams` function for better logging and response management. * refactor: Standardize user ID parameter naming across user-related routes - Updated all user-related route handlers to use `user_id` instead of `id` for better clarity and consistency. - Modified the routing definitions in `mod.rs` to reflect the new parameter naming convention. - Enhanced the `list_permission_groups` function to accept `user_id` as a parameter, improving clarity in the handler's signature. - Ensured all relevant functions now consistently handle the `user_id` parameter, streamlining the codebase and improving maintainability. * feat: Enhance team management with role-based assignments - Introduced a new `TeamInfoRole` enum to represent user roles within teams, replacing the previous boolean `assigned` field. - Updated the `list_teams` handler to return team roles instead of assignment status, improving clarity on user roles. - Refactored the `put_teams` handler to support role-based assignments, allowing for more granular control over team memberships. - Added new PUT routes for dataset groups and permission groups in the user assets router, enhancing API capabilities. - Improved SQL queries for team assignments to utilize role information, streamlining database interactions. * feat: Add organization_id to DatasetGroupPermission and update dataset group handler - Introduced a new `organization_id` field in the `DatasetGroupPermission` struct to associate permissions with specific organizations. - Updated the `put_dataset_groups_handler` to include `organization_id` when creating or updating dataset group permissions, enhancing the API's capability to manage permissions at the organizational level. - Improved SQL query formatting for better readability in the handler. * add list components for permission settings * feat: Introduce assets module and update routing for permission groups - Added a new `assets` module to organize related routes. - Updated the routing in `mod.rs` to nest the `assets` router under the `/:permission_group_id` path, enhancing the structure and clarity of the API. - Maintained existing routes for managing permission groups while improving modularity. * create permission group users * refactor: Update list_permission_groups_handler to use user_id and improve SQL queries - Changed the parameter in the SQL query from `user.id` to `user_id` for consistency with the updated user ID parameter naming convention. - Enhanced the SQL query to count distinct dataset permissions and utilize `bool_or` for identity checks, improving accuracy and performance. - Cleaned up the grouping in the SQL query by removing unnecessary fields, streamlining the data retrieval process. * add listing for dataset groups * feat: Add PUT routes for user and dataset management in assets module - Introduced new PUT routes for managing users and dataset groups in the assets module. - Updated the router to support PUT requests for `/users`, `/dataset_groups`, and `/datasets`, enhancing the API's functionality for resource updates. - Improved modularity by organizing related routes within the assets module. * add listing for dataset groups * add list for datasets * assigned permissions * Add attributes and teams * feat: Add DatasetToDatasetGroup model and update schema - Introduced a new `DatasetToDatasetGroup` struct to represent the relationship between datasets and dataset groups, including fields for timestamps and optional deletion. - Updated the database schema to include `updated_at` and `deleted_at` fields for the `datasets_to_dataset_groups` table, enhancing data tracking capabilities. - Refactored the routing in `mod.rs` to include a nested router for assets, improving the organization of dataset group routes. * invalidate query if user id is present * create a dataset modal added to user page * add team modal to teams * assigned popup * feat: Enhance user retrieval with dataset information - Added new structs `DatasetLineage` and `DatasetInfo` to represent dataset details and lineage. - Updated `UserResponse` to include a list of datasets associated with the user. - Refactored `get_user_information` function to concurrently fetch user info, direct datasets, permission group datasets, and organization datasets using `tokio::spawn` for improved performance. - Implemented logic to compile datasets based on direct access and permission group access, including lineage tracking for better data representation. - Enhanced error handling during database queries to ensure robust user information retrieval. * feat: Enhance dataset access retrieval in user and dataset overview - Updated `get_dataset_overview` to include dataset group access and permission group to dataset group access, improving the granularity of dataset permissions. - Introduced new queries to fetch dataset groups and their associated permissions, enhancing the dataset overview for users. - Refactored `get_user_information` to concurrently retrieve dataset groups and permission group datasets, optimizing performance with `tokio::spawn`. - Enhanced lineage tracking for datasets, allowing for better representation of user permissions across dataset groups and permission groups. - Improved error handling during database queries to ensure robust data retrieval. * lineage props passed * refactor: Streamline dataset access logic in get_user_information - Simplified access control logic for datasets based on user roles, consolidating conditions for WorkspaceAdmin, DataAdmin, Querier, Viewer, and RestrictedQuerier. - Enhanced dataset lineage tracking to provide clearer representation of user permissions across various dataset access types. - Removed redundant code related to dataset processing, improving readability and maintainability of the `get_user_information` function. - Ensured that datasets are correctly categorized based on direct access, permission group access, and organization datasets, optimizing the overall data retrieval process. * feat: Refactor dataset overview access lineage in get_dataset_overview - Introduced a default access lineage for users, ensuring consistent representation of user permissions. - Simplified the addition of user roles to the lineage, consolidating logic for WorkspaceAdmin, DataAdmin, Querier, and Viewer roles. - Enhanced lineage tracking for RestrictedQuerier role to include direct dataset access and permission group lineage, improving granularity of dataset permissions. - Removed redundant code related to dataset and permission group lineage, optimizing readability and maintainability of the `get_dataset_overview` function. * add datasetgroup handler * feat: Improve dataset access control and lineage tracking - Enhanced the `get_dataset_overview` function to refine access control for the `RestrictedQuerier` role, allowing for more granular permission checks based on various access paths. - Updated the `get_user_information` function to streamline dataset processing, ensuring that datasets are categorized correctly based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability of both functions. - Improved lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * clickable pills * fix build errors * add endpoints for dataset groups, and permission groups * refactor: Enhance dataset access control and lineage tracking - Streamlined the `get_dataset_overview` function to improve access control for the `RestrictedQuerier` role, ensuring more precise permission checks. - Updated the `get_user_information` function to optimize dataset processing, categorizing datasets based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability. - Enhanced lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * better handling for create a permission group * Refetch on team created for now * add additional pages * permission group modal update * added pages for permission groups * add user permission list * add datasets to permission groups page * Update dependencies and refactor Snowflake query handling - Downgraded the `base64` crate version in `Cargo.toml` from `0.22.1` to `0.21`. - Refactored the `snowflake_query` function in `snowflake_query.rs` to improve data type handling, including support for additional Arrow data types and enhanced null value checks. - Updated the `route_to_query` function in `query_router.rs` to use mutable `snowflake_client` for better state management during query execution. - Improved error handling for closing the Snowflake client session, ensuring proper logging of any issues encountered. * add datasets to permission groups page * Refactor Snowflake client connection to remove warehouse and database IDs - Updated the `get_snowflake_client` function to no longer require `warehouse_id` and `database_id`, simplifying the connection process. - This change enhances flexibility in client initialization and aligns with recent updates to Snowflake API handling. * add permission groups lists * add dataset group in permission group area * remove console logs * Big nate/bus 936 if a user is not an admin we should not see the permission (#57) * app sidebar settings should hide * rename api directories * list empty state * offset for row * refetch after user default access changed (#58) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * check if user has an org first * user organization can be null * get datasets only if they have an org * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * add hidden prop to row listing component * reroute to correct link for datasets * disable editing my own user * restrict certain routes to admins * small users page * restrict adding people to admins * update packages * Update shimmer text component * Update ShimmerText.tsx * update how no datasets is presented * fix permission check on post_dataset rest (#59) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * fix: Update SQL migration and seed data for user attributes - Modified the SQL migration to specify the schema for the `users` table, ensuring clarity in the update statement. - Adjusted the seed data for `users_to_organizations` to change the `organization_id` from 'public' to 'none', reflecting a more accurate state for user roles and organization associations. - Ensured consistency in the formatting of SQL insert statements for better readability. * fix: Prevent users from updating their own profiles - Added a check in the `update_user_handler` to prevent users from updating their own information, returning an error if they attempt to do so. - This change enhances security by ensuring that users cannot modify their own records, which could lead to unauthorized changes. * refactor: Simplify dashboard permission queries by removing team-based joins - Removed left joins with `teams_to_users` table in dashboard permission queries - Simplified permission checks to only filter by direct user ID - Updated queries in `get_user_dashboard_permission`, `get_bulk_user_dashboard_permission`, and `list_dashboards_handler` - Streamlined SQL query logic for more direct and efficient permission checks * add secure middleware checks * Update supabase server to secure and http only * dashboard permissions fix * set cookies * set cookies * Enhance email functionality by adding HTML escaping - Added `html-escape` crate to `Cargo.toml` for HTML escaping. - Updated email template processing to escape HTML in message and button text, preventing potential XSS vulnerabilities. - Modified test cases to include HTML content in email parameters, ensuring proper handling and escaping. This change improves security by sanitizing user input in email communications. * add csp policy middleware * Update cspPolicyMiddleware.ts * prevent email param injection * Dal/cli-updates-skip-dbt (#67) * feat: enhance deploy command with skip_dbt option - Updated the Deploy command to accept a `skip_dbt` boolean argument, allowing users to bypass the dbt run during deployment. - Refactored the deploy function to conditionally execute the dbt command based on the `skip_dbt` flag, improving deployment flexibility. * Refactor query engine and CLI commands for improved functionality and error handling - Updated `get_bigquery_columns` and `get_snowflake_columns` functions to enhance column name handling and ensure proper error reporting. - Modified `get_snowflake_client` to accept a database ID for better connection management. - Enhanced the `deploy` command in the CLI to include additional parameters (`path`, `data_source_name`, `schema`, `env`) for more flexible deployments. - Improved error handling and reporting in the `deploy` function, including detailed summaries of deployment errors and successful file processing. - Updated `get_model_files` to accept a directory path and added checks for file existence, enhancing robustness. - Adjusted model file structures to include schema information and refined the upload process to handle optional parameters more effectively. These changes collectively improve the usability and reliability of the query engine and deployment process. * Update dataset DDL generation to include optional YML file content - Modified `generate_sql_agent` to append optional YML file content to dataset DDL - Ensures more comprehensive dataset representation during SQL agent generation - Handles cases where YML file might be present or absent gracefully * chore: Enhance SQL agent generation with data source type context (#69) - Added support for retrieving and passing data source type during SQL agent generation - Updated SQL generation prompts to include data source type in system messages - Modified `generate_sql_agent` function to fetch and utilize data source type information - Improved SQL generation context by dynamically incorporating data source type details * Snowflake limit to prevent memory issues --------- Co-authored-by: Nate Kelley <nate@buster.so> Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> |
|
|
dal
|
48447d5bc5
|
Snowflake limit to prevent memory issues | |
|
github-actions[bot]
|
4c004d1929 | chore(release): update version to 0.0.11 | |
|
dal
|
f2544872b4
|
Staging (#70)
* chore: add release-please configuration * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * update package versions * Make users page (#39) * create users pages * abstract more components to correct folders * carve out expection if we are hiding the select all * user query * Add ability to change default access * Update package-lock.json * Update package-lock.json * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Big nate/bus 924 make additional permissions pages (#50) * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * finalize permission popup Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * more elegant infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * add additional bulk popup menus Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * make pages unique Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * change how padding is applied to list Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * start user dataset lineage Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Refactor SQL query in list_assets_handler to use a Common Table Expression (CTE) for improved readability and maintainability. The CTE, `distinct_assets`, simplifies the selection of distinct asset records before applying the final ordering and limiting. * add cursor rules for web directory + jest (#52) * on change update for segments * PRevent clicking passthrough on users click * feat: Add assets module and nest routes in user router - Introduced a new `assets` module to handle asset-related routes. - Updated the user router to nest the `assets` routes under the user ID path, enhancing the organization of API endpoints. - This change improves the structure and maintainability of the user-related routes in the API. * feat: Enhance user attribute listing with authorization checks - Updated the `list_attributes_handler` to include authorization checks for user roles and organization IDs. - Implemented error handling for unauthorized access to user attributes. - Refactored the SQL query to retrieve user attributes based on the authenticated user's organization, improving security and data integrity. - This change ensures that only authorized users can list attributes, enhancing the overall security of the API. * pass through dataset overview * feat: Update dataset group listing to include permissions - Enhanced the `list_dataset_groups` function to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset group. - Modified the `DatasetGroupInfo` struct to include `permission_id` and `assigned` fields, reflecting the new data structure. - Refactored the SQL query to group by necessary fields and ensure accurate permission data is returned, improving the functionality and security of dataset group listings. * feat: Add DatasetGroupPermission model and schema - Introduced a new `DatasetGroupPermission` struct in `models.rs` to represent permissions associated with dataset groups. - Updated the database schema in `schema.rs` to include the `dataset_groups_permissions` table, defining its structure and relationships. - Modified the `is_user_workspace_admin_or_data_admin` function in `checks.rs` to correctly reference the user's organization role, enhancing role validation logic. * add component for users inputs * feat: Update dataset group listing to include dataset group permissions - Modified the `list_dataset_groups` function to accept an additional `id` parameter for filtering dataset groups based on user permissions. - Updated the SQL query to join with the `dataset_groups_permissions` table, allowing retrieval of permission counts for each dataset group. - Refactored the `DatasetGroupInfo` struct to replace `permission_id` with `permission_count`, enhancing clarity and accuracy in the data representation. - Ensured that the query groups by the new permission structure, improving the functionality and security of dataset group listings. * feat: Refactor dataset listing to include user-specific permissions - Updated the `list_datasets` function to accept an additional `id` parameter for filtering datasets based on user permissions. - Enhanced the SQL query to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset. - Refactored the `DatasetInfo` struct to include an `assigned` field, improving clarity in the dataset representation. - Improved error handling for dataset retrieval, ensuring robust logging and response management. * fix: Correct user role attribute and enhance read-only logic in list_attributes_handler - Updated the user role attribute key from "role" to "organization_role" for accurate role retrieval. - Introduced a read-only flag for specific user attributes, improving data integrity by clearly indicating which attributes should not be modified. - Enhanced error handling for user role retrieval, ensuring robust responses for missing or incorrect attributes. * normalize header for list * new line * popup conatiner for users * feat: Enhance user authorization checks and refactor related functions - Added user authorization checks in `list_attributes`, `list_dataset_groups`, `list_datasets`, `list_permission_groups`, and `list_teams` functions to ensure only users with appropriate roles can access these resources. - Refactored the `list_teams_handler` to accept `user_id` as a parameter, improving clarity and consistency across user-related functions. - Updated SQL queries to utilize the new authorization checks, enhancing security and data integrity. - Removed redundant column allowances in `list_teams` permissions, streamlining the codebase. * pass last child as index * feat: Enhance permission group handling and streamline SQL queries - Expanded the `allow_columns_to_appear_in_same_group_by_clause!` macro in `models.rs` to include additional columns for datasets and users, improving query flexibility. - Refactored the `list_permission_groups` function to include dataset count and assigned status, enhancing the information returned for each permission group. - Updated SQL queries in `list_permission_groups` to utilize left joins for better data retrieval and to ensure accurate permission checks. - Removed redundant column allowances in various files, streamlining the codebase and improving maintainability. * feat: Add PUT route for updating teams in user assets - Introduced a new module `put_teams` to handle updates for teams. - Added a PUT route for `/teams` in the user assets router, allowing for team modifications. - Enhanced the routing capabilities of the user assets API to support both GET and PUT requests for teams. * onchagne appsegmetned update * create permission user endpoints * move files to match new page structure * refactor: Clean up routing and improve PUT teams handler - Reformatted imports in `mod.rs` for better readability. - Commented out the PUT route for `/teams` in the user assets router, indicating a potential future change or deprecation. - Updated the `put_teams` handler to return a `NoContent` response upon successful execution, enhancing clarity in API responses. - Improved error handling in the `put_teams` function for better logging and response management. * refactor: Standardize user ID parameter naming across user-related routes - Updated all user-related route handlers to use `user_id` instead of `id` for better clarity and consistency. - Modified the routing definitions in `mod.rs` to reflect the new parameter naming convention. - Enhanced the `list_permission_groups` function to accept `user_id` as a parameter, improving clarity in the handler's signature. - Ensured all relevant functions now consistently handle the `user_id` parameter, streamlining the codebase and improving maintainability. * feat: Enhance team management with role-based assignments - Introduced a new `TeamInfoRole` enum to represent user roles within teams, replacing the previous boolean `assigned` field. - Updated the `list_teams` handler to return team roles instead of assignment status, improving clarity on user roles. - Refactored the `put_teams` handler to support role-based assignments, allowing for more granular control over team memberships. - Added new PUT routes for dataset groups and permission groups in the user assets router, enhancing API capabilities. - Improved SQL queries for team assignments to utilize role information, streamlining database interactions. * feat: Add organization_id to DatasetGroupPermission and update dataset group handler - Introduced a new `organization_id` field in the `DatasetGroupPermission` struct to associate permissions with specific organizations. - Updated the `put_dataset_groups_handler` to include `organization_id` when creating or updating dataset group permissions, enhancing the API's capability to manage permissions at the organizational level. - Improved SQL query formatting for better readability in the handler. * add list components for permission settings * feat: Introduce assets module and update routing for permission groups - Added a new `assets` module to organize related routes. - Updated the routing in `mod.rs` to nest the `assets` router under the `/:permission_group_id` path, enhancing the structure and clarity of the API. - Maintained existing routes for managing permission groups while improving modularity. * create permission group users * refactor: Update list_permission_groups_handler to use user_id and improve SQL queries - Changed the parameter in the SQL query from `user.id` to `user_id` for consistency with the updated user ID parameter naming convention. - Enhanced the SQL query to count distinct dataset permissions and utilize `bool_or` for identity checks, improving accuracy and performance. - Cleaned up the grouping in the SQL query by removing unnecessary fields, streamlining the data retrieval process. * add listing for dataset groups * feat: Add PUT routes for user and dataset management in assets module - Introduced new PUT routes for managing users and dataset groups in the assets module. - Updated the router to support PUT requests for `/users`, `/dataset_groups`, and `/datasets`, enhancing the API's functionality for resource updates. - Improved modularity by organizing related routes within the assets module. * add listing for dataset groups * add list for datasets * assigned permissions * Add attributes and teams * feat: Add DatasetToDatasetGroup model and update schema - Introduced a new `DatasetToDatasetGroup` struct to represent the relationship between datasets and dataset groups, including fields for timestamps and optional deletion. - Updated the database schema to include `updated_at` and `deleted_at` fields for the `datasets_to_dataset_groups` table, enhancing data tracking capabilities. - Refactored the routing in `mod.rs` to include a nested router for assets, improving the organization of dataset group routes. * invalidate query if user id is present * create a dataset modal added to user page * add team modal to teams * assigned popup * feat: Enhance user retrieval with dataset information - Added new structs `DatasetLineage` and `DatasetInfo` to represent dataset details and lineage. - Updated `UserResponse` to include a list of datasets associated with the user. - Refactored `get_user_information` function to concurrently fetch user info, direct datasets, permission group datasets, and organization datasets using `tokio::spawn` for improved performance. - Implemented logic to compile datasets based on direct access and permission group access, including lineage tracking for better data representation. - Enhanced error handling during database queries to ensure robust user information retrieval. * feat: Enhance dataset access retrieval in user and dataset overview - Updated `get_dataset_overview` to include dataset group access and permission group to dataset group access, improving the granularity of dataset permissions. - Introduced new queries to fetch dataset groups and their associated permissions, enhancing the dataset overview for users. - Refactored `get_user_information` to concurrently retrieve dataset groups and permission group datasets, optimizing performance with `tokio::spawn`. - Enhanced lineage tracking for datasets, allowing for better representation of user permissions across dataset groups and permission groups. - Improved error handling during database queries to ensure robust data retrieval. * lineage props passed * refactor: Streamline dataset access logic in get_user_information - Simplified access control logic for datasets based on user roles, consolidating conditions for WorkspaceAdmin, DataAdmin, Querier, Viewer, and RestrictedQuerier. - Enhanced dataset lineage tracking to provide clearer representation of user permissions across various dataset access types. - Removed redundant code related to dataset processing, improving readability and maintainability of the `get_user_information` function. - Ensured that datasets are correctly categorized based on direct access, permission group access, and organization datasets, optimizing the overall data retrieval process. * feat: Refactor dataset overview access lineage in get_dataset_overview - Introduced a default access lineage for users, ensuring consistent representation of user permissions. - Simplified the addition of user roles to the lineage, consolidating logic for WorkspaceAdmin, DataAdmin, Querier, and Viewer roles. - Enhanced lineage tracking for RestrictedQuerier role to include direct dataset access and permission group lineage, improving granularity of dataset permissions. - Removed redundant code related to dataset and permission group lineage, optimizing readability and maintainability of the `get_dataset_overview` function. * add datasetgroup handler * feat: Improve dataset access control and lineage tracking - Enhanced the `get_dataset_overview` function to refine access control for the `RestrictedQuerier` role, allowing for more granular permission checks based on various access paths. - Updated the `get_user_information` function to streamline dataset processing, ensuring that datasets are categorized correctly based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability of both functions. - Improved lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * clickable pills * fix build errors * add endpoints for dataset groups, and permission groups * refactor: Enhance dataset access control and lineage tracking - Streamlined the `get_dataset_overview` function to improve access control for the `RestrictedQuerier` role, ensuring more precise permission checks. - Updated the `get_user_information` function to optimize dataset processing, categorizing datasets based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability. - Enhanced lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * better handling for create a permission group * Refetch on team created for now * add additional pages * permission group modal update * added pages for permission groups * add user permission list * add datasets to permission groups page * Update dependencies and refactor Snowflake query handling - Downgraded the `base64` crate version in `Cargo.toml` from `0.22.1` to `0.21`. - Refactored the `snowflake_query` function in `snowflake_query.rs` to improve data type handling, including support for additional Arrow data types and enhanced null value checks. - Updated the `route_to_query` function in `query_router.rs` to use mutable `snowflake_client` for better state management during query execution. - Improved error handling for closing the Snowflake client session, ensuring proper logging of any issues encountered. * add datasets to permission groups page * Refactor Snowflake client connection to remove warehouse and database IDs - Updated the `get_snowflake_client` function to no longer require `warehouse_id` and `database_id`, simplifying the connection process. - This change enhances flexibility in client initialization and aligns with recent updates to Snowflake API handling. * add permission groups lists * add dataset group in permission group area * remove console logs * Big nate/bus 936 if a user is not an admin we should not see the permission (#57) * app sidebar settings should hide * rename api directories * list empty state * offset for row * refetch after user default access changed (#58) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * check if user has an org first * user organization can be null * get datasets only if they have an org * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * add hidden prop to row listing component * reroute to correct link for datasets * disable editing my own user * restrict certain routes to admins * small users page * restrict adding people to admins * update packages * Update shimmer text component * Update ShimmerText.tsx * update how no datasets is presented * fix permission check on post_dataset rest (#59) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * fix: Update SQL migration and seed data for user attributes - Modified the SQL migration to specify the schema for the `users` table, ensuring clarity in the update statement. - Adjusted the seed data for `users_to_organizations` to change the `organization_id` from 'public' to 'none', reflecting a more accurate state for user roles and organization associations. - Ensured consistency in the formatting of SQL insert statements for better readability. * fix: Prevent users from updating their own profiles - Added a check in the `update_user_handler` to prevent users from updating their own information, returning an error if they attempt to do so. - This change enhances security by ensuring that users cannot modify their own records, which could lead to unauthorized changes. * refactor: Simplify dashboard permission queries by removing team-based joins - Removed left joins with `teams_to_users` table in dashboard permission queries - Simplified permission checks to only filter by direct user ID - Updated queries in `get_user_dashboard_permission`, `get_bulk_user_dashboard_permission`, and `list_dashboards_handler` - Streamlined SQL query logic for more direct and efficient permission checks * add secure middleware checks * Update supabase server to secure and http only * dashboard permissions fix * set cookies * set cookies * Enhance email functionality by adding HTML escaping - Added `html-escape` crate to `Cargo.toml` for HTML escaping. - Updated email template processing to escape HTML in message and button text, preventing potential XSS vulnerabilities. - Modified test cases to include HTML content in email parameters, ensuring proper handling and escaping. This change improves security by sanitizing user input in email communications. * add csp policy middleware * Update cspPolicyMiddleware.ts * prevent email param injection * Dal/cli-updates-skip-dbt (#67) * feat: enhance deploy command with skip_dbt option - Updated the Deploy command to accept a `skip_dbt` boolean argument, allowing users to bypass the dbt run during deployment. - Refactored the deploy function to conditionally execute the dbt command based on the `skip_dbt` flag, improving deployment flexibility. * Refactor query engine and CLI commands for improved functionality and error handling - Updated `get_bigquery_columns` and `get_snowflake_columns` functions to enhance column name handling and ensure proper error reporting. - Modified `get_snowflake_client` to accept a database ID for better connection management. - Enhanced the `deploy` command in the CLI to include additional parameters (`path`, `data_source_name`, `schema`, `env`) for more flexible deployments. - Improved error handling and reporting in the `deploy` function, including detailed summaries of deployment errors and successful file processing. - Updated `get_model_files` to accept a directory path and added checks for file existence, enhancing robustness. - Adjusted model file structures to include schema information and refined the upload process to handle optional parameters more effectively. These changes collectively improve the usability and reliability of the query engine and deployment process. * Update dataset DDL generation to include optional YML file content - Modified `generate_sql_agent` to append optional YML file content to dataset DDL - Ensures more comprehensive dataset representation during SQL agent generation - Handles cases where YML file might be present or absent gracefully * chore: Enhance SQL agent generation with data source type context (#69) - Added support for retrieving and passing data source type during SQL agent generation - Updated SQL generation prompts to include data source type in system messages - Modified `generate_sql_agent` function to fetch and utilize data source type information - Improved SQL generation context by dynamically incorporating data source type details --------- Co-authored-by: Nate Kelley <nate@buster.so> Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> |
|
|
dal
|
466abcf0bc
|
Merge branch 'main' into staging | |
|
dal
|
6271e00b70
|
chore: Enhance SQL agent generation with data source type context (#69)
- Added support for retrieving and passing data source type during SQL agent generation - Updated SQL generation prompts to include data source type in system messages - Modified `generate_sql_agent` function to fetch and utilize data source type information - Improved SQL generation context by dynamically incorporating data source type details |
|
|
github-actions[bot]
|
3f676f38cb | chore(release): update version to 0.0.10 | |
|
dal
|
43da79e0cf
|
fix: cli tooling (#68)
* chore: add release-please configuration * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * update package versions * Make users page (#39) * create users pages * abstract more components to correct folders * carve out expection if we are hiding the select all * user query * Add ability to change default access * Update package-lock.json * Update package-lock.json * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Big nate/bus 924 make additional permissions pages (#50) * create virtua list component * only debounce if there is text * prefetch on demand * add a popup for permissions * Update web/src/components/list/BusterList/BusterListReactWindow.tsx Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> --------- Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * finalize permission popup Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * more elegant infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * add additional bulk popup menus Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * make pages unique Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * change how padding is applied to list Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * infinite list component Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * start user dataset lineage Co-Authored-By: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> * Refactor SQL query in list_assets_handler to use a Common Table Expression (CTE) for improved readability and maintainability. The CTE, `distinct_assets`, simplifies the selection of distinct asset records before applying the final ordering and limiting. * add cursor rules for web directory + jest (#52) * on change update for segments * PRevent clicking passthrough on users click * feat: Add assets module and nest routes in user router - Introduced a new `assets` module to handle asset-related routes. - Updated the user router to nest the `assets` routes under the user ID path, enhancing the organization of API endpoints. - This change improves the structure and maintainability of the user-related routes in the API. * feat: Enhance user attribute listing with authorization checks - Updated the `list_attributes_handler` to include authorization checks for user roles and organization IDs. - Implemented error handling for unauthorized access to user attributes. - Refactored the SQL query to retrieve user attributes based on the authenticated user's organization, improving security and data integrity. - This change ensures that only authorized users can list attributes, enhancing the overall security of the API. * pass through dataset overview * feat: Update dataset group listing to include permissions - Enhanced the `list_dataset_groups` function to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset group. - Modified the `DatasetGroupInfo` struct to include `permission_id` and `assigned` fields, reflecting the new data structure. - Refactored the SQL query to group by necessary fields and ensure accurate permission data is returned, improving the functionality and security of dataset group listings. * feat: Add DatasetGroupPermission model and schema - Introduced a new `DatasetGroupPermission` struct in `models.rs` to represent permissions associated with dataset groups. - Updated the database schema in `schema.rs` to include the `dataset_groups_permissions` table, defining its structure and relationships. - Modified the `is_user_workspace_admin_or_data_admin` function in `checks.rs` to correctly reference the user's organization role, enhancing role validation logic. * add component for users inputs * feat: Update dataset group listing to include dataset group permissions - Modified the `list_dataset_groups` function to accept an additional `id` parameter for filtering dataset groups based on user permissions. - Updated the SQL query to join with the `dataset_groups_permissions` table, allowing retrieval of permission counts for each dataset group. - Refactored the `DatasetGroupInfo` struct to replace `permission_id` with `permission_count`, enhancing clarity and accuracy in the data representation. - Ensured that the query groups by the new permission structure, improving the functionality and security of dataset group listings. * feat: Refactor dataset listing to include user-specific permissions - Updated the `list_datasets` function to accept an additional `id` parameter for filtering datasets based on user permissions. - Enhanced the SQL query to join with the `dataset_permissions` table, allowing retrieval of permission details for each dataset. - Refactored the `DatasetInfo` struct to include an `assigned` field, improving clarity in the dataset representation. - Improved error handling for dataset retrieval, ensuring robust logging and response management. * fix: Correct user role attribute and enhance read-only logic in list_attributes_handler - Updated the user role attribute key from "role" to "organization_role" for accurate role retrieval. - Introduced a read-only flag for specific user attributes, improving data integrity by clearly indicating which attributes should not be modified. - Enhanced error handling for user role retrieval, ensuring robust responses for missing or incorrect attributes. * normalize header for list * new line * popup conatiner for users * feat: Enhance user authorization checks and refactor related functions - Added user authorization checks in `list_attributes`, `list_dataset_groups`, `list_datasets`, `list_permission_groups`, and `list_teams` functions to ensure only users with appropriate roles can access these resources. - Refactored the `list_teams_handler` to accept `user_id` as a parameter, improving clarity and consistency across user-related functions. - Updated SQL queries to utilize the new authorization checks, enhancing security and data integrity. - Removed redundant column allowances in `list_teams` permissions, streamlining the codebase. * pass last child as index * feat: Enhance permission group handling and streamline SQL queries - Expanded the `allow_columns_to_appear_in_same_group_by_clause!` macro in `models.rs` to include additional columns for datasets and users, improving query flexibility. - Refactored the `list_permission_groups` function to include dataset count and assigned status, enhancing the information returned for each permission group. - Updated SQL queries in `list_permission_groups` to utilize left joins for better data retrieval and to ensure accurate permission checks. - Removed redundant column allowances in various files, streamlining the codebase and improving maintainability. * feat: Add PUT route for updating teams in user assets - Introduced a new module `put_teams` to handle updates for teams. - Added a PUT route for `/teams` in the user assets router, allowing for team modifications. - Enhanced the routing capabilities of the user assets API to support both GET and PUT requests for teams. * onchagne appsegmetned update * create permission user endpoints * move files to match new page structure * refactor: Clean up routing and improve PUT teams handler - Reformatted imports in `mod.rs` for better readability. - Commented out the PUT route for `/teams` in the user assets router, indicating a potential future change or deprecation. - Updated the `put_teams` handler to return a `NoContent` response upon successful execution, enhancing clarity in API responses. - Improved error handling in the `put_teams` function for better logging and response management. * refactor: Standardize user ID parameter naming across user-related routes - Updated all user-related route handlers to use `user_id` instead of `id` for better clarity and consistency. - Modified the routing definitions in `mod.rs` to reflect the new parameter naming convention. - Enhanced the `list_permission_groups` function to accept `user_id` as a parameter, improving clarity in the handler's signature. - Ensured all relevant functions now consistently handle the `user_id` parameter, streamlining the codebase and improving maintainability. * feat: Enhance team management with role-based assignments - Introduced a new `TeamInfoRole` enum to represent user roles within teams, replacing the previous boolean `assigned` field. - Updated the `list_teams` handler to return team roles instead of assignment status, improving clarity on user roles. - Refactored the `put_teams` handler to support role-based assignments, allowing for more granular control over team memberships. - Added new PUT routes for dataset groups and permission groups in the user assets router, enhancing API capabilities. - Improved SQL queries for team assignments to utilize role information, streamlining database interactions. * feat: Add organization_id to DatasetGroupPermission and update dataset group handler - Introduced a new `organization_id` field in the `DatasetGroupPermission` struct to associate permissions with specific organizations. - Updated the `put_dataset_groups_handler` to include `organization_id` when creating or updating dataset group permissions, enhancing the API's capability to manage permissions at the organizational level. - Improved SQL query formatting for better readability in the handler. * add list components for permission settings * feat: Introduce assets module and update routing for permission groups - Added a new `assets` module to organize related routes. - Updated the routing in `mod.rs` to nest the `assets` router under the `/:permission_group_id` path, enhancing the structure and clarity of the API. - Maintained existing routes for managing permission groups while improving modularity. * create permission group users * refactor: Update list_permission_groups_handler to use user_id and improve SQL queries - Changed the parameter in the SQL query from `user.id` to `user_id` for consistency with the updated user ID parameter naming convention. - Enhanced the SQL query to count distinct dataset permissions and utilize `bool_or` for identity checks, improving accuracy and performance. - Cleaned up the grouping in the SQL query by removing unnecessary fields, streamlining the data retrieval process. * add listing for dataset groups * feat: Add PUT routes for user and dataset management in assets module - Introduced new PUT routes for managing users and dataset groups in the assets module. - Updated the router to support PUT requests for `/users`, `/dataset_groups`, and `/datasets`, enhancing the API's functionality for resource updates. - Improved modularity by organizing related routes within the assets module. * add listing for dataset groups * add list for datasets * assigned permissions * Add attributes and teams * feat: Add DatasetToDatasetGroup model and update schema - Introduced a new `DatasetToDatasetGroup` struct to represent the relationship between datasets and dataset groups, including fields for timestamps and optional deletion. - Updated the database schema to include `updated_at` and `deleted_at` fields for the `datasets_to_dataset_groups` table, enhancing data tracking capabilities. - Refactored the routing in `mod.rs` to include a nested router for assets, improving the organization of dataset group routes. * invalidate query if user id is present * create a dataset modal added to user page * add team modal to teams * assigned popup * feat: Enhance user retrieval with dataset information - Added new structs `DatasetLineage` and `DatasetInfo` to represent dataset details and lineage. - Updated `UserResponse` to include a list of datasets associated with the user. - Refactored `get_user_information` function to concurrently fetch user info, direct datasets, permission group datasets, and organization datasets using `tokio::spawn` for improved performance. - Implemented logic to compile datasets based on direct access and permission group access, including lineage tracking for better data representation. - Enhanced error handling during database queries to ensure robust user information retrieval. * feat: Enhance dataset access retrieval in user and dataset overview - Updated `get_dataset_overview` to include dataset group access and permission group to dataset group access, improving the granularity of dataset permissions. - Introduced new queries to fetch dataset groups and their associated permissions, enhancing the dataset overview for users. - Refactored `get_user_information` to concurrently retrieve dataset groups and permission group datasets, optimizing performance with `tokio::spawn`. - Enhanced lineage tracking for datasets, allowing for better representation of user permissions across dataset groups and permission groups. - Improved error handling during database queries to ensure robust data retrieval. * lineage props passed * refactor: Streamline dataset access logic in get_user_information - Simplified access control logic for datasets based on user roles, consolidating conditions for WorkspaceAdmin, DataAdmin, Querier, Viewer, and RestrictedQuerier. - Enhanced dataset lineage tracking to provide clearer representation of user permissions across various dataset access types. - Removed redundant code related to dataset processing, improving readability and maintainability of the `get_user_information` function. - Ensured that datasets are correctly categorized based on direct access, permission group access, and organization datasets, optimizing the overall data retrieval process. * feat: Refactor dataset overview access lineage in get_dataset_overview - Introduced a default access lineage for users, ensuring consistent representation of user permissions. - Simplified the addition of user roles to the lineage, consolidating logic for WorkspaceAdmin, DataAdmin, Querier, and Viewer roles. - Enhanced lineage tracking for RestrictedQuerier role to include direct dataset access and permission group lineage, improving granularity of dataset permissions. - Removed redundant code related to dataset and permission group lineage, optimizing readability and maintainability of the `get_dataset_overview` function. * add datasetgroup handler * feat: Improve dataset access control and lineage tracking - Enhanced the `get_dataset_overview` function to refine access control for the `RestrictedQuerier` role, allowing for more granular permission checks based on various access paths. - Updated the `get_user_information` function to streamline dataset processing, ensuring that datasets are categorized correctly based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability of both functions. - Improved lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * clickable pills * fix build errors * add endpoints for dataset groups, and permission groups * refactor: Enhance dataset access control and lineage tracking - Streamlined the `get_dataset_overview` function to improve access control for the `RestrictedQuerier` role, ensuring more precise permission checks. - Updated the `get_user_information` function to optimize dataset processing, categorizing datasets based on direct access and permission group access. - Removed redundant code and improved readability by consolidating logic for user roles, enhancing maintainability. - Enhanced lineage tracking for datasets, providing a clearer representation of user permissions across different access types. * better handling for create a permission group * Refetch on team created for now * add additional pages * permission group modal update * added pages for permission groups * add user permission list * add datasets to permission groups page * Update dependencies and refactor Snowflake query handling - Downgraded the `base64` crate version in `Cargo.toml` from `0.22.1` to `0.21`. - Refactored the `snowflake_query` function in `snowflake_query.rs` to improve data type handling, including support for additional Arrow data types and enhanced null value checks. - Updated the `route_to_query` function in `query_router.rs` to use mutable `snowflake_client` for better state management during query execution. - Improved error handling for closing the Snowflake client session, ensuring proper logging of any issues encountered. * add datasets to permission groups page * Refactor Snowflake client connection to remove warehouse and database IDs - Updated the `get_snowflake_client` function to no longer require `warehouse_id` and `database_id`, simplifying the connection process. - This change enhances flexibility in client initialization and aligns with recent updates to Snowflake API handling. * add permission groups lists * add dataset group in permission group area * remove console logs * Big nate/bus 936 if a user is not an admin we should not see the permission (#57) * app sidebar settings should hide * rename api directories * list empty state * offset for row * refetch after user default access changed (#58) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * check if user has an org first * user organization can be null * get datasets only if they have an org * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * add hidden prop to row listing component * reroute to correct link for datasets * disable editing my own user * restrict certain routes to admins * small users page * restrict adding people to admins * update packages * Update shimmer text component * Update ShimmerText.tsx * update how no datasets is presented * fix permission check on post_dataset rest (#59) * fix permission check on post_dataset rest * refactor: enhance dataset overview access lineage and permission checks - Updated the `get_dataset_overview` function to conditionally add default access lineage based on user roles and existing access paths. - Simplified the logic for adding user roles to the lineage, ensuring clarity and maintainability. - Improved handling for the `RestrictedQuerier` role to include checks for existing access before adding default lineage, enhancing permission accuracy. - Streamlined code by removing redundant checks and consolidating role handling, optimizing overall readability. * feat: Enhance permission group handling and data retrieval - Introduced a new `PermissionGroupInfo` struct to encapsulate detailed information about permission groups, including user and dataset counts. - Updated the `get_permission_group` and `list_permission_groups` functions to improve data retrieval and error handling. - Refactored SQL queries in `list_permission_groups` to include additional joins for counting users and datasets associated with permission groups, enhancing the overall functionality and clarity of the API. - Streamlined code for better readability and maintainability, ensuring consistent handling of user and permission group data. * refactor: Improve dataset access handling and permission checks - Enhanced the `get_restricted_user_datasets` and `get_restricted_user_datasets_with_metadata` functions to include additional permission checks for dataset groups and permission groups. - Consolidated SQL queries to ensure proper filtering of deleted records and improved clarity in dataset retrieval logic. - Introduced new joins and filters to handle dataset group permissions, ensuring accurate access control for users. - Streamlined code for better readability and maintainability, enhancing overall functionality in dataset access management. * fix: Update SQL migration and seed data for user attributes - Modified the SQL migration to specify the schema for the `users` table, ensuring clarity in the update statement. - Adjusted the seed data for `users_to_organizations` to change the `organization_id` from 'public' to 'none', reflecting a more accurate state for user roles and organization associations. - Ensured consistency in the formatting of SQL insert statements for better readability. * fix: Prevent users from updating their own profiles - Added a check in the `update_user_handler` to prevent users from updating their own information, returning an error if they attempt to do so. - This change enhances security by ensuring that users cannot modify their own records, which could lead to unauthorized changes. * refactor: Simplify dashboard permission queries by removing team-based joins - Removed left joins with `teams_to_users` table in dashboard permission queries - Simplified permission checks to only filter by direct user ID - Updated queries in `get_user_dashboard_permission`, `get_bulk_user_dashboard_permission`, and `list_dashboards_handler` - Streamlined SQL query logic for more direct and efficient permission checks * add secure middleware checks * Update supabase server to secure and http only * dashboard permissions fix * set cookies * set cookies * Enhance email functionality by adding HTML escaping - Added `html-escape` crate to `Cargo.toml` for HTML escaping. - Updated email template processing to escape HTML in message and button text, preventing potential XSS vulnerabilities. - Modified test cases to include HTML content in email parameters, ensuring proper handling and escaping. This change improves security by sanitizing user input in email communications. * add csp policy middleware * Update cspPolicyMiddleware.ts * prevent email param injection * Dal/cli-updates-skip-dbt (#67) * feat: enhance deploy command with skip_dbt option - Updated the Deploy command to accept a `skip_dbt` boolean argument, allowing users to bypass the dbt run during deployment. - Refactored the deploy function to conditionally execute the dbt command based on the `skip_dbt` flag, improving deployment flexibility. * Refactor query engine and CLI commands for improved functionality and error handling - Updated `get_bigquery_columns` and `get_snowflake_columns` functions to enhance column name handling and ensure proper error reporting. - Modified `get_snowflake_client` to accept a database ID for better connection management. - Enhanced the `deploy` command in the CLI to include additional parameters (`path`, `data_source_name`, `schema`, `env`) for more flexible deployments. - Improved error handling and reporting in the `deploy` function, including detailed summaries of deployment errors and successful file processing. - Updated `get_model_files` to accept a directory path and added checks for file existence, enhancing robustness. - Adjusted model file structures to include schema information and refined the upload process to handle optional parameters more effectively. These changes collectively improve the usability and reliability of the query engine and deployment process. * Update dataset DDL generation to include optional YML file content - Modified `generate_sql_agent` to append optional YML file content to dataset DDL - Ensures more comprehensive dataset representation during SQL agent generation - Handles cases where YML file might be present or absent gracefully --------- Co-authored-by: Nate Kelley <nate@buster.so> Co-authored-by: Nate Kelley <133379588+nate-kelley-buster@users.noreply.github.com> Co-authored-by: ellipsis-dev[bot] <65095814+ellipsis-dev[bot]@users.noreply.github.com> |
|
|
dal
|
39dfc053fc
|
Merge branch 'main' into staging | |
|
dal
|
5202438fa8
|
Dal/cli-updates-skip-dbt (#67)
* feat: enhance deploy command with skip_dbt option - Updated the Deploy command to accept a `skip_dbt` boolean argument, allowing users to bypass the dbt run during deployment. - Refactored the deploy function to conditionally execute the dbt command based on the `skip_dbt` flag, improving deployment flexibility. * Refactor query engine and CLI commands for improved functionality and error handling - Updated `get_bigquery_columns` and `get_snowflake_columns` functions to enhance column name handling and ensure proper error reporting. - Modified `get_snowflake_client` to accept a database ID for better connection management. - Enhanced the `deploy` command in the CLI to include additional parameters (`path`, `data_source_name`, `schema`, `env`) for more flexible deployments. - Improved error handling and reporting in the `deploy` function, including detailed summaries of deployment errors and successful file processing. - Updated `get_model_files` to accept a directory path and added checks for file existence, enhancing robustness. - Adjusted model file structures to include schema information and refined the upload process to handle optional parameters more effectively. These changes collectively improve the usability and reliability of the query engine and deployment process. * Update dataset DDL generation to include optional YML file content - Modified `generate_sql_agent` to append optional YML file content to dataset DDL - Ensures more comprehensive dataset representation during SQL agent generation - Handles cases where YML file might be present or absent gracefully |
|
Nate Kelley
|
63c5ba2190
|
Merge pull request #66 from buster-so/nate/security-remediation
Nate/security remediation |
|
|
Nate Kelley
|
2208977a87
|
prevent email param injection | |
|
Nate Kelley
|
62ce1b8c9c
|
Update cspPolicyMiddleware.ts | |
|
Nate Kelley
|
4a777efab7
|
Merge pull request #65 from buster-so/nate/security-remediation
Nate/security remediation |