Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

DilmurodMak · 2024-12-02T19:21:24Z

Pull Request Overview

This PR updates and enhances the Databricks deployment process using Terraform and Asset Bundle Deployment via GitHub Actions. It simplifies deployment for multi environment deployment.

Key Highlights

Folder Structure:
- Sample Code is Organized under single_tech_samples/databricks/databricks_terraform
- Includes directories for:
  - Infra: Terraform code for Azure resource deployment.
  - tests: Scripts and workflows for Databricks testing.
  - utils: Helper scripts like generate-databricks-workflows.sh.
  - workflows: Pre-generated Databricks workflows to run tests in Databricks workspace.
CI/CD Pipelines:
- Linting Pipeline: Validates Python notebooks and workflows.
- Sandbox Deployment: Validates and deploys Databricks assets, executes test workflows.
- Development Deployment: Deploys assets to the development environment following successful sandbox deployment and runs the same tests in development environment

Testing Steps

The Sample code covers the deployment from sandbox to development environment.

Create a new branch and submit a PR to main.
Verify the following:
- Linting Pipeline runs and validates code.
- Sandbox Deployment Pipeline validates and tests workflows during the PR.
Merge the PR to main.
Observe:
- Sandbox assets are deployed, and test workflows are executed successfully.
- Development deployment triggers upon successful sandbox completion.

ydaponte · 2024-12-04T09:41:15Z

@DilmurodMak - one of the validations is failing, can you take a look?

DilmurodMak · 2024-12-04T20:20:57Z

@DilmurodMak - one of the validations is failing, can you take a look?

@ydaponte , The pipeline are templates, it requires databricks workspaces exist and its urls are set in databricks.yml file. There for its failing. We can put it as template reference in the single solution doc instead of running in PRs if we do not want to trigger it

…ying resources

ydaponte

Leaving some comments that need to be addressed before we can merge into main. There are some best practices and alignment with the overall repo that will need to be done as for example the creation of a devcontainer for the sample. Thanks for the great work so far!

ydaponte · 2024-12-16T08:24:29Z

single_tech_samples/databricks/databricks_terraform/README.md

+
+This repository contains databricks deployment using terraform and databricks asset bundle deployment using Github Actions.
+
+## Prerequisites


Suggested change

## Prerequisites

Removing the Prerequisites from here as you can consolidate later in the ReadMe. Also in the second prerequisite you jump into another ReadMe - I think the flow is not perfect.

First READ.me is for the root - It is intended to describe overall repo,
Second READ.me is in Infra folder - its intended to give instructions how to deploy all the resources using terraform.

I will make it more clear

ydaponte · 2024-12-16T08:24:43Z

single_tech_samples/databricks/databricks_terraform/README.md

+
+## Prerequisites
+
+- Azure Subscription


Suggested change

- Azure Subscription

ydaponte · 2024-12-16T08:24:55Z

single_tech_samples/databricks/databricks_terraform/README.md

+## Prerequisites
+
+- Azure Subscription
+- Deploy Azure Resources using Terraform Code. See [./Infra/README.MD](./Infra/README.md)


Suggested change

- Deploy Azure Resources using Terraform Code. See [./Infra/README.MD](./Infra/README.md)

This is a repetition from the later pre-requisites: Ensure Sandbox and Development Resources are deployed in Azure using Terraform code in ./Infra folder. I'm suggesting removing it from here.

single_tech_samples/databricks/databricks_terraform/README.md

ydaponte · 2024-12-16T11:14:00Z

single_tech_samples/databricks/databricks_terraform/Infra/README.md

+
+### Pre-requisites
+- Clone the repository
+- Install Terraform CLI if not installed already [Terraform Installation](https://learn.hashicorp.com/tutorials/terraform/install-cli)


You can add this pre-requirements to be installed when launching the devcontainer

ydaponte · 2024-12-16T11:16:01Z

single_tech_samples/databricks/databricks_terraform/Infra/README.md

+cd Infra/deployment
+```
+
+2. Login to Azure


The Login to Azure should be probably the first step, second step should be the az account set and third step should be the Change directory to ...

ydaponte · 2024-12-16T11:18:37Z

single_tech_samples/databricks/databricks_terraform/Infra/README.md

+- **`adb-workspace`** - Deploys Databricks workspace
+- **`metastore-and-users`** - Creates Databricks Connector, Creates Storage Account, Give storage access rights to connector, Creates Metastore / Assigns Workspace to Metastore, and Finally Retrieves alls users, groups, and service principals from Azure AD.
+- **`adb-unity-catalog`** - Gives databricks access rights to the connector, Creates containers in the storage account, and creates external locations for the containers. Creates unity catalog and grants permissions user groups. Finally, creates **`bronze` `silver` `gold`** schemas under the catalog and gives the required permissions to the user groups.
+


If the script fails with an error - lack of permissions or something else - is the script idempotent? Meaning, can we re-run and the script will continue where it left of? Can we make a note about that?

Yes, script can be run where its left off. It references the state files of each terraform model when running, and based on the state it can continue where its left off.

I will make a note of that

There is a note on that in READ.me of Infra, that the end of the doc

ydaponte · 2024-12-16T11:20:28Z

...mples/databricks/databricks_terraform/images/databricks-asset-bundle-deploymeny-pipeline.png

Instead of the png, can you please commit the drawio.svg version file of this diagram? We are starting to create a standard in the repo for that.

These was originally images, I will try to recreate in draw.io if necessary

ydaponte · 2024-12-16T11:21:00Z

single_tech_samples/databricks/databricks_terraform/images/architecture.png

Is there an editable version of this diagram that can be commited instead?

Co-authored-by: Yennifer Santos <[email protected]>

github-actions · 2024-12-17T18:43:38Z

single_tech_samples/databricks/README.md

+
+## Samples
+
+- [IaC deployment of Azure Databricks](./databricks_ci_cd/README.md) - This sample demonstrates how to deploy an Azure Databricks environment using ARM templates.


🚫 [linkspector] _{reported by reviewdog 🐶}
Cannot reach ./databricks_ci_cd/README.md. Status: 404 Cannot find: ./databricks_ci_cd/README.md

DilmurodMak added 11 commits November 19, 2024 15:35

linting pipeline

22247d7

Infra Deployment

931373d

terraform samples azure databricks

70dd64e

asset bundles terraform

820b75d

terraform samples databricks

1513ade

terraform code

0c19567

asset bundle linting code

b8fdf45

linting pipeline

3754af7

databricks asset bundle setup

254ff18

sandbox and dev deployment pipelines

56824a9

databricks asset bundles sample code

fa4e787

DilmurodMak requested review from hemaverma and ydaponte December 2, 2024 19:21

DilmurodMak had a problem deploying to sandbox December 2, 2024 19:21 — with GitHub Actions Failure

DilmurodMak added 2 commits December 2, 2024 15:04

yaml lint

080ca8e

doc lint

ed1e96a

DilmurodMak had a problem deploying to sandbox December 2, 2024 20:06 — with GitHub Actions Failure

DilmurodMak had a problem deploying to sandbox December 2, 2024 20:39 — with GitHub Actions Failure

yaml indentation issue

c7f4208

ydaponte assigned DilmurodMak Dec 4, 2024

ydaponte added the single-tech: azure-databricks Related to Azure Databricks single-tech sample label Dec 4, 2024

ydaponte added this to the E2E - parking_sensor_databricks - V1 milestone Dec 4, 2024

update varaible instruction

3c8f9e4

DilmurodMak had a problem deploying to sandbox December 6, 2024 14:20 — with GitHub Actions Failure

wait 120 sec metastore creation

47eb1bd

DilmurodMak had a problem deploying to sandbox December 11, 2024 21:02 — with GitHub Actions Failure

metastore name

d874dd6

DilmurodMak had a problem deploying to sandbox December 11, 2024 21:31 — with GitHub Actions Failure

updated using single shell script for deploying resources and destoro…

9417ead

…ying resources

DilmurodMak had a problem deploying to sandbox December 12, 2024 20:27 — with GitHub Actions Failure

shell script cleanup

e32e68d

DilmurodMak had a problem deploying to sandbox December 12, 2024 20:32 — with GitHub Actions Failure

doc update

50f2bcf

DilmurodMak had a problem deploying to sandbox December 12, 2024 20:43 — with GitHub Actions Failure

ydaponte requested changes Dec 16, 2024

View reviewed changes

Update single_tech_samples/databricks/databricks_terraform/README.md

36cb922

Co-authored-by: Yennifer Santos <[email protected]>

DilmurodMak had a problem deploying to sandbox December 16, 2024 14:48 — with GitHub Actions Failure

Update single_tech_samples/databricks/databricks_terraform/README.md

82da343

Co-authored-by: Yennifer Santos <[email protected]>

DilmurodMak had a problem deploying to sandbox December 16, 2024 14:48 — with GitHub Actions Failure

update doc and add devcontainer

309e849

DilmurodMak requested a review from ydaponte December 16, 2024 18:22

DilmurodMak added 3 commits December 16, 2024 15:56

adding doc update

784b520

add visio link to the doc for images

e30e28b

merge main resolve conflcits

7a2695b

github-actions bot reviewed Dec 17, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

DilmurodMak commented Dec 2, 2024

ydaponte commented Dec 4, 2024

DilmurodMak commented Dec 4, 2024 •

edited

Loading

ydaponte left a comment

ydaponte Dec 16, 2024

DilmurodMak Dec 16, 2024

ydaponte Dec 16, 2024

ydaponte Dec 16, 2024

ydaponte Dec 16, 2024

ydaponte Dec 16, 2024

ydaponte Dec 16, 2024

DilmurodMak Dec 16, 2024

DilmurodMak Dec 16, 2024

ydaponte Dec 16, 2024

DilmurodMak Dec 16, 2024

ydaponte Dec 16, 2024

github-actions bot Dec 17, 2024


		This repository contains databricks deployment using terraform and databricks asset bundle deployment using Github Actions.

		## Prerequisites


		## Samples

		- [IaC deployment of Azure Databricks](./databricks_ci_cd/README.md) - This sample demonstrates how to deploy an Azure Databricks environment using ARM templates.

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

Are you sure you want to change the base?

Databricks Sample - Terraform IaC for Azure Databricks and Asset Bundle Deployment via CI/CD #911

Conversation

DilmurodMak commented Dec 2, 2024

Pull Request Overview

Key Highlights

Testing Steps

ydaponte commented Dec 4, 2024

DilmurodMak commented Dec 4, 2024 • edited Loading

ydaponte left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

github-actions bot Dec 17, 2024

Choose a reason for hiding this comment

DilmurodMak commented Dec 4, 2024 •

edited

Loading