Terraform Interview Questions and Answers Part 8 - Terraform for an Azure Data Platform

Terraform Interview Q&A · Part 8 of 10

In this post we are going to cover Terraform for an Azure data platform. These are 7 real Terraform interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario: you must deliver a full dev data platform on Azure in one week.

A data platform on Azure is usually a lake on ADLS Gen2, a Key Vault, Data Factory and Databricks, all wired together with identities and private networking. Interviewers ask about it because this is where Terraform meets real security: keys, secrets, roles and who owns what.

Part 8 of the Real Time Terraform Interview Questions and Answers series on TechBrothersIT.

On this page
  1. The short answer
  2. Let's set up the project
  3. Question 1: Your manager asks for a data lake on Azure with Terraform. Which resources make up a secure ADLS Gen2 setup?
  4. Question 2: Secrets are pasted into tfvars files. How do you handle secrets with Key Vault and Terraform?
  5. Question 3: A pipeline needs to read the lake without keys. How do managed identities and role assignments work in Terraform?
  6. Question 4: The team wants Databricks workspaces created by Terraform. How do the azurerm and databricks providers work together?
  7. Question 5: A teammate asks whether Terraform should also deploy Data Factory pipelines or only the factory. Where do you draw the line?
  8. Question 6: Every resource has different tags and names. How do you enforce naming and tagging standards?
  9. Scenario 7: You must deliver a full dev data platform on Azure in one week with Terraform. What is your plan?
  10. The full script

The short answer

Say this first

A secure lake is a storage account with the hierarchical namespace, shared keys off, public access disabled and private endpoints. Secrets flow through ephemeral values into write-only arguments, pipelines use managed identities with scoped role assignments, and the databricks provider manages what's inside the workspace. Terraform owns the Data Factory platform, not the pipelines, and locals, validation and tests enforce names and tags.

Let's set up the project

We use a small Terraform 1.17 project with real azurerm 5.11 and databricks 1.140 code for a dev platform. Nothing here is applied to Azure. We check the code with terraform validate, and with terraform test using mocked providers, so no Azure credentials are needed. Here are the starting files.

versions.tf
terraform {
  required_version = ">= 1.17"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 5.11"
    }
    databricks = {
      source  = "databricks/databricks"
      version = "~> 1.140"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}
providers.tf
provider "azurerm" {
  features {}

  storage_use_azuread             = true # Entra ID, no account keys
  resource_provider_registrations = "none"
  resource_providers_to_register = [
    "Microsoft.Storage", "Microsoft.KeyVault", "Microsoft.Sql",
    "Microsoft.DataFactory", "Microsoft.Databricks",
    "Microsoft.ManagedIdentity", "Microsoft.Network",
  ]
}

# inside the workspace: configured from the azurerm workspace
provider "databricks" {
  host                        = azurerm_databricks_workspace.dbw.workspace_url
  azure_workspace_resource_id = azurerm_databricks_workspace.dbw.id
}
variables.tf
variable "environment" {
  type = string

  validation {
    condition     = contains(["dev", "test", "prod"], var.environment)
    error_message = "environment must be one of dev, test, prod."
  }
}

variable "project" {
  type        = string
  description = "Short project code, used in every name."

  validation {
    condition     = can(regex("^[a-z]{2,8}$", var.project))
    error_message = "project must be 2 to 8 lowercase letters."
  }
}
inputs.tf
variable "cost_center" {
  type = string
}

variable "location" {
  type    = string
  default = "eastus2"
}

variable "private_subnet_id" {
  type        = string
  description = "Subnet for the private endpoints."
}

variable "private_dns_zone_ids" {
  type        = map(string)
  description = "Private DNS zone ID for each endpoint: blob and dfs."
}

variable "dbw_vnet" {
  type = object({
    id                  = string
    host_subnet         = string
    container_subnet    = string
    host_nsg_assoc      = string
    container_nsg_assoc = string
  })
  description = "Databricks VNet injection: VNet, delegated subnets, NSG associations."
}
locals.tf
locals {
  base = "${var.project}-data-${var.environment}" # sales-data-dev

  names = {
    rg       = "rg-${local.base}"
    lake     = "st${var.project}data${var.environment}" # max 24
    kv       = "kv-${var.project}-${var.environment}"
    sql      = "sql-${local.base}"
    adf      = "adf-${local.base}"
    dbw      = "dbw-${local.base}"
    pipeline = "id-${local.base}-pipeline"
  }

  tags = {
    environment = var.environment
    project     = var.project
    cost_center = var.cost_center
    managed_by  = "terraform"
  }
}
main.tf
resource "azurerm_resource_group" "data" {
  name     = local.names.rg
  location = var.location
  tags     = local.tags
}
lake.tf
resource "azurerm_storage_account" "lake" {
  name                              = local.names.lake
  resource_group_name               = azurerm_resource_group.data.name
  location                          = var.location
  account_tier                      = "Standard"
  account_replication_type          = "ZRS"
  is_hns_enabled                    = true # ADLS Gen2
  shared_access_key_enabled         = false
  public_network_access             = "Disabled"
  min_tls_version                   = "TLS1_2"
  infrastructure_encryption_enabled = true
  tags                              = local.tags

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
  }

  blob_properties {
    delete_retention_policy {
      days = 14
    }
    container_delete_retention_policy {
      days = 14
    }
  }
}
lake_layers.tf
resource "azurerm_storage_data_lake_gen2_filesystem" "layer" {
  for_each           = toset(["landing", "raw", "curated"])
  name               = each.key
  storage_account_id = azurerm_storage_account.lake.id
}

resource "azurerm_private_endpoint" "lake" {
  for_each            = toset(["blob", "dfs"])
  name                = "pe-${local.names.lake}-${each.key}"
  resource_group_name = azurerm_resource_group.data.name
  location            = var.location
  subnet_id           = var.private_subnet_id
  tags                = local.tags

  private_service_connection {
    name                           = each.key
    private_connection_resource_id = azurerm_storage_account.lake.id
    subresource_names              = [each.key]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "privatelink-${each.key}"
    private_dns_zone_ids = [var.private_dns_zone_ids[each.key]]
  }
}
keyvault.tf
data "azurerm_client_config" "current" {}

resource "azurerm_key_vault" "kv" {
  name                          = local.names.kv
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  tenant_id                     = data.azurerm_client_config.current.tenant_id
  sku_name                      = "standard"
  rbac_authorization_enabled    = true
  purge_protection_enabled      = true
  soft_delete_retention_days    = 30
  public_network_access_enabled = false
  tags                          = local.tags
}
sql.tf
# generated during the run: never in Git, tfvars, plan or state
ephemeral "random_password" "sql_admin" {
  length = 32
}

resource "azurerm_key_vault_secret" "sql_admin" {
  name             = "sql-admin-password"
  key_vault_id     = azurerm_key_vault.kv.id
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = 1 # bump to rotate
}

resource "azurerm_mssql_server" "meta" {
  name                          = local.names.sql
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  version                       = "12.0"
  administrator_login           = "sqladmin"
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false
  tags                          = local.tags

  administrator_login_password_wo_version = 1 # bump to rotate
  administrator_login_password_wo = (
    ephemeral.random_password.sql_admin.result
  )
}
identity.tf
resource "azurerm_user_assigned_identity" "pipeline" {
  name                = local.names.pipeline
  resource_group_name = azurerm_resource_group.data.name
  location            = var.location
  tags                = local.tags
}

resource "azurerm_role_assignment" "pipeline_lake" {
  scope                = azurerm_storage_account.lake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_user_assigned_identity.pipeline.principal_id
  principal_type       = "ServicePrincipal"
}
adf.tf
resource "azurerm_data_factory" "adf" {
  name                            = local.names.adf
  resource_group_name             = azurerm_resource_group.data.name
  location                        = var.location
  public_network_enabled          = false
  managed_virtual_network_enabled = true
  tags                            = local.tags

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.pipeline.id]
  }

  dynamic "github_configuration" { # only dev is connected to Git
    for_each = var.environment == "dev" ? [1] : []
    content {
      account_name    = "sales-data-org"
      repository_name = "adf-sales"
      branch_name     = "main"
      root_folder     = "/adf"
    }
  }
}
databricks.tf
resource "azurerm_databricks_workspace" "dbw" {
  name                        = local.names.dbw
  resource_group_name         = azurerm_resource_group.data.name
  location                    = var.location
  sku                         = "premium" # needed for Unity Catalog
  managed_resource_group_name = "rg-${local.base}-dbw-managed"
  tags                        = local.tags

  custom_parameters { # VNet injection: clusters reach the private lake
    no_public_ip        = true
    virtual_network_id  = var.dbw_vnet.id
    public_subnet_name  = var.dbw_vnet.host_subnet
    private_subnet_name = var.dbw_vnet.container_subnet

    public_subnet_network_security_group_association_id  = var.dbw_vnet.host_nsg_assoc
    private_subnet_network_security_group_association_id = var.dbw_vnet.container_nsg_assoc
  }
}
access_connector.tf
resource "azurerm_databricks_access_connector" "uc" {
  name                = "dbac-${local.base}"
  resource_group_name = azurerm_resource_group.data.name
  location            = var.location
  tags                = local.tags

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_role_assignment" "uc_lake" {
  scope                = azurerm_storage_account.lake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_type       = "ServicePrincipal"
  principal_id = (
    azurerm_databricks_access_connector.uc.identity[0].principal_id
  )
}
unity.tf
locals {
  lake_dfs = "${azurerm_storage_account.lake.name}.dfs.core.windows.net"
}

resource "databricks_storage_credential" "lake" {
  name = "sc-${local.base}"

  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.uc.id
  }
}

resource "databricks_external_location" "curated" {
  name            = "curated"
  url             = "abfss://curated@${local.lake_dfs}/"
  credential_name = databricks_storage_credential.lake.name
  depends_on      = [azurerm_role_assignment.uc_lake]
}
tests/mocks/azurerm.tfmock.hcl
# Mocked azurerm: realistic IDs, because the provider still
# validates the values that reach its arguments.
mock_data "azurerm_client_config" {
  defaults = {
    tenant_id = "11111111-1111-1111-1111-111111111111"
  }
}

mock_resource "azurerm_resource_group" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev"
  }
}

mock_resource "azurerm_storage_account" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.Storage/storageAccounts/stsalesdatadev"
  }
}

mock_resource "azurerm_key_vault" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.KeyVault/vaults/kv-sales-dev"
  }
}

mock_resource "azurerm_user_assigned_identity" {
  defaults = {
    id           = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-sales-data-dev-pipeline"
    principal_id = "22222222-2222-2222-2222-222222222222"
  }
}

mock_resource "azurerm_databricks_access_connector" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.Databricks/accessConnectors/dbac-sales-data-dev"
  }
}

mock_resource "azurerm_databricks_workspace" {
  defaults = {
    id            = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.Databricks/workspaces/dbw-sales-data-dev"
    workspace_url = "adb-1234567890123456.7.azuredatabricks.net"
  }
}
tests/terraform.tfvars
environment       = "dev"
project           = "sales"
cost_center       = "cc-4410"
private_subnet_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev/subnets/snet-private-endpoints"
private_dns_zone_ids = {
  blob = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net"
  dfs  = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/privateDnsZones/privatelink.dfs.core.windows.net"
}
dbw_vnet = {
  id                  = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev"
  host_subnet         = "snet-dbw-host"
  container_subnet    = "snet-dbw-container"
  host_nsg_assoc      = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev/subnets/snet-dbw-host"
  container_nsg_assoc = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev/subnets/snet-dbw-container"
}
tests/lake.tftest.hcl
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "lake_is_private_and_keyless" {
  command = plan

  assert {
    condition     = azurerm_storage_account.lake.is_hns_enabled
    error_message = "The lake needs the hierarchical namespace."
  }

  assert {
    condition     = !azurerm_storage_account.lake.shared_access_key_enabled
    error_message = "Shared keys must be off: Entra ID only."
  }

  assert {
    condition     = azurerm_storage_account.lake.public_network_access == "Disabled"
    error_message = "The lake must not be reachable from the internet."
  }
}
tests/secrets.tftest.hcl
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "no_password_in_state" {
  command = apply

  assert {
    condition     = azurerm_key_vault_secret.sql_admin.value == null
    error_message = "The secret value must not be stored in state."
  }

  assert {
    condition     = azurerm_mssql_server.meta.administrator_login_password == null
    error_message = "The SQL password must not be stored in state."
  }
}
tests/access.tftest.hcl
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "pipeline_identity_gets_the_lake_only" {
  command = apply

  assert {
    condition     = azurerm_role_assignment.pipeline_lake.scope == azurerm_storage_account.lake.id
    error_message = "Scope the role to the lake, not the subscription."
  }

  assert {
    condition     = azurerm_role_assignment.pipeline_lake.principal_id == azurerm_user_assigned_identity.pipeline.principal_id
    error_message = "The role must go to the pipeline identity."
  }

  assert {
    condition     = contains(azurerm_data_factory.adf.identity[0].identity_ids, azurerm_user_assigned_identity.pipeline.id)
    error_message = "Data Factory must run as the pipeline identity."
  }
}
tests/databricks.tftest.hcl
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "unity_catalog_points_at_the_lake" {
  command = plan

  assert {
    condition     = try(azurerm_databricks_workspace.dbw.custom_parameters[0].virtual_network_id, null) == var.dbw_vnet.id
    error_message = "The workspace must be injected into our VNet."
  }

  assert {
    condition     = databricks_external_location.curated.url == "abfss://curated@stsalesdatadev.dfs.core.windows.net/"
    error_message = "The external location must point at the curated layer."
  }
}
tests/naming.tftest.hcl
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "names_follow_the_standard" {
  command = plan

  assert {
    condition     = azurerm_storage_account.lake.name == "stsalesdatadev"
    error_message = "The storage account name breaks the standard."
  }

  assert {
    condition     = azurerm_data_factory.adf.name == "adf-sales-data-dev"
    error_message = "The Data Factory name breaks the standard."
  }
}

run "every_resource_has_a_cost_center" {
  command = plan

  assert {
    condition = alltrue([for t in [
      azurerm_resource_group.data.tags,
      azurerm_storage_account.lake.tags,
      azurerm_key_vault.kv.tags,
      azurerm_data_factory.adf.tags,
      azurerm_databricks_workspace.dbw.tags,
    ] : t["cost_center"] == "cc-4410"])
    error_message = "Every resource needs the cost_center tag."
  }
}

run "rejects_a_bad_project_code" {
  command = plan

  variables {
    project = "Sales-Data"
  }

  expect_failures = [var.project]
}

Question 1: Your manager asks for a data lake on Azure with Terraform. Which resources make up a secure ADLS Gen2 setup?

A resource group, a storage account with the hierarchical namespace on, a filesystem per layer, and private endpoints for blob and dfs with their private DNS zones.

The hierarchical namespace, is_hns_enabled = true, is what makes the account ADLS Gen2. In lake_layers.tf, for_each creates the landing, raw and curated filesystems and the two private endpoints.

Then we lock the account down in lake.tf: shared keys off, so only Entra ID, public_network_access = "Disabled", network rules that deny by default, TLS1_2, and soft delete for blobs and containers. Let's run terraform validate on the whole platform.

Terminal
$ terraform validate
Success! The configuration is valid.

Success! The configuration is valid. Now a test with a mocked azurerm provider checks the plan. We run terraform test for the lake test file.

lake.tf
resource "azurerm_storage_account" "lake" {
  name                              = local.names.lake
  resource_group_name               = azurerm_resource_group.data.name
  location                          = var.location
  account_tier                      = "Standard"
  account_replication_type          = "ZRS"
  is_hns_enabled                    = true # ADLS Gen2
  shared_access_key_enabled         = false
  public_network_access             = "Disabled"
  min_tls_version                   = "TLS1_2"
  infrastructure_encryption_enabled = true
  tags                              = local.tags

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
  }

  blob_properties {
    delete_retention_policy {
      days = 14
    }
    container_delete_retention_policy {
      days = 14
    }
  }
}
Terminal
$ terraform test -filter=tests/lake.tftest.hcl
tests/lake.tftest.hcl... in progress
  run "lake_is_private_and_keyless"... pass
tests/lake.tftest.hcl... tearing down
tests/lake.tftest.hcl... pass

Success! 1 passed, 0 failed.

run "lake_is_private_and_keyless"... pass: namespace on, shared keys off, public access disabled.

The pitfall is old examples. azurerm 5 deprecated public_network_access_enabled in favor of public_network_access. Let's paste the 4.x-style argument back in and validate.

lake.tf
resource "azurerm_storage_account" "lake" {
  name                              = local.names.lake
  resource_group_name               = azurerm_resource_group.data.name
  location                          = var.location
  account_tier                      = "Standard"
  account_replication_type          = "ZRS"
  is_hns_enabled                    = true # ADLS Gen2
  shared_access_key_enabled         = false
  public_network_access_enabled     = false # 4.x style
  min_tls_version                   = "TLS1_2"
  infrastructure_encryption_enabled = true
  tags                              = local.tags

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
  }

  blob_properties {
    delete_retention_policy {
      days = 14
    }
    container_delete_retention_policy {
      days = 14
    }
  }
}
Terminal
$ terraform validate
Warning: Argument is deprecated

  with azurerm_storage_account.lake,
  on lake.tf line 9, in resource "azurerm_storage_account" "lake":
   9:   public_network_access_enabled     = false # 4.x style

`public_network_access_enabled` has been deprecated in favour of
`public_network_access` and will be removed in v6.0 of the AzureRM provider
Success! The configuration is valid, but there were some validation warnings
as shown above.

Validate warns: Argument is deprecated, and it will be removed in v6.0 of the AzureRM provider.

Tip

With shared keys off, set storage_use_azuread = true in the provider, so Terraform uses Entra ID for the filesystems too. The identity that runs it needs a data role on the account, and a network path to it.

Question 2: Secrets are pasted into tfvars files. How do you handle secrets with Key Vault and Terraform?

Keep them out of Git, out of tfvars, and out of state.

Three rules to remember:

  • Not in Git, not in .tfvars, not in state
  • sensitive = true hides it in the CLI output only: it is still in state
  • data "azurerm_key_vault_secret": the value lands in state

The pattern since Terraform 1.11 is an ephemeral random_password, created during the run and never stored. It goes into write-only arguments: value_wo on the Key Vault secret, and administrator_login_password_wo on the SQL server. Let's validate sql.tf and run the secrets test, which applies the config with a mocked azurerm and checks the state.

sql.tf
# generated during the run: never in Git, tfvars, plan or state
ephemeral "random_password" "sql_admin" {
  length = 32
}

resource "azurerm_key_vault_secret" "sql_admin" {
  name             = "sql-admin-password"
  key_vault_id     = azurerm_key_vault.kv.id
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = 1 # bump to rotate
}

resource "azurerm_mssql_server" "meta" {
  name                          = local.names.sql
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  version                       = "12.0"
  administrator_login           = "sqladmin"
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false
  tags                          = local.tags

  administrator_login_password_wo_version = 1 # bump to rotate
  administrator_login_password_wo = (
    ephemeral.random_password.sql_admin.result
  )
}
Terminal
$ terraform validate
Success! The configuration is valid.
$ terraform test -filter=tests/secrets.tftest.hcl
tests/secrets.tftest.hcl... in progress
  run "no_password_in_state"... pass
tests/secrets.tftest.hcl... tearing down
tests/secrets.tftest.hcl... pass

Success! 1 passed, 0 failed.

Validate passes, and run "no_password_in_state"... pass: neither the secret value nor the SQL password is stored.

Now let's put the ephemeral value into the normal administrator_login_password argument and validate again.

sql.tf
# generated during the run: never in Git, tfvars, plan or state
ephemeral "random_password" "sql_admin" {
  length = 32
}

resource "azurerm_key_vault_secret" "sql_admin" {
  name             = "sql-admin-password"
  key_vault_id     = azurerm_key_vault.kv.id
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = 1 # bump to rotate
}

resource "azurerm_mssql_server" "meta" {
  name                          = local.names.sql
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  version                       = "12.0"
  administrator_login           = "sqladmin"
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false
  tags                          = local.tags

  administrator_login_password = (
    ephemeral.random_password.sql_admin.result
  )
}
Terminal (expected error)
$ terraform validate
Error: Invalid use of ephemeral value

  with azurerm_mssql_server.meta,
  on sql.tf line 23, in resource "azurerm_mssql_server" "meta":
  23:   administrator_login_password = (
  24:     ephemeral.random_password.sql_admin.result
  25:   )

Ephemeral values are not valid for "administrator_login_password", because it
is not a write-only attribute and must be persisted to state.

Validate refuses: Ephemeral values are not valid for "administrator_login_password", because it is not a write-only attribute and must be persisted to state.

Terraform can't compare a write-only value, so to rotate, you bump the version, value_wo_version or administrator_login_password_wo_version.

Real project

The vault itself uses RBAC, purge protection and no public access, and apps read the secret at runtime, with their own identity.

Question 3: A pipeline needs to read the lake without keys. How do managed identities and role assignments work in Terraform?

Create a user-assigned managed identity, attach it to Data Factory, and give it a data role on the lake with a role assignment. No keys, and no secrets to rotate.

The role assignment in identity.tf has three parts:

  • The scope: the lake account, stsalesdatadev
  • The role: Storage Blob Data Contributor, because this pipeline also writes curated. For read only, Storage Blob Data Reader is enough
  • The principal: the principal_id of id-sales-data-dev-pipeline

Then Data Factory runs as that identity:

adf.tf (excerpt)
# adf.tf: Data Factory runs as the pipeline identity
identity {
  type         = "UserAssigned"
  identity_ids = [azurerm_user_assigned_identity.pipeline.id]
}

Let's run the access test. It applies the config with a mocked provider and checks the wiring.

Terminal
$ terraform test -filter=tests/access.tftest.hcl
tests/access.tftest.hcl... in progress
  run "pipeline_identity_gets_the_lake_only"... pass
tests/access.tftest.hcl... tearing down
tests/access.tftest.hcl... pass

Success! 1 passed, 0 failed.

run "pipeline_identity_gets_the_lake_only"... pass: the scope is the lake, the principal is the pipeline identity, and Data Factory runs as it.

Two tips. Set principal_type = "ServicePrincipal": it helps when a brand-new identity hasn't replicated yet. And the identity that runs Terraform needs permission to create role assignments on that scope.

Why user-assigned

It lives on its own, so you can grant access before the factory exists, and reuse it. Databricks gets its own: the access connector's SystemAssigned identity, with the same role on the lake.

Question 4: The team wants Databricks workspaces created by Terraform. How do the azurerm and databricks providers work together?

azurerm creates the Azure side, and the databricks provider manages what is inside the workspace.

First, azurerm in databricks.tf and access_connector.tf. The workspace is on the premium tier, which Unity Catalog needs. It is injected into our virtual network, so clusters reach the private lake. And there is an access connector, whose identity gets a data role on the lake.

Then the databricks provider is configured with host = azurerm_databricks_workspace.dbw.workspace_url and azure_workspace_resource_id = azurerm_databricks_workspace.dbw.id. In unity.tf, a storage credential uses the access connector, and an external location points at the curated filesystem. Let's run the mocked plan test.

Terminal
$ terraform test -filter=tests/databricks.tftest.hcl
tests/databricks.tftest.hcl... in progress
  run "unity_catalog_points_at_the_lake"... pass
tests/databricks.tftest.hcl... tearing down
tests/databricks.tftest.hcl... pass

Success! 1 passed, 0 failed.

run "unity_catalog_points_at_the_lake"... pass: the workspace is in our network, and the external location points at abfss://curated@stsalesdatadev.dfs.core.windows.net/.

Pitfall

Provider arguments should only use values known before apply, and on a new environment the workspace doesn't exist yet. In a real project, I split it: one state for the Azure resources (workspace, access connector, roles), one for what's inside the workspace (credentials, locations, catalogs, clusters, jobs).

Account-level objects, like the network connectivity config that gives serverless compute private endpoints to the lake, need a second databricks provider, with host = "https://accounts.azuredatabricks.net" and account_id. And if Unity Catalog was enabled automatically, there is no metastore to create.

Question 5: A teammate asks whether Terraform should also deploy Data Factory pipelines or only the factory. Where do you draw the line?

Terraform owns the platform: the factory, its identity, the managed virtual network and the Git connection. Everything inside the factory, pipelines, datasets, linked services and triggers, is application code, built in ADF Studio.

azurerm could do both. Let's count the Data Factory resource types in the provider schema.

Terminal
$ terraform providers schema -json > schema.json
$ jq -r '.provider_schemas[].resource_schemas | keys[]' schema.json > types.txt
$ grep -c '^azurerm_data_factory' types.txt
51
$ grep -E '^azurerm_data_factory_(pipeline|trigger_schedule|dataset_parquet)$' types.txt
azurerm_data_factory_dataset_parquet
azurerm_data_factory_pipeline
azurerm_data_factory_trigger_schedule

There are 51 resource types that start with azurerm_data_factory, including azurerm_data_factory_pipeline, azurerm_data_factory_trigger_schedule and azurerm_data_factory_dataset_parquet.

But pipelines written in HCL are hard to build and review. So only the dev factory is connected to Git, with the dynamic "github_configuration" block in adf.tf. Test and prod get their contents from the ADF CI/CD process: publish from Git, then deploy the template.

ObjectOwnerHow it is deployed
Factory, identity, managed VNet, Git connectionTerraformplan and apply
Its role assignments, Key Vault, the lakeTerraformplan and apply
Pipelines, datasets, linked services, triggersADF Git + CI/CDpublish, then deploy the template

The pitfall is two owners. The published template includes the linked services, so if Terraform also manages them, each deployment undoes the other. One owner per object.

Real project

Terraform outputs the factory name and Key Vault URL, and the ADF release passes them in as parameters.

Question 6: Every resource has different tags and names. How do you enforce naming and tagging standards?

In code first. locals.tf builds every name and the tag map from a few inputs, environment, project and cost center, and every resource uses them.

Validation stops bad inputs. The project code must be 2 to 8 lowercase letters, which also keeps the storage account name within 24 characters.

Then tests prove it. Let's run the naming tests.

Terminal
$ terraform test -filter=tests/naming.tftest.hcl
tests/naming.tftest.hcl... in progress
  run "names_follow_the_standard"... pass
  run "every_resource_has_a_cost_center"... pass
  run "rejects_a_bad_project_code"... pass
tests/naming.tftest.hcl... tearing down
tests/naming.tftest.hcl... pass

Success! 3 passed, 0 failed.

Success! 3 passed, 0 failed.: the names follow the standard, every resource has a cost center, and the bad project code Sales-Data is rejected.

And know the provider: azurerm has no provider-level default tags. Let's check the provider schema for any provider-level setting with tags in its name.

Terminal
$ jq '[.provider_schemas[].provider.block | .attributes, .block_types | objects | keys[]] | any(test("tags"))' schema.json
false

The check returns false. So pass the tags on every resource, or wrap resources in modules that do.

In Azure too

Enforce it with a policy that requires or inherits tags. If the policy adds tags of its own, use ignore_changes on just those keys, as in part 7.

Scenario 7: You must deliver a full dev data platform on Azure in one week with Terraform. What is your plan?

One layer per day, and a pipeline that checks everything before anything is applied.

  • Day 1: state backend, OIDC identity, names and tags, resource providers
  • Day 2: resource group, network, Databricks subnets, private DNS zones
  • Day 3: lake, Key Vault, private endpoints, write-only secrets
  • Day 4: Data Factory, Databricks, Unity Catalog
  • Day 5: pipeline: fmt, validate, test, plan

On day 1, note that azurerm 5 registers no resource providers by default, so providers.tf sets resource_provider_registrations = "none" and lists the ones we need in resource_providers_to_register.

On day 5, every pull request runs the format check, validate and the tests. Let's run all three.

Terminal
$ terraform fmt -check -recursive
$ terraform validate
Success! The configuration is valid.
$ terraform test
tests/access.tftest.hcl... in progress
  run "pipeline_identity_gets_the_lake_only"... pass
tests/access.tftest.hcl... tearing down
tests/access.tftest.hcl... pass
tests/databricks.tftest.hcl... in progress
  run "unity_catalog_points_at_the_lake"... pass
tests/databricks.tftest.hcl... tearing down
tests/databricks.tftest.hcl... pass
tests/lake.tftest.hcl... in progress
  run "lake_is_private_and_keyless"... pass
tests/lake.tftest.hcl... tearing down
tests/lake.tftest.hcl... pass
tests/naming.tftest.hcl... in progress
  run "names_follow_the_standard"... pass
  run "every_resource_has_a_cost_center"... pass
  run "rejects_a_bad_project_code"... pass
tests/naming.tftest.hcl... tearing down
tests/naming.tftest.hcl... pass
tests/secrets.tftest.hcl... in progress
  run "no_password_in_state"... pass
tests/secrets.tftest.hcl... tearing down
tests/secrets.tftest.hcl... pass

Success! 7 passed, 0 failed.

fmt -check prints nothing, validate passes, and the tests end with Success! 7 passed, 0 failed., with no Azure credentials.

Real project

Then plan and apply to dev through the pipeline, never from a laptop. The workspace contents get their own state, and on Friday I show the team a plan with no changes.

Key points to remember

  • Lake: hierarchical namespace, no shared keys, no public access, private endpoints
  • Secrets: ephemeral values into write-only arguments, nothing in state
  • Access: managed identities + role assignments scoped to the lake
  • azurerm builds the workspace, databricks manages what is inside
  • Terraform owns the ADF platform; locals, validation and tests enforce names and tags

The full script

Here is a Python script that writes all the files and runs every command in order. You need Terraform 1.17 on your PATH; run it in an empty folder. It only validates and tests, so it needs no Azure credentials.

Show the full script that writes the files and runs every step
Full script - writes the Terraform files and runs every step (Terraform 1.17 on your PATH), run it with Python
# ========================================================================
# Terraform Interview Q&A #8 - Terraform for an Azure data platform
# TechBrothersIT. Names and sample values are invented for teaching.
# Every step below was run with Terraform 1.17.0; the outputs in the video and the blog post are the real outputs.
# Providers used: hashicorp/random, null, local, time (they create nothing in any cloud). Azure and Databricks
# code in this series is checked with terraform validate only: nothing is created in a real subscription.
# To run it yourself (Linux, macOS or WSL; some steps use grep / cat):
#   1. Install Terraform 1.17 from developer.hashicorp.com/terraform/install and put it on your PATH.
#   2. python Terraform_Interview_QA_08_Code.py   in an empty folder (terraform init downloads the providers).
# ========================================================================

import os, subprocess

def write(path, text):
    if text is None:
        if os.path.exists(path): os.remove(path)
        return
    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
    with open(path, "w", encoding="utf-8") as f:
        f.write(text.strip("\n") + "\n")

def run(cmd, expect_error=False, allow_error=False, env=None):
    print("\n$ " + cmd, flush=True)
    rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
    if rc != 0 and not (expect_error or allow_error):
        raise SystemExit("stopped: this command was expected to succeed")

# Step 0: the starting files
write('versions.tf', """\
terraform {
  required_version = ">= 1.17"

  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 5.11"
    }
    databricks = {
      source  = "databricks/databricks"
      version = "~> 1.140"
    }
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}
""")
write('providers.tf', """\
provider "azurerm" {
  features {}

  storage_use_azuread             = true # Entra ID, no account keys
  resource_provider_registrations = "none"
  resource_providers_to_register = [
    "Microsoft.Storage", "Microsoft.KeyVault", "Microsoft.Sql",
    "Microsoft.DataFactory", "Microsoft.Databricks",
    "Microsoft.ManagedIdentity", "Microsoft.Network",
  ]
}

# inside the workspace: configured from the azurerm workspace
provider "databricks" {
  host                        = azurerm_databricks_workspace.dbw.workspace_url
  azure_workspace_resource_id = azurerm_databricks_workspace.dbw.id
}
""")
write('variables.tf', """\
variable "environment" {
  type = string

  validation {
    condition     = contains(["dev", "test", "prod"], var.environment)
    error_message = "environment must be one of dev, test, prod."
  }
}

variable "project" {
  type        = string
  description = "Short project code, used in every name."

  validation {
    condition     = can(regex("^[a-z]{2,8}$", var.project))
    error_message = "project must be 2 to 8 lowercase letters."
  }
}
""")
write('inputs.tf', """\
variable "cost_center" {
  type = string
}

variable "location" {
  type    = string
  default = "eastus2"
}

variable "private_subnet_id" {
  type        = string
  description = "Subnet for the private endpoints."
}

variable "private_dns_zone_ids" {
  type        = map(string)
  description = "Private DNS zone ID for each endpoint: blob and dfs."
}

variable "dbw_vnet" {
  type = object({
    id                  = string
    host_subnet         = string
    container_subnet    = string
    host_nsg_assoc      = string
    container_nsg_assoc = string
  })
  description = "Databricks VNet injection: VNet, delegated subnets, NSG associations."
}
""")
write('locals.tf', """\
locals {
  base = "${var.project}-data-${var.environment}" # sales-data-dev

  names = {
    rg       = "rg-${local.base}"
    lake     = "st${var.project}data${var.environment}" # max 24
    kv       = "kv-${var.project}-${var.environment}"
    sql      = "sql-${local.base}"
    adf      = "adf-${local.base}"
    dbw      = "dbw-${local.base}"
    pipeline = "id-${local.base}-pipeline"
  }

  tags = {
    environment = var.environment
    project     = var.project
    cost_center = var.cost_center
    managed_by  = "terraform"
  }
}
""")
write('main.tf', """\
resource "azurerm_resource_group" "data" {
  name     = local.names.rg
  location = var.location
  tags     = local.tags
}
""")
write('lake.tf', """\
resource "azurerm_storage_account" "lake" {
  name                              = local.names.lake
  resource_group_name               = azurerm_resource_group.data.name
  location                          = var.location
  account_tier                      = "Standard"
  account_replication_type          = "ZRS"
  is_hns_enabled                    = true # ADLS Gen2
  shared_access_key_enabled         = false
  public_network_access             = "Disabled"
  min_tls_version                   = "TLS1_2"
  infrastructure_encryption_enabled = true
  tags                              = local.tags

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
  }

  blob_properties {
    delete_retention_policy {
      days = 14
    }
    container_delete_retention_policy {
      days = 14
    }
  }
}
""")
write('lake_layers.tf', """\
resource "azurerm_storage_data_lake_gen2_filesystem" "layer" {
  for_each           = toset(["landing", "raw", "curated"])
  name               = each.key
  storage_account_id = azurerm_storage_account.lake.id
}

resource "azurerm_private_endpoint" "lake" {
  for_each            = toset(["blob", "dfs"])
  name                = "pe-${local.names.lake}-${each.key}"
  resource_group_name = azurerm_resource_group.data.name
  location            = var.location
  subnet_id           = var.private_subnet_id
  tags                = local.tags

  private_service_connection {
    name                           = each.key
    private_connection_resource_id = azurerm_storage_account.lake.id
    subresource_names              = [each.key]
    is_manual_connection           = false
  }

  private_dns_zone_group {
    name                 = "privatelink-${each.key}"
    private_dns_zone_ids = [var.private_dns_zone_ids[each.key]]
  }
}
""")
write('keyvault.tf', """\
data "azurerm_client_config" "current" {}

resource "azurerm_key_vault" "kv" {
  name                          = local.names.kv
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  tenant_id                     = data.azurerm_client_config.current.tenant_id
  sku_name                      = "standard"
  rbac_authorization_enabled    = true
  purge_protection_enabled      = true
  soft_delete_retention_days    = 30
  public_network_access_enabled = false
  tags                          = local.tags
}
""")
write('sql.tf', """\
# generated during the run: never in Git, tfvars, plan or state
ephemeral "random_password" "sql_admin" {
  length = 32
}

resource "azurerm_key_vault_secret" "sql_admin" {
  name             = "sql-admin-password"
  key_vault_id     = azurerm_key_vault.kv.id
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = 1 # bump to rotate
}

resource "azurerm_mssql_server" "meta" {
  name                          = local.names.sql
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  version                       = "12.0"
  administrator_login           = "sqladmin"
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false
  tags                          = local.tags

  administrator_login_password_wo_version = 1 # bump to rotate
  administrator_login_password_wo = (
    ephemeral.random_password.sql_admin.result
  )
}
""")
write('identity.tf', """\
resource "azurerm_user_assigned_identity" "pipeline" {
  name                = local.names.pipeline
  resource_group_name = azurerm_resource_group.data.name
  location            = var.location
  tags                = local.tags
}

resource "azurerm_role_assignment" "pipeline_lake" {
  scope                = azurerm_storage_account.lake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_id         = azurerm_user_assigned_identity.pipeline.principal_id
  principal_type       = "ServicePrincipal"
}
""")
write('adf.tf', """\
resource "azurerm_data_factory" "adf" {
  name                            = local.names.adf
  resource_group_name             = azurerm_resource_group.data.name
  location                        = var.location
  public_network_enabled          = false
  managed_virtual_network_enabled = true
  tags                            = local.tags

  identity {
    type         = "UserAssigned"
    identity_ids = [azurerm_user_assigned_identity.pipeline.id]
  }

  dynamic "github_configuration" { # only dev is connected to Git
    for_each = var.environment == "dev" ? [1] : []
    content {
      account_name    = "sales-data-org"
      repository_name = "adf-sales"
      branch_name     = "main"
      root_folder     = "/adf"
    }
  }
}
""")
write('databricks.tf', """\
resource "azurerm_databricks_workspace" "dbw" {
  name                        = local.names.dbw
  resource_group_name         = azurerm_resource_group.data.name
  location                    = var.location
  sku                         = "premium" # needed for Unity Catalog
  managed_resource_group_name = "rg-${local.base}-dbw-managed"
  tags                        = local.tags

  custom_parameters { # VNet injection: clusters reach the private lake
    no_public_ip        = true
    virtual_network_id  = var.dbw_vnet.id
    public_subnet_name  = var.dbw_vnet.host_subnet
    private_subnet_name = var.dbw_vnet.container_subnet

    public_subnet_network_security_group_association_id  = var.dbw_vnet.host_nsg_assoc
    private_subnet_network_security_group_association_id = var.dbw_vnet.container_nsg_assoc
  }
}
""")
write('access_connector.tf', """\
resource "azurerm_databricks_access_connector" "uc" {
  name                = "dbac-${local.base}"
  resource_group_name = azurerm_resource_group.data.name
  location            = var.location
  tags                = local.tags

  identity {
    type = "SystemAssigned"
  }
}

resource "azurerm_role_assignment" "uc_lake" {
  scope                = azurerm_storage_account.lake.id
  role_definition_name = "Storage Blob Data Contributor"
  principal_type       = "ServicePrincipal"
  principal_id = (
    azurerm_databricks_access_connector.uc.identity[0].principal_id
  )
}
""")
write('unity.tf', """\
locals {
  lake_dfs = "${azurerm_storage_account.lake.name}.dfs.core.windows.net"
}

resource "databricks_storage_credential" "lake" {
  name = "sc-${local.base}"

  azure_managed_identity {
    access_connector_id = azurerm_databricks_access_connector.uc.id
  }
}

resource "databricks_external_location" "curated" {
  name            = "curated"
  url             = "abfss://curated@${local.lake_dfs}/"
  credential_name = databricks_storage_credential.lake.name
  depends_on      = [azurerm_role_assignment.uc_lake]
}
""")
write('tests/mocks/azurerm.tfmock.hcl', """\
# Mocked azurerm: realistic IDs, because the provider still
# validates the values that reach its arguments.
mock_data "azurerm_client_config" {
  defaults = {
    tenant_id = "11111111-1111-1111-1111-111111111111"
  }
}

mock_resource "azurerm_resource_group" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev"
  }
}

mock_resource "azurerm_storage_account" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.Storage/storageAccounts/stsalesdatadev"
  }
}

mock_resource "azurerm_key_vault" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.KeyVault/vaults/kv-sales-dev"
  }
}

mock_resource "azurerm_user_assigned_identity" {
  defaults = {
    id           = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.ManagedIdentity/userAssignedIdentities/id-sales-data-dev-pipeline"
    principal_id = "22222222-2222-2222-2222-222222222222"
  }
}

mock_resource "azurerm_databricks_access_connector" {
  defaults = {
    id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.Databricks/accessConnectors/dbac-sales-data-dev"
  }
}

mock_resource "azurerm_databricks_workspace" {
  defaults = {
    id            = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-sales-data-dev/providers/Microsoft.Databricks/workspaces/dbw-sales-data-dev"
    workspace_url = "adb-1234567890123456.7.azuredatabricks.net"
  }
}
""")
write('tests/terraform.tfvars', """\
environment       = "dev"
project           = "sales"
cost_center       = "cc-4410"
private_subnet_id = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev/subnets/snet-private-endpoints"
private_dns_zone_ids = {
  blob = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/privateDnsZones/privatelink.blob.core.windows.net"
  dfs  = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/privateDnsZones/privatelink.dfs.core.windows.net"
}
dbw_vnet = {
  id                  = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev"
  host_subnet         = "snet-dbw-host"
  container_subnet    = "snet-dbw-container"
  host_nsg_assoc      = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev/subnets/snet-dbw-host"
  container_nsg_assoc = "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/rg-network-dev/providers/Microsoft.Network/virtualNetworks/vnet-data-dev/subnets/snet-dbw-container"
}
""")
write('tests/lake.tftest.hcl', """\
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "lake_is_private_and_keyless" {
  command = plan

  assert {
    condition     = azurerm_storage_account.lake.is_hns_enabled
    error_message = "The lake needs the hierarchical namespace."
  }

  assert {
    condition     = !azurerm_storage_account.lake.shared_access_key_enabled
    error_message = "Shared keys must be off: Entra ID only."
  }

  assert {
    condition     = azurerm_storage_account.lake.public_network_access == "Disabled"
    error_message = "The lake must not be reachable from the internet."
  }
}
""")
write('tests/secrets.tftest.hcl', """\
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "no_password_in_state" {
  command = apply

  assert {
    condition     = azurerm_key_vault_secret.sql_admin.value == null
    error_message = "The secret value must not be stored in state."
  }

  assert {
    condition     = azurerm_mssql_server.meta.administrator_login_password == null
    error_message = "The SQL password must not be stored in state."
  }
}
""")
write('tests/access.tftest.hcl', """\
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "pipeline_identity_gets_the_lake_only" {
  command = apply

  assert {
    condition     = azurerm_role_assignment.pipeline_lake.scope == azurerm_storage_account.lake.id
    error_message = "Scope the role to the lake, not the subscription."
  }

  assert {
    condition     = azurerm_role_assignment.pipeline_lake.principal_id == azurerm_user_assigned_identity.pipeline.principal_id
    error_message = "The role must go to the pipeline identity."
  }

  assert {
    condition     = contains(azurerm_data_factory.adf.identity[0].identity_ids, azurerm_user_assigned_identity.pipeline.id)
    error_message = "Data Factory must run as the pipeline identity."
  }
}
""")
write('tests/databricks.tftest.hcl', """\
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "unity_catalog_points_at_the_lake" {
  command = plan

  assert {
    condition     = try(azurerm_databricks_workspace.dbw.custom_parameters[0].virtual_network_id, null) == var.dbw_vnet.id
    error_message = "The workspace must be injected into our VNet."
  }

  assert {
    condition     = databricks_external_location.curated.url == "abfss://curated@stsalesdatadev.dfs.core.windows.net/"
    error_message = "The external location must point at the curated layer."
  }
}
""")
write('tests/naming.tftest.hcl', """\
mock_provider "azurerm" {
  source = "./tests/mocks"
}

mock_provider "databricks" {}

run "names_follow_the_standard" {
  command = plan

  assert {
    condition     = azurerm_storage_account.lake.name == "stsalesdatadev"
    error_message = "The storage account name breaks the standard."
  }

  assert {
    condition     = azurerm_data_factory.adf.name == "adf-sales-data-dev"
    error_message = "The Data Factory name breaks the standard."
  }
}

run "every_resource_has_a_cost_center" {
  command = plan

  assert {
    condition = alltrue([for t in [
      azurerm_resource_group.data.tags,
      azurerm_storage_account.lake.tags,
      azurerm_key_vault.kv.tags,
      azurerm_data_factory.adf.tags,
      azurerm_databricks_workspace.dbw.tags,
    ] : t["cost_center"] == "cc-4410"])
    error_message = "Every resource needs the cost_center tag."
  }
}

run "rejects_a_bad_project_code" {
  command = plan

  variables {
    project = "Sales-Data"
  }

  expect_failures = [var.project]
}
""")
run('terraform fmt -check -recursive')
run('terraform init -backend=false')

# Step 1: Validate the whole dev platform against azurerm 5.11 and databricks 1.140 (no Azure calls)
run('terraform validate')

# Step 2: The same lake with a 4.x-style argument copied from an old example
write('lake.tf', """\
resource "azurerm_storage_account" "lake" {
  name                              = local.names.lake
  resource_group_name               = azurerm_resource_group.data.name
  location                          = var.location
  account_tier                      = "Standard"
  account_replication_type          = "ZRS"
  is_hns_enabled                    = true # ADLS Gen2
  shared_access_key_enabled         = false
  public_network_access_enabled     = false # 4.x style
  min_tls_version                   = "TLS1_2"
  infrastructure_encryption_enabled = true
  tags                              = local.tags

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
  }

  blob_properties {
    delete_retention_policy {
      days = 14
    }
    container_delete_retention_policy {
      days = 14
    }
  }
}
""")
run('terraform validate')

# Step 3: Back to the 5.x argument; a mocked test checks the lake settings in the plan
write('lake.tf', """\
resource "azurerm_storage_account" "lake" {
  name                              = local.names.lake
  resource_group_name               = azurerm_resource_group.data.name
  location                          = var.location
  account_tier                      = "Standard"
  account_replication_type          = "ZRS"
  is_hns_enabled                    = true # ADLS Gen2
  shared_access_key_enabled         = false
  public_network_access             = "Disabled"
  min_tls_version                   = "TLS1_2"
  infrastructure_encryption_enabled = true
  tags                              = local.tags

  network_rules {
    default_action = "Deny"
    bypass         = ["AzureServices"]
  }

  blob_properties {
    delete_retention_policy {
      days = 14
    }
    container_delete_retention_policy {
      days = 14
    }
  }
}
""")
run('terraform test -filter=tests/lake.tftest.hcl')

# Step 4: An ephemeral value in a normal argument: validate refuses
write('sql.tf', """\
# generated during the run: never in Git, tfvars, plan or state
ephemeral "random_password" "sql_admin" {
  length = 32
}

resource "azurerm_key_vault_secret" "sql_admin" {
  name             = "sql-admin-password"
  key_vault_id     = azurerm_key_vault.kv.id
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = 1 # bump to rotate
}

resource "azurerm_mssql_server" "meta" {
  name                          = local.names.sql
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  version                       = "12.0"
  administrator_login           = "sqladmin"
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false
  tags                          = local.tags

  administrator_login_password = (
    ephemeral.random_password.sql_admin.result
  )
}
""")
run('terraform validate', expect_error=True)   # EXPECTED TO FAIL: the video shows this error

# Step 5: Write-only arguments: validate passes, and a mocked apply finds no password in state
write('sql.tf', """\
# generated during the run: never in Git, tfvars, plan or state
ephemeral "random_password" "sql_admin" {
  length = 32
}

resource "azurerm_key_vault_secret" "sql_admin" {
  name             = "sql-admin-password"
  key_vault_id     = azurerm_key_vault.kv.id
  value_wo         = ephemeral.random_password.sql_admin.result
  value_wo_version = 1 # bump to rotate
}

resource "azurerm_mssql_server" "meta" {
  name                          = local.names.sql
  resource_group_name           = azurerm_resource_group.data.name
  location                      = var.location
  version                       = "12.0"
  administrator_login           = "sqladmin"
  minimum_tls_version           = "1.2"
  public_network_access_enabled = false
  tags                          = local.tags

  administrator_login_password_wo_version = 1 # bump to rotate
  administrator_login_password_wo = (
    ephemeral.random_password.sql_admin.result
  )
}
""")
run('terraform validate')
run('terraform test -filter=tests/secrets.tftest.hcl')

# Step 6: A mocked apply checks the identity and role wiring
run('terraform test -filter=tests/access.tftest.hcl')

# Step 7: A mocked plan checks that Unity Catalog points at the curated layer
run('terraform test -filter=tests/databricks.tftest.hcl')

# Step 8: How many Data Factory resource types azurerm 5.11 has
run('terraform providers schema -json > schema.json')
run("jq -r '.provider_schemas[].resource_schemas | keys[]' schema.json > types.txt")
run("grep -c '^azurerm_data_factory' types.txt")
run("grep -E '^azurerm_data_factory_(pipeline|trigger_schedule|dataset_parquet)$' types.txt")

# Step 9: The naming and tagging tests
run('terraform test -filter=tests/naming.tftest.hcl')

# Step 10: Does any provider have a provider-level tags setting?
run('jq \'[.provider_schemas[].provider.block | .attributes, .block_types | objects | keys[]] | any(test("tags"))\' schema.json')

# Step 11: Day five: the pull request checks for the whole platform
run('terraform fmt -check -recursive')
run('terraform validate')
run('terraform test')
Next in this series

Part 9: CI/CD, Testing and Policy

The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run with Terraform 1.17.0; IDs, timestamps and durations are masked. Azure and Databricks code is checked with terraform validate. Part of the Real Time Terraform Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Terraform documentation (developer.hashicorp.com/terraform) for anything you quote in an interview.

No comments:

Post a Comment

Note: Only a member of this blog may post a comment.

Subscribe to: Post Comments (Atom)