Terraform Interview Questions and Answers Part 10 - Real-World Manager Scenarios

Terraform Interview Q&A · Part 10 of 10

In this post we are going to cover real-world manager scenarios with Terraform. These are 7 real Terraform interview questions for data engineers, asked the way they come up on a real project, and the last one is a scenario from a final round: how would you run Terraform for fifty engineers across many teams?

These are the questions where the interviewer wants to know what you do when things go wrong: a portal fix at 2 AM, a plan full of destroys, a stuck lock, a leaked secret. The answer is almost always the same: read before you act, and put the fix into code.

Part 10 of the Real Time Terraform Interview Questions and Answers series on TechBrothersIT.

On this page
  1. The short answer
  2. Let's set up the project
  3. Question 1: Someone fixed an outage in the portal at 2 AM and now the plan wants to undo it. What do you do in the morning?
  4. Question 2: A plan shows twenty resources to destroy and the change was only supposed to add a tag. How do you investigate?
  5. Question 3: The state is locked and the person who ran apply has left for the day. How do you handle a stuck lock?
  6. Question 4: Leadership asks whether to use Terraform, OpenTofu, Bicep or Pulumi. How do you decide?
  7. Question 5: One state file now has eight hundred resources and every plan takes fifteen minutes. How do you split it?
  8. Question 6: A secret was committed to Git and is also in the state. What do you do right now?
  9. Scenario 7: In the final round, the hiring manager asks how you would run Terraform for fifty engineers across many teams. What is your design?
  10. The full script

The short answer

Say this first

In an incident, read before you act: plan -refresh-only for drift, the reason lines and action_reason for destroys, the lock info before force-unlock. Fix with code: put hand fixes into code, moved blocks for renames, import + removed blocks to split state. A leaked secret is rotated first, then moved to Key Vault with ephemeral values and write-only arguments.

Let's set up the project

We use a small Terraform 1.17 project with one folder per scenario. Here we use local resources, local_file, terraform_data, random_string and time_sleep, plus a tiny local HTTP state server that stands in for the Azure blob. So every plan and apply is real and nothing is created in Azure. The Azure code in the secure folder is checked with terraform validate. Here are the starting files.

pipeline/main.tf
terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
  }
}

resource "local_file" "pipeline" {
  filename = "${path.module}/config/nightly-load.json"
  content = jsonencode({
    workers         = 4
    timeout_minutes = 60
  })
}
modules/lake/main.tf
variable "environment" {
  type = string
}

variable "tags" {
  type    = map(string)
  default = {}
}

locals {
  containers = toset(["landing", "raw", "curated", "gold"])
}

resource "terraform_data" "container" {
  for_each = local.containers
  input = {
    name = "${each.key}-${var.environment}"
    tags = var.tags
  }
}
lake/main.tf
module "lake" {
  source      = "../modules/lake"
  environment = "prod"
}
locked/main.tf
terraform {
  # a local HTTP state server stands in for the Azure blob
  backend "http" {
    address        = "http://127.0.0.1:18710/state"
    lock_address   = "http://127.0.0.1:18710/state"
    unlock_address = "http://127.0.0.1:18710/state"
  }
  required_providers {
    time = {
      source  = "hashicorp/time"
      version = "~> 0.14"
    }
  }
}

# stands in for a long nightly apply
resource "time_sleep" "nightly_load" {
  create_duration = "60s"
}
locked/state_server.py
# A tiny HTTP state backend with locking (Terraform's "http" backend protocol).
# It stands in for the Azure blob: a lock stays until someone unlocks it.
import os
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer

STATE, LOCK = 'remote/state.json', 'remote/lock.json'


class Handler(BaseHTTPRequestHandler):
    def body(self):
        return self.rfile.read(int(self.headers.get('Content-Length', 0)))

    def reply(self, code, data=b''):
        self.send_response(code)
        self.send_header('Content-Length', str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        if not os.path.exists(STATE):
            return self.reply(404)
        self.reply(200, open(STATE, 'rb').read())

    def do_POST(self):
        open(STATE, 'wb').write(self.body())
        self.reply(200)

    def do_LOCK(self):
        info = self.body()
        if os.path.exists(LOCK):       # already locked: return the holder
            return self.reply(423, open(LOCK, 'rb').read())
        open(LOCK, 'wb').write(info)
        self.reply(200)

    def do_UNLOCK(self):
        if os.path.exists(LOCK):
            os.remove(LOCK)
        self.reply(200)

    def log_message(self, *args):
        pass


os.makedirs('remote', exist_ok=True)
HTTPServer(('127.0.0.1', int(sys.argv[1])), Handler).serve_forever()
platform/versions.tf
terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}
platform/main.tf
resource "terraform_data" "network" {
  for_each = toset(["hub", "spoke-data"])
  input    = "vnet-sales-${each.key}"
}

# stands in for the lake resources
resource "random_string" "lake" {
  for_each = toset(["landing", "raw", "curated"])
  length   = 6
}
lake-stack/versions.tf
terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}
lake-stack/main.tf
locals {
  # IDs from the old state (terraform state show)
  lake_ids = {
    landing = "lnd4x7"
    raw     = "raw9k2"
    curated = "cur3m8"
  }
}

import {
  for_each = local.lake_ids
  to       = random_string.lake[each.key]
  id       = each.value
}

resource "random_string" "lake" {
  for_each = local.lake_ids
  length   = 6
}
secrets/main.tf
variable "sql_admin_password" {
  type      = string
  sensitive = true
}

resource "terraform_data" "sql_login" {
  input = {
    login    = "sqladmin"
    password = var.sql_admin_password
  }
}
secrets/terraform.tfvars
sql_admin_password = "Winter-2026-sales"
secure/versions.tf
terraform {
  required_version = ">= 1.17.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 5.11"
    }
  }
}

provider "azurerm" {
  features {}
}
secure/main.tf
variable "key_vault_id" {
  type = string
}

# read at plan and apply time, never stored in plan or state
ephemeral "azurerm_key_vault_secret" "pg_admin" {
  name         = "pg-admin-password"
  key_vault_id = var.key_vault_id
}

locals {
  pg_password = ephemeral.azurerm_key_vault_secret.pg_admin.value
}

resource "azurerm_postgresql_flexible_server" "sales" {
  name                = "psql-sales-prod"
  resource_group_name = "rg-sales-data-prod"
  location            = "westeurope"
  version             = "16"
  sku_name            = "GP_Standard_D2ds_v5"
  storage_mb          = 65536

  administrator_login               = "pgadmin"
  administrator_password_wo         = local.pg_password
  administrator_password_wo_version = 2
}

Question 1: Someone fixed an outage in the portal at 2 AM and now the plan wants to undo it. What do you do in the morning?

First, stop any apply. Then find out exactly what changed, and if the fix is right, put it into the code.

Let's play the 2 AM fix. In the pipeline folder, the on-call engineer edits the config by hand: 8 workers and 120 minutes. Then we run terraform plan -refresh-only, which shows what changed outside Terraform and proposes no actions.

Terminal
$ printf '{"timeout_minutes":120,"workers":8}' > config/nightly-load.json
$ terraform plan -refresh-only -input=false
local_file.pipeline: Refreshing state... [id=abe37c46e64da32ec08f0f6106d0f95f534b4726]

Note: Objects have changed outside of Terraform

Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:

  # local_file.pipeline has been deleted
  - resource "local_file" "pipeline" {
      - content              = jsonencode(
            {
              - timeout_minutes = 60
              - workers         = 4
            }
        ) -> null
      - content_base64sha256 = "lP5ZaXcQ5MUf2JLkzvGpidGPPGEfw5wA/dO9zUpQPow=" -> null
      - content_base64sha512 =
        "fwKFUbyS0Pq3L5HHTf+zOwpuip26MQnD4W+kYjDiBQD1nNDOCzCroU18vYG+t5eWmQ/auh0sRkmqraSXSIdFKQ==" ->
          null
      - content_md5          = "b2d6fc126b136d83eff8f4e69aa32097" -> null
      - content_sha1         = "abe37c46e64da32ec08f0f6106d0f95f534b4726" -> null
      - content_sha256       = "94fe59697710e4c51fd892e4cef1a989d18f3c611fc39c00fdd3bdcd4a503e8c" ->
        null
      - content_sha512       =
        "7f028551bc92d0fab72f91c74dffb33b0a6e8a9dba3109c3e16fa46230e20500f59cd0ce0b30aba14d7cbd81beb7979
          6990fdaba1d2c4649aaada49748874529" -> null
      - directory_permission = "0777" -> null
      - file_permission      = "0777" -> null
      - filename             = "./config/nightly-load.json" -> null
      - id                   = "abe37c46e64da32ec08f0f6106d0f95f534b4726" -> null
    }

This is a refresh-only plan, so Terraform will not take any actions to undo
these. If you were expecting these changes then you can apply this plan to
record the updated values in the Terraform state without changing any remote
objects.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

The local provider sees an edited file as a different object, so it says local_file.pipeline has been deleted. The old values were workers = 4 and timeout_minutes = 60.

Now a normal terraform plan:

Terminal
$ terraform plan -input=false
local_file.pipeline: Refreshing state... [id=abe37c46e64da32ec08f0f6106d0f95f534b4726]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_file.pipeline will be created
  + resource "local_file" "pipeline" {
      + content              = jsonencode(
            {
              + timeout_minutes = 60
              + workers         = 4
            }
        )
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "./config/nightly-load.json"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

It would write the file back with workers = 4 and timeout_minutes = 60. That is the plan undoing the 2 AM fix. If applies run on a schedule, pause them first.

So the fix goes into the code, in a pull request: 8 workers and 120 minutes. We plan, apply and plan again.

pipeline/main.tf
terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
  }
}

resource "local_file" "pipeline" {
  filename = "${path.module}/config/nightly-load.json"
  content = jsonencode({
    workers         = 8
    timeout_minutes = 120
  })
}
Terminal
$ terraform plan -input=false
local_file.pipeline: Refreshing state... [id=abe37c46e64da32ec08f0f6106d0f95f534b4726]

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # local_file.pipeline will be created
  + resource "local_file" "pipeline" {
      + content              = jsonencode(
            {
              + timeout_minutes = 120
              + workers         = 8
            }
        )
      + content_base64sha256 = (known after apply)
      + content_base64sha512 = (known after apply)
      + content_md5          = (known after apply)
      + content_sha1         = (known after apply)
      + content_sha256       = (known after apply)
      + content_sha512       = (known after apply)
      + directory_permission = "0777"
      + file_permission      = "0777"
      + filename             = "./config/nightly-load.json"
      + id                   = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.
$ terraform apply -auto-approve -input=false | tail -n 1
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
$ terraform plan -input=false
local_file.pipeline: Refreshing state... [id=2906909d02afe03b8367a6d196567a3d7f062f03]

No changes. Your infrastructure matches the configuration.

Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

The local file is rewritten once, then No changes. A real Azure resource would go straight to no changes.

Real project

Ask why someone needed the portal. Give on-call a break glass path that always ends in a pull request. And if a setting is really owned outside Terraform, use ignore_changes for it.

Question 2: A plan shows twenty resources to destroy and the change was only supposed to add a tag. How do you investigate?

Don't approve it. A tag change is normally an update in place, so destroys mean something else changed.

Here the pull request adds a cost_center tag, and also renames the module block from lake to datalake. We run terraform init and save a plan.

lake/main.tf
module "datalake" {
  source      = "../modules/lake"
  environment = "prod"
  tags        = { cost_center = "sales" }
}
Terminal
$ terraform init -input=false
Initializing the backend...

Initializing modules...
- datalake in ../modules/lake

Initializing provider plugins...
- terraform.io/builtin/terraform is built in to Terraform

Terraform has been successfully initialized!

You may now begin working with Terraform. Try running "terraform plan" to see
any changes that are required for your infrastructure. All Terraform commands
should now work.

If you ever set or change modules or backend configuration for Terraform,
rerun this command to reinitialize your working directory. If you forget, other
commands will detect it and remind you to do so if necessary.
$ terraform plan -refresh=false -input=false -out=tfplan
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
  - destroy

Terraform will perform the following actions:

  # module.datalake.terraform_data.container["curated"] will be created
  + resource "terraform_data" "container" {
      + id     = (known after apply)
      + input  = {
          + name = "curated-prod"
          + tags = {
              + cost_center = "sales"
            }
        }
      + output = (known after apply)
    }

  # module.datalake.terraform_data.container["gold"] will be created
  + resource "terraform_data" "container" {
      + id     = (known after apply)
      + input  = {
          + name = "gold-prod"
          + tags = {
              + cost_center = "sales"
            }
        }
      + output = (known after apply)
    }

  # module.datalake.terraform_data.container["landing"] will be created
  + resource "terraform_data" "container" {
      + id     = (known after apply)
      + input  = {
          + name = "landing-prod"
          + tags = {
              + cost_center = "sales"
            }
        }
      + output = (known after apply)
    }

  # module.datalake.terraform_data.container["raw"] will be created
  + resource "terraform_data" "container" {
      + id     = (known after apply)
      + input  = {
          + name = "raw-prod"
          + tags = {
              + cost_center = "sales"
            }
        }
      + output = (known after apply)
    }

  # module.lake.terraform_data.container["curated"] will be destroyed
  # (because terraform_data.container is not in configuration)
  - resource "terraform_data" "container" {
      - id     = "<uuid>" -> null
      - input  = {
          - name = "curated-prod"
          - tags = {}
        } -> null
      - output = {
          - name = "curated-prod"
          - tags = {}
        } -> null
    }

  # module.lake.terraform_data.container["gold"] will be destroyed
  # (because terraform_data.container is not in configuration)
  - resource "terraform_data" "container" {
      - id     = "<uuid>" -> null
      - input  = {
          - name = "gold-prod"
          - tags = {}
        } -> null
      - output = {
          - name = "gold-prod"
          - tags = {}
        } -> null
    }

  # module.lake.terraform_data.container["landing"] will be destroyed
  # (because terraform_data.container is not in configuration)
  - resource "terraform_data" "container" {
      - id     = "<uuid>" -> null
      - input  = {
          - name = "landing-prod"
          - tags = {}
        } -> null
      - output = {
          - name = "landing-prod"
          - tags = {}
        } -> null
    }

  # module.lake.terraform_data.container["raw"] will be destroyed
  # (because terraform_data.container is not in configuration)
  - resource "terraform_data" "container" {
      - id     = "<uuid>" -> null
      - input  = {
          - name = "raw-prod"
          - tags = {}
        } -> null
      - output = {
          - name = "raw-prod"
          - tags = {}
        } -> null
    }

Plan: 4 to add, 0 to change, 4 to destroy.

─────────────────────────────────────────────────────────────────────────────

Saved the plan to: tfplan

To perform exactly these actions, run the following command to apply:
    terraform apply "tfplan"

Read the reason under each address. Every container would be destroyed and created again, because terraform_data.container is not in configuration: Plan: 4 to add, 0 to change, 4 to destroy.

The plan JSON says it for every delete in action_reason. Let's pull that out with jq.

Terminal
$ terraform show -json tfplan | jq -r '.resource_changes[] | select(.change.actions | index("delete")) | "\(.address) \(.action_reason)"'
module.lake.terraform_data.container["curated"] delete_because_no_resource_config
module.lake.terraform_data.container["gold"] delete_because_no_resource_config
module.lake.terraform_data.container["landing"] delete_because_no_resource_config
module.lake.terraform_data.container["raw"] delete_because_no_resource_config

Every delete is delete_because_no_resource_config. With twenty destroys, that is how I group them by cause.

The fix is a moved block from module.lake to module.datalake. We plan again.

lake/main.tf
module "datalake" {
  source      = "../modules/lake"
  environment = "prod"
  tags        = { cost_center = "sales" }
}

moved {
  from = module.lake
  to   = module.datalake
}
Terminal
$ terraform plan -refresh=false -input=false
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place

Terraform will perform the following actions:

  # module.datalake.terraform_data.container["curated"] will be updated in-place
  # (moved from module.lake.terraform_data.container["curated"])
  ~ resource "terraform_data" "container" {
        id     = "<uuid>"
      ~ input  = {
            name = "curated-prod"
          ~ tags = {
              + cost_center = "sales"
            }
        }
      ~ output = {
          - name = "curated-prod"
          - tags = {}
        } -> (known after apply)
    }

  # module.datalake.terraform_data.container["gold"] will be updated in-place
  # (moved from module.lake.terraform_data.container["gold"])
  ~ resource "terraform_data" "container" {
        id     = "<uuid>"
      ~ input  = {
            name = "gold-prod"
          ~ tags = {
              + cost_center = "sales"
            }
        }
      ~ output = {
          - name = "gold-prod"
          - tags = {}
        } -> (known after apply)
    }

  # module.datalake.terraform_data.container["landing"] will be updated in-place
  # (moved from module.lake.terraform_data.container["landing"])
  ~ resource "terraform_data" "container" {
        id     = "<uuid>"
      ~ input  = {
            name = "landing-prod"
          ~ tags = {
              + cost_center = "sales"
            }
        }
      ~ output = {
          - name = "landing-prod"
          - tags = {}
        } -> (known after apply)
    }

  # module.datalake.terraform_data.container["raw"] will be updated in-place
  # (moved from module.lake.terraform_data.container["raw"])
  ~ resource "terraform_data" "container" {
        id     = "<uuid>"
      ~ input  = {
            name = "raw-prod"
          ~ tags = {
              + cost_center = "sales"
            }
        }
      ~ output = {
          - name = "raw-prod"
          - tags = {}
        } -> (known after apply)
    }

Plan: 0 to add, 4 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Now the plan is what the pull request promised: four updates in place, each moved from module.lake..., and 0 to destroy.

Other usual causes:

  • forces replacement: an argument that cannot be updated in place
  • index [N] is out of range for count: a list shifted
  • A provider upgrade: read the upgrade guide

Or the wrong state.

Pitfall

Never hide it with -target, and make the pipeline ask for extra approval when a plan has deletes.

Question 3: The state is locked and the person who ran apply has left for the day. How do you handle a stuck lock?

Read the lock info, make sure the run that holds it is really dead, and only then force-unlock with that lock ID.

In the locked folder, a small local HTTP state server stands in for the Azure blob. Let's start an apply and kill it with kill -9, the way a pipeline agent dies, so it never releases its lock.

Terminal
$ terraform apply -auto-approve -input=false > apply.log 2>&1 & echo $! > apply.pid
$ sleep 5; kill -9 $(cat apply.pid)
$ tail -n 1 apply.log
time_sleep.nightly_load: Creating...

The last log line is time_sleep.nightly_load: Creating.... Now we run the next plan. We hide the Created: line, because that time changes every run.

Terminal
$ terraform plan -input=false > locked.txt 2>&1; echo "exit code: $?"
exit code: 1
$ grep -v "Created:" locked.txt
Error: Error acquiring the state lock

Error message: HTTP remote state already locked:
ID=<uuid>
Lock Info:
  ID:        <uuid>
  Path:
  Operation: OperationTypeApply
  Who:       root@vm
  Version:   1.17.0
  Info:

Terraform acquires a state lock to protect the state from being written
by multiple users at the same time. Please resolve the issue above and try
again. For most commands, you can disable locking with the "-lock=false"
flag, but this is not recommended.

It fails with exit code: 1 and Error acquiring the state lock. The lock info shows the ID, the operation, OperationTypeApply, who held it, and the Terraform version, 1.17.0.

The steps are:

  • Read the lock info: ID, operation, who, when
  • Make sure that run is dead: pipeline run, agent, or the person
  • terraform force-unlock LOCK_ID: never -lock=false, never delete the state

The run is dead, so we force-unlock with that ID and plan again.

Terminal
$ terraform force-unlock -force "$(sed -n 's/^  ID: *//p' locked.txt)"
Terraform state has been successfully unlocked!

The state has been unlocked, and Terraform commands should now be able to
obtain a new lock on the remote state.
$ terraform plan -input=false
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create

Terraform will perform the following actions:

  # time_sleep.nightly_load will be created
  + resource "time_sleep" "nightly_load" {
      + create_duration = "60s"
      + id              = (known after apply)
    }

Plan: 1 to add, 0 to change, 0 to destroy.

─────────────────────────────────────────────────────────────────────────────

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Terraform state has been successfully unlocked!, and the next plan works. The killed apply may not have saved what it created, so in Azure, look for objects that are missing from state and import them.

Azure

With the azurerm backend, the lock is a lease on the state blob, so a crashed agent leaves it locked. Use force-unlock only when the run is dead; breaking the lease yourself is the last resort.

Question 4: Leadership asks whether to use Terraform, OpenTofu, Bicep or Pulumi. How do you decide?

Start from the platform, not the tool. For an Azure data platform with Databricks, I'd pick Terraform or OpenTofu: one language and one workflow for Azure, Databricks, and everything around them.

  • Terraform is under the Business Source License since 1.6, and IBM owns HashiCorp. Internal production use is fine; it restricts competing hosted offerings.
  • OpenTofu is the open source fork: MPL 2.0, CNCF Sandbox, the tofu CLI, and extras like state encryption.
  • Bicep is Microsoft's language for Azure only. It compiles to ARM templates and keeps no state. Good if everything is Azure, but it doesn't manage what's inside Databricks, like catalogs and jobs.
  • Pulumi uses general-purpose languages, like Python and TypeScript, under Apache 2.0, and keeps state like Terraform. A fit for teams who want real code and unit tests.
TerraformOpenTofuBicepPulumi
LicenseBUSL 1.1 (since 1.6)MPL 2.0MITApache 2.0
OwnerIBM (HashiCorp)Linux Foundation, CNCFMicrosoftPulumi
Reachany providerany providerAzure onlymany clouds
Statestate file + backendstate file, can encryptno statestate, like Terraform
LanguageHCLHCL (CLI: tofu)Bicep, compiles to ARMPython, TypeScript, ...
ExtrasHCP Terraform, Stackscommunity roadmapnative to Azuretests in your language
My answer to leadership

Decide on skills, clouds, license and support, then run a small pilot. And CDK for Terraform is no longer an option: it was archived in December 2025.

Question 5: One state file now has eight hundred resources and every plan takes fifteen minutes. How do you split it?

Split by lifecycle and owner: network, lake, compute, each with its own state, so one change doesn't plan eight hundred resources. It's a refactoring, so nothing may be destroyed.

Plans are slow because every plan refreshes every object. Let's count the refresh reads in the platform folder, first with a normal plan and then with -minimal-refresh.

Terminal
$ terraform plan -input=false | grep "Refreshing state" | wc -l
5
$ terraform plan -minimal-refresh -input=false | grep "Refreshing state" | wc -l
0

A normal plan does 5 refresh reads. Terraform 1.17 adds -minimal-refresh, which skips objects with no planned change: 0. Faster, but you may plan against a stale record of the real objects. It is a shortcut, not the fix.

The fix is to move the lake into its own state. In lake-stack, one import block with for_each over the IDs from the old state brings the lake in. With azurerm, the ID is the Azure resource ID.

Terminal
$ terraform plan -input=false -out=tfplan | grep -E "# |Plan:"
  # random_string.lake["curated"] will be imported
  # random_string.lake["landing"] will be imported
  # random_string.lake["raw"] will be imported
Plan: 3 to import, 0 to add, 0 to change, 0 to destroy.
$ terraform apply -input=false tfplan | tail -n 1
Apply complete! Resources: 3 imported, 0 added, 0 changed, 0 destroyed.

Plan: 3 to import, 0 to add, 0 to change, 0 to destroy., then 3 imported.

Then the old state lets go, with a removed block and destroy = false.

platform/main.tf
resource "terraform_data" "network" {
  for_each = toset(["hub", "spoke-data"])
  input    = "vnet-sales-${each.key}"
}

# the lake moved to its own state: forget it here, keep it running
removed {
  from = random_string.lake
  lifecycle {
    destroy = false
  }
}
Terminal
$ terraform plan -refresh=false -input=false -out=tfplan | grep -E "not be destroyed|is set|Plan:"
 # random_string.lake["curated"] will no longer be managed by Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)
 # random_string.lake["landing"] will no longer be managed by Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)
 # random_string.lake["raw"] will no longer be managed by Terraform, but will not be destroyed
 # (destroy = false is set in the configuration)
Plan: 0 to add, 0 to change, 0 to destroy.
$ terraform apply -input=false tfplan
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Each lake object will no longer be managed by Terraform, but will not be destroyed, and the plan has 0 to destroy. Let's check both states.

Terminal
$ terraform -chdir=platform plan -input=false | grep "No changes"
No changes. Your infrastructure matches the configuration.
$ terraform -chdir=lake-stack plan -input=false | grep "No changes"
No changes. Your infrastructure matches the configuration.

Both say No changes.

Real project

I do it in one change window, one slice at a time, and connect the states with outputs and data sources instead of one big state.

Question 6: A secret was committed to Git and is also in the state. What do you do right now?

Treat it as compromised and rotate it first: a new password in the database and in Key Vault. Cleanup comes second, because the old value is already in clones, the state, and old state versions.

Deleting the file is not enough. In the secrets folder, let's git rm the tfvars file, then look at the previous commit and search the state.

Terminal
$ git rm -q terraform.tfvars && git commit -qm "Remove tfvars"
$ git show HEAD~1:terraform.tfvars
sql_admin_password = "Winter-2026-sales"
$ grep -n "Winter" terraform.tfstate
21:                "password": "Winter-2026-sales"
34:                "password": "Winter-2026-sales"

git show HEAD~1:terraform.tfvars still shows Winter-2026-sales. And the state file holds it in plain text, twice, on lines 21 and 34, even though the variable is marked sensitive.

So, in order:

  • Rotate it now: database and Key Vault, a new value
  • Check where it was used: apps, pipelines, linked services
  • Purge the Git history: everyone re-clones
  • Restrict the state: and its old versions and snapshots

Then fix the pattern. In secure/main.tf, the password lives only in Key Vault. An ephemeral azurerm_key_vault_secret reads it at plan and apply time, and the write-only argument administrator_password_wo sends it to PostgreSQL without storing it. Bump administrator_password_wo_version to send a new one. We validate it against azurerm 5.11.

Terminal
$ terraform validate
Success! The configuration is valid.

Success! The configuration is valid.

Prevent the next one

Turn on secret scanning on the repository and in the pipeline, and keep secrets out of tfvars files, ever.

Scenario 7: In the final round, the hiring manager asks how you would run Terraform for fifty engineers across many teams. What is your design?

A platform team owns the paved road, and product teams own their stacks.

Let's build it up one layer at a time:

  • Platform team: modules and a private registry
  • Product teams: small states, one per environment
  • One pipeline: plan on the pull request, approved apply
  • Azure and Databricks: an OIDC identity per environment

Shared modules live in a private registry with versions. Every change goes through one pipeline: a plan on the pull request, a review, an approved apply, and no standing write access for people. Guardrails are code: tests and policy checks in the pipeline, Azure Policy as the safety net, and a nightly drift job on every state.

HCP Terraform or Terraform Enterprise gives you much of this built in: projects, workspaces, policy sets, dynamic credentials, and Stacks. Or build it on GitHub Actions or Azure DevOps with the azurerm backend.

Then the people side: CODEOWNERS per folder, a changelog for every module, planned provider upgrades, and a runbook for locks, drift and leaked secrets, the scenarios from this post.

In one line

Paved road, one pipeline, guardrails as code, and runbooks for the bad days.

Key points to remember

  • Drift: plan -refresh-only, then put the fix into code
  • Unexpected destroys: read the reasons and action_reason; fix with moved
  • Stuck lock: read the lock info, confirm the run is dead, force-unlock
  • Split state with import + removed (destroy = false): 0 to destroy
  • Leaked secret: rotate first; then Key Vault, ephemeral, write-only

The full script

Here is a Python script that writes all the files and runs every command in order. You need Terraform 1.17 on your PATH; run it in an empty folder.

Show the full script that writes the files and runs every step
Full script - writes the Terraform files and runs every step (Terraform 1.17 on your PATH), run it with Python
# ========================================================================
# Terraform Interview Q&A #10 - Real-world manager scenarios
# TechBrothersIT. Names and sample values are invented for teaching.
# Every step below was run with Terraform 1.17.0; the outputs in the video and the blog post are the real outputs.
# Providers used: hashicorp/random, null, local, time (they create nothing in any cloud). Azure and Databricks
# code in this series is checked with terraform validate only: nothing is created in a real subscription.
# To run it yourself (Linux, macOS or WSL; some steps use grep / cat):
#   1. Install Terraform 1.17 from developer.hashicorp.com/terraform/install and put it on your PATH.
#   2. python Terraform_Interview_QA_10_Code.py   in an empty folder (terraform init downloads the providers).
# ========================================================================

import os, subprocess

def write(path, text):
    if text is None:
        if os.path.exists(path): os.remove(path)
        return
    os.makedirs(os.path.dirname(path) or ".", exist_ok=True)
    with open(path, "w", encoding="utf-8") as f:
        f.write(text.strip("\n") + "\n")

def run(cmd, expect_error=False, allow_error=False, env=None):
    print("\n$ " + cmd, flush=True)
    rc = subprocess.run(cmd, shell=True, env=dict(os.environ, **(env or {}))).returncode
    if rc != 0 and not (expect_error or allow_error):
        raise SystemExit("stopped: this command was expected to succeed")

# Environment used by every step
os.environ.update({'GIT_AUTHOR_NAME': 'Data Platform', 'GIT_AUTHOR_EMAIL': 'data-platform@example.com', 'GIT_COMMITTER_NAME': 'Data Platform', 'GIT_COMMITTER_EMAIL': 'data-platform@example.com', 'GIT_AUTHOR_DATE': '2026-10-01T09:00:00Z', 'GIT_COMMITTER_DATE': '2026-10-01T09:00:00Z'})

# Step 0: the starting files
write('pipeline/main.tf', """\
terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
  }
}

resource "local_file" "pipeline" {
  filename = "${path.module}/config/nightly-load.json"
  content = jsonencode({
    workers         = 4
    timeout_minutes = 60
  })
}
""")
write('modules/lake/main.tf', """\
variable "environment" {
  type = string
}

variable "tags" {
  type    = map(string)
  default = {}
}

locals {
  containers = toset(["landing", "raw", "curated", "gold"])
}

resource "terraform_data" "container" {
  for_each = local.containers
  input = {
    name = "${each.key}-${var.environment}"
    tags = var.tags
  }
}
""")
write('lake/main.tf', """\
module "lake" {
  source      = "../modules/lake"
  environment = "prod"
}
""")
write('locked/main.tf', """\
terraform {
  # a local HTTP state server stands in for the Azure blob
  backend "http" {
    address        = "http://127.0.0.1:18710/state"
    lock_address   = "http://127.0.0.1:18710/state"
    unlock_address = "http://127.0.0.1:18710/state"
  }
  required_providers {
    time = {
      source  = "hashicorp/time"
      version = "~> 0.14"
    }
  }
}

# stands in for a long nightly apply
resource "time_sleep" "nightly_load" {
  create_duration = "60s"
}
""")
write('locked/state_server.py', """\
# A tiny HTTP state backend with locking (Terraform's "http" backend protocol).
# It stands in for the Azure blob: a lock stays until someone unlocks it.
import os
import sys
from http.server import BaseHTTPRequestHandler, HTTPServer

STATE, LOCK = 'remote/state.json', 'remote/lock.json'


class Handler(BaseHTTPRequestHandler):
    def body(self):
        return self.rfile.read(int(self.headers.get('Content-Length', 0)))

    def reply(self, code, data=b''):
        self.send_response(code)
        self.send_header('Content-Length', str(len(data)))
        self.end_headers()
        self.wfile.write(data)

    def do_GET(self):
        if not os.path.exists(STATE):
            return self.reply(404)
        self.reply(200, open(STATE, 'rb').read())

    def do_POST(self):
        open(STATE, 'wb').write(self.body())
        self.reply(200)

    def do_LOCK(self):
        info = self.body()
        if os.path.exists(LOCK):       # already locked: return the holder
            return self.reply(423, open(LOCK, 'rb').read())
        open(LOCK, 'wb').write(info)
        self.reply(200)

    def do_UNLOCK(self):
        if os.path.exists(LOCK):
            os.remove(LOCK)
        self.reply(200)

    def log_message(self, *args):
        pass


os.makedirs('remote', exist_ok=True)
HTTPServer(('127.0.0.1', int(sys.argv[1])), Handler).serve_forever()
""")
write('platform/versions.tf', """\
terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}
""")
write('platform/main.tf', """\
resource "terraform_data" "network" {
  for_each = toset(["hub", "spoke-data"])
  input    = "vnet-sales-${each.key}"
}

# stands in for the lake resources
resource "random_string" "lake" {
  for_each = toset(["landing", "raw", "curated"])
  length   = 6
}
""")
write('lake-stack/versions.tf', """\
terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.9"
    }
  }
}
""")
write('lake-stack/main.tf', """\
locals {
  # IDs from the old state (terraform state show)
  lake_ids = {
    landing = "lnd4x7"
    raw     = "raw9k2"
    curated = "cur3m8"
  }
}

import {
  for_each = local.lake_ids
  to       = random_string.lake[each.key]
  id       = each.value
}

resource "random_string" "lake" {
  for_each = local.lake_ids
  length   = 6
}
""")
write('secrets/main.tf', """\
variable "sql_admin_password" {
  type      = string
  sensitive = true
}

resource "terraform_data" "sql_login" {
  input = {
    login    = "sqladmin"
    password = var.sql_admin_password
  }
}
""")
write('secrets/terraform.tfvars', """\
sql_admin_password = "Winter-2026-sales"
""")
write('secure/versions.tf', """\
terraform {
  required_version = ">= 1.17.0"
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "~> 5.11"
    }
  }
}

provider "azurerm" {
  features {}
}
""")
write('secure/main.tf', """\
variable "key_vault_id" {
  type = string
}

# read at plan and apply time, never stored in plan or state
ephemeral "azurerm_key_vault_secret" "pg_admin" {
  name         = "pg-admin-password"
  key_vault_id = var.key_vault_id
}

locals {
  pg_password = ephemeral.azurerm_key_vault_secret.pg_admin.value
}

resource "azurerm_postgresql_flexible_server" "sales" {
  name                = "psql-sales-prod"
  resource_group_name = "rg-sales-data-prod"
  location            = "westeurope"
  version             = "16"
  sku_name            = "GP_Standard_D2ds_v5"
  storage_mb          = 65536

  administrator_login               = "pgadmin"
  administrator_password_wo         = local.pg_password
  administrator_password_wo_version = 2
}
""")
run('terraform fmt -check -recursive')
run('cd pipeline && terraform init -input=false && terraform apply -auto-approve -input=false')
run('cd lake && terraform init -input=false && terraform apply -auto-approve -input=false')
run('fuser -k -n tcp 18710 > /dev/null 2>&1; cd locked && (python3 state_server.py 18710 > server.log 2>&1 & echo $! > server.pid) && sleep 1 && terraform init -input=false')
run('cd platform && terraform init -input=false && terraform import -input=false \'random_string.lake["landing"]\' lnd4x7 && terraform import -input=false \'random_string.lake["raw"]\' raw9k2 && terraform import -input=false \'random_string.lake["curated"]\' cur3m8 && terraform apply -auto-approve -input=false')
run('cd lake-stack && terraform init -input=false')
run('cd secrets && git init -q -b main && git add main.tf terraform.tfvars && git commit -qm "Add the SQL login" && terraform init -input=false && terraform apply -auto-approve -input=false')
run('cd secure && terraform init -input=false')

# Step 1: At 2 AM the on-call engineer edits the config by hand; refresh-only shows the drift
run('cd pipeline && { printf \'{"timeout_minutes":120,"workers":8}\' > config/nightly-load.json; }')
run('cd pipeline && { terraform plan -refresh-only -input=false; }')

# Step 2: A normal plan would undo the 2 AM fix
run('cd pipeline && { terraform plan -input=false; }')

# Step 3: The fix goes into the code: plan, apply, and no changes
write('pipeline/main.tf', """\
terraform {
  required_providers {
    local = {
      source  = "hashicorp/local"
      version = "~> 2.9"
    }
  }
}

resource "local_file" "pipeline" {
  filename = "${path.module}/config/nightly-load.json"
  content = jsonencode({
    workers         = 8
    timeout_minutes = 120
  })
}
""")
run('cd pipeline && { terraform plan -input=false; }')
run('cd pipeline && { terraform apply -auto-approve -input=false | tail -n 1; }')
run('cd pipeline && { terraform plan -input=false; }')

# Step 4: The PR adds a tag and also renames the module block
write('lake/main.tf', """\
module "datalake" {
  source      = "../modules/lake"
  environment = "prod"
  tags        = { cost_center = "sales" }
}
""")
run('cd lake && { terraform init -input=false; }')
run('cd lake && { terraform plan -refresh=false -input=false -out=tfplan; }')

# Step 5: action_reason for every delete in the plan JSON
run('cd lake && { terraform show -json tfplan | jq -r \'.resource_changes[] | select(.change.actions | index("delete")) | "\\(.address) \\(.action_reason)"\'; }')

# Step 6: A moved block: only the tag change is left
write('lake/main.tf', """\
module "datalake" {
  source      = "../modules/lake"
  environment = "prod"
  tags        = { cost_center = "sales" }
}

moved {
  from = module.lake
  to   = module.datalake
}
""")
run('cd lake && { terraform plan -refresh=false -input=false; }')

# Step 7: A pipeline agent dies (kill -9) in the middle of an apply
run('cd locked && { terraform apply -auto-approve -input=false > apply.log 2>&1 & echo $! > apply.pid; }')
run('cd locked && { sleep 5; kill -9 $(cat apply.pid); }')
run('cd locked && { tail -n 1 apply.log; }')

# Step 8: The next plan cannot get the lock (the Created timestamp is filtered out)
run('cd locked && { terraform plan -input=false > locked.txt 2>&1; echo "exit code: $?"; }')
run('cd locked && { grep -v "Created:" locked.txt; }')

# Step 9: The run is dead: force-unlock with the lock ID, then plan again
run('cd locked && { terraform force-unlock -force "$(sed -n \'s/^  ID: *//p\' locked.txt)"; }')
run('cd locked && { terraform plan -input=false; }')

# Step 10: Stop the local state server
run('kill "$(cat locked/server.pid)"; echo "state server stopped"')

# Step 11: Every plan refreshes every object; -minimal-refresh skips unchanged ones
run('cd platform && { terraform plan -input=false | grep "Refreshing state" | wc -l; }')
run('cd platform && { terraform plan -minimal-refresh -input=false | grep "Refreshing state" | wc -l; }')

# Step 12: The new lake state imports the lake resources
run('cd lake-stack && { terraform plan -input=false -out=tfplan | grep -E "# |Plan:"; }')
run('cd lake-stack && { terraform apply -input=false tfplan | tail -n 1; }')

# Step 13: The old state forgets them with a removed block (destroy = false)
write('platform/main.tf', """\
resource "terraform_data" "network" {
  for_each = toset(["hub", "spoke-data"])
  input    = "vnet-sales-${each.key}"
}

# the lake moved to its own state: forget it here, keep it running
removed {
  from = random_string.lake
  lifecycle {
    destroy = false
  }
}
""")
run('cd platform && { terraform plan -refresh=false -input=false -out=tfplan | grep -E "not be destroyed|is set|Plan:"; }')
run('cd platform && { terraform apply -input=false tfplan; }')

# Step 14: Both states: no changes
run('terraform -chdir=platform plan -input=false | grep "No changes"')
run('terraform -chdir=lake-stack plan -input=false | grep "No changes"')

# Step 15: Deleting the file is not enough: Git history and the state still have it
run('cd secrets && { git rm -q terraform.tfvars && git commit -qm "Remove tfvars"; }', env={'GIT_AUTHOR_NAME': 'Data Platform', 'GIT_AUTHOR_EMAIL': 'data-platform@example.com', 'GIT_COMMITTER_NAME': 'Data Platform', 'GIT_COMMITTER_EMAIL': 'data-platform@example.com', 'GIT_AUTHOR_DATE': '2026-10-01T09:00:00Z', 'GIT_COMMITTER_DATE': '2026-10-01T09:00:00Z'})
run('cd secrets && { git show HEAD~1:terraform.tfvars; }', env={'GIT_AUTHOR_NAME': 'Data Platform', 'GIT_AUTHOR_EMAIL': 'data-platform@example.com', 'GIT_COMMITTER_NAME': 'Data Platform', 'GIT_COMMITTER_EMAIL': 'data-platform@example.com', 'GIT_AUTHOR_DATE': '2026-10-01T09:00:00Z', 'GIT_COMMITTER_DATE': '2026-10-01T09:00:00Z'})
run('cd secrets && { grep -n "Winter" terraform.tfstate; }', env={'GIT_AUTHOR_NAME': 'Data Platform', 'GIT_AUTHOR_EMAIL': 'data-platform@example.com', 'GIT_COMMITTER_NAME': 'Data Platform', 'GIT_COMMITTER_EMAIL': 'data-platform@example.com', 'GIT_AUTHOR_DATE': '2026-10-01T09:00:00Z', 'GIT_COMMITTER_DATE': '2026-10-01T09:00:00Z'})

# Step 16: The fix: an ephemeral Key Vault secret into a write-only argument (validate, azurerm 5.11)
run('cd secure && { terraform validate; }')
That is the whole series

All 10 parts are listed on the Real Time Terraform Interview Questions and Answers page on this blog.

The sample data on this page is invented for teaching. Every terminal output is the real output of the command above it, run with Terraform 1.17.0; IDs, timestamps and durations are masked. Azure and Databricks code is checked with terraform validate. Part of the Real Time Terraform Interview Questions and Answers series on TechBrothersIT. Features, limits and their status (GA or preview) are current as of October 2026; check the official Terraform documentation (developer.hashicorp.com/terraform) for anything you quote in an interview.

No comments:
Subscribe to: Posts (Atom)