Skip to content
Runs on Temporal

How runs work

In First steps you ran vect plan and vect apply and they just worked. This guide explains what was going on underneath: what a run is, why vect builds it on Temporal, what happens when part of a run fails, how approvals and cancellation work, and how the same CLI runs either in your terminal or against a long-lived server with its own workers.

The examples come from three small projects under docs/site/examples/topics/runs/ in the vect repository. They use only terraform_data and the random provider, so you can run them against a local Temporal dev server. Run IDs, digests, and timestamps in the output will differ from yours.

What a run is

Every vect plan, vect apply, and vect destroy is one run, and a run is one Temporal workflow in one workspace. Its workflow ID is vect/<workspace>/<run id>, so a run you started in the tutorial is a workflow called vect/dev/01a1db02-... that you can find in the Temporal UI.

The workflow doesn’t do the work itself. It owns the shape of the dependency graph and the status of each node, and it schedules activities for everything else: evaluating a node’s inputs, taking its lease, planning it, applying it, and writing the run’s result. Activities are where the executors, the tofu processes, and the stores live. The workflow’s history holds only the graph shape, per-node status, and references to values in the encrypted value store. It never holds configuration, plan bytes, or evaluated values, so values never enter Temporal’s event history, and a big configuration doesn’t make a big history.

You don’t write any of this. What you get from it is the same thing a Temporal user gets: the run is durable. It isn’t a process in your terminal, it’s a record in Temporal that workers carry forward. A worker can restart in the middle of it, and an approval can wait for hours.

Coming from Terraform: a Terraform run is a process, and everything about it (the lock, the progress, the approval prompt) lives and dies with that process or with the CI job that hosts it. A vect run outlives the command that started it. vect apply is a client that starts the run and watches it, and the run keeps its own state in Temporal and in Inventory.

Facts are pinned at the start

Before any node runs, the run resolves its facts: the plan-time globals and data blocks that the graph is built from. It stores them once, records a digest, and uses only those pinned values until it ends. A data block read at the start isn’t read again halfway through, so one run can’t see two different answers to the same question.

You can see the digests on the run record, with vect runs <run id>. This is the apply we’ll look at in a moment, which was started with a fact file that sets a global (the node results that follow the header are left out here):

vect runs 01a1e26c-233a-70d6-bf14-ccd38dac5360
run 01a1e26c-233a-70d6-bf14-ccd38dac5360: apply of workspace "dev": failed
scope: full
started: 2026-11-15T15:40:29Z  finished: 2026-11-15T15:40:30Z
vect version: 0.1.8
bundle: sha256:047c29af8cf4522b1e602f91bb9da3dfefee3ba2c5ace3d34476c99a1e5289fb
source: sha256:ea28b6083f833b58e60a47344fb536e920065851a7527e1f1cd70c9ddb41b43f
facts: sha256:367f658732f5034ec44093e9559ed3e0c72a59b97a6f1c3dda91a3bff06f126b
metadata: -

A run with nothing to pin, like the recovery apply later in this guide, shows sha256:e3b0c442...b855, the digest of empty input.

A run also records the vect version that produced it, and refuses a bundle whose required_version your vect doesn’t satisfy. RUN-RUN-2 to RUN-RUN-5 have the full list.

Plan, then apply

The two commands differ in one important way. vect plan changes nothing: it plans every node it can, reports what each would do, lists the approval requests it would raise, and leaves it at that. It takes no leases and calls no mutating executor method.

vect apply doesn’t apply “the plan”. It plans each node again immediately before applying it, once that node’s upstreams have converged. That’s why Part 3 showed a node as deferred by the executor during plan: its inputs depend on something that doesn’t exist yet. During apply those inputs are known by the time its turn comes, so it gets planned then.

Here is the first plan of the pipeline project, with the engine’s own lines left out:

terraform.database: running
terraform.docs: running
terraform.docs: planned will run
terraform.database: planned will run
terraform.migrate: running
terraform.migrate: deferred unknown inputs: inputs.database
terraform.service: running
terraform.service: deferred unknown inputs: inputs.schema
plan of workspace "dev": succeeded (run 01a1e773-26c4-73b9-b508-949af06c9f4d)
  terraform.database  run    will run
  terraform.docs      run    will run
  terraform.migrate   defer  unknown inputs: inputs.database
  terraform.service   defer  unknown inputs: inputs.schema

The project has four roots. database and docs stand alone, migrate needs the database’s name, and service needs what migrate produces:

main.hcl
global "break_migration" {
  type  = bool
  value = false
}

executor "terraform" "database" {
  source = "./roots/database"
}

executor "terraform" "migrate" {
  source = "./roots/migrate"
  inputs = {
    database = terraform.database.name
    fail     = break_migration
  }
}

executor "terraform" "service" {
  source = "./roots/service"
  inputs = {
    schema = terraform.migrate.schema
  }
}

executor "terraform" "docs" {
  source = "./roots/docs"
}

output "service" {
  value = terraform.service.endpoint
}

The migrate root runs a local-exec provisioner that exits non-zero when fail is true. That gives us a node we can break on purpose.

When part of a run fails

Let’s break it. A fact file can set a global for one run, so this one flips break_migration without touching the configuration:

break.facts.hcl
global "break_migration" {
  value = true
}
vect apply --facts break.facts.hcl
…
apply of workspace "dev": failed (run 01a1e773-348a-7bd7-a266-5018cde6a880)
  terraform.database  applied (run)  will run
  terraform.docs      applied (run)  will run
  terraform.migrate   failed         terraform.migrate: apply: tofu apply: exit status 1: Error: local-exec provisioner error
  …
  terraform.service  skipped  upstream terraform.migrate failed
…

(The summary is trimmed; vect prints the engine’s whole error. You may also see a couple of Temporal client log lines on standard error for the failed activity. They’re noise.)

Read it as a graph. migrate failed, so service, which comes after it, was skipped. database and docs don’t depend on migrate, so they kept running and finished. The run is failed and the command exits 1, because anything failed or skipped fails the run. But the work that did succeed is real and is committed:

vect inventory
ADDRESS             VERSION  STATUS
terraform.database  1        converged
terraform.docs      1        converged
terraform.migrate   0        none

POINTER  BUNDLE                                                                   SCOPE  APPLIED               METADATA
1        sha256:701cdd1d98fed7913b84c712a6b089bac3c01d6604f3ea8e4ee2fd92a68e91e4  full   2026-11-17T03:00:50Z  

The second table is the deployment pointer history; even this failed run appended one, since part of it was applied. migrate has no committed version, database and docs do. A failure stops what has to come after it and nothing else.

Recovering is planning again

Now fix the problem by running without the fact file, and start with a plan:

vect plan
…
terraform.migrate: planned will update, destroying 1 resource (see approval requests) (needs recovery after a failed change)
…
  terraform.migrate   update  will update, destroying 1 resource (see approval requests) (needs recovery after a failed change)
  terraform.service   run     will run
approval requests:
  apr_ded8f3227d9db5077ac58867  destroy  terraform.migrate::terraform_data.migrate  destroy terraform.migrate::terraform_data.migrate

vect doesn’t resume the failed apply, and it never retries a mutating step for you. It plans the node again from the real state of the thing it manages. OpenTofu left the half-created resource tainted, so the new plan replaces it, and a replacement contains a destroy. That’s a destroy like any other: it raises an approval request, and it goes through the normal approval rules. You can think of recovery as “plan again, then apply, under the same rules as any other change”.

This is a deliberate rule, not a gap. An activity that applies a change is never retried automatically, because a retry after a lost response could mutate twice. Executors are written to be idempotent, and the recovery path is a fresh plan. The same goes for a run that is interrupted while a node is mid-change: the node’s record keeps an in-flight marker, and the next plan says so (we’ll see that under cancellation). Activities that don’t touch your infrastructure, like evaluating inputs or writing the run record, do retry with backoff, since repeating them is safe.

A failed or stopped terraform node can also leave its backend’s state lock behind. When the node is recovered, vect forces that lock once, but only if the previous attempt’s recorded exit shows its engine stopped and the lock’s owner and creation time match that attempt. Otherwise it never forces: the node fails and the error names the lock ID. When the previous attempt can’t show that its engine stopped (a lost worker), the node is uncertain instead, and a run skips it. Once you have checked that the lock is the stopped operation’s, vect inventory resolve <address> --force-unlock <lock-id> releases it and clears the mark; see the CLI reference. Spec: EXEC-IDEM-4, FLEET-CONC-14. The s3 backend vect derives from a workspace’s state store has no lock, since the node lease serves as one.

Coming from Terraform: this is the same model as a failed terraform apply followed by another plan and apply: the state tells the truth, and a re-plan decides what to do next. The difference is that vect applies it across many roots. The nodes that succeeded stay committed, the ones that didn’t are planned again, and the executor’s own idempotency and recovery behavior is documented per executor.

Approvals

An approval request is raised for each destroy a plan reports, each moved that would act, and anything an executor asks to have approved. A request blocks only its own node and the nodes that must come after it. Independent nodes keep running.

How a request is answered depends on who is watching:

  • A person at a terminal is prompted, as you saw in Part 3.
  • A script or CI job running CLI-direct has no one to ask. By default the request is rejected (unattended_approvals = "reject"), or approved if you pass --auto-approve or set the policy to approve.
  • A run on a server waits. The request stays pending, and anyone with permission can answer it from another terminal.

That last case is what makes approvals useful beyond one person’s laptop. It needs a served runner, and a served runner needs your own identity provider, so you can’t run the next few transcripts with only what’s in this guide. To reproduce them you need:

  • an OIDC issuer URL for the authenticator "oidc" block (the serve.vect shown below), reachable from the server;
  • tokens for two principals, one that starts runs and one (bob@example.com here) that may approve, each with aud set to vect and a lifetime of 15 minutes or less, passed with --token-file or VECT_TOKEN_FILE;
  • VECT_CONFIG pointing at the server configuration in the client’s shell as well, so the client knows its workspace (default_workspace);
  • a running server: VECT_CONFIG=serve.vect vect serve, which prints vect: serving on 127.0.0.1:8443, with VECT_API_URL=http://127.0.0.1:8443 set for the client.

The transcripts here were captured against a throwaway local issuer. Here a run was started against a server with no terminal attached, and it paused:

vect apply --metadata ticket=OPS-142
…
terraform.migrate: awaiting_approval awaiting approval
  approval request apr_ded8f3227d9db5077ac58867  destroy  terraform.migrate::terraform_data.migrate  destroy terraform.migrate::terraform_data.migrate
  answer with: vect approve --run 01a1e773-46a0-7a6f-9dfc-5f441beb48e5 --workspace dev <request-id>  (or reject)
…

From another terminal (or another person), vect approve --run with no request ID lists what the run is waiting for, and with an ID it answers:

vect approve --run 01a1e773-46a0-7a6f-9dfc-5f441beb48e5
run 01a1e773-46a0-7a6f-9dfc-5f441beb48e5 awaits 1 approval request:
  apr_ded8f3227d9db5077ac58867  destroy  terraform.migrate::terraform_data.migrate  destroy terraform.migrate::terraform_data.migrate
vect approve --run 01a1e773-46a0-7a6f-9dfc-5f441beb48e5 apr_ded8f3227d9db5077ac58867
approved 1 approval request

The first terminal’s run then continues on its own, and the run record keeps who answered. vect runs <run id> shows it, after the same header and node results as above:

vect runs 01a1e773-46a0-7a6f-9dfc-5f441beb48e5
…
  terraform.migrate   applied    will update, destroying 1 resource (see approval requests)
  terraform.service   applied    will run
approval decisions:
  apr_ded8f3227d9db5077ac58867  destroy  terraform.migrate::terraform_data.migrate  approved by bob@example.com at 2026-11-17T03:00:54Z

vect reject is the counterpart. A rejected request skips its node and everything that must come after it. For an apply that’s the node’s dependents; for a destroy, where the order is reversed, it’s the nodes it depends on.

Three details are worth knowing:

  • Waits are bounded. Each request waits for approval_timeout, which is 30 minutes unless you set it on the workspace or pass --approval-timeout for one run. When time runs out, the request counts as a rejection and is reported as a timeout, so a forgotten run ends instead of hanging.
  • Long waits are cheap. A run that is only waiting on approvals can hand its state to a fresh Temporal execution (continue-as-new), carrying the pending requests and the leases it holds. A wait of any length doesn’t run into Temporal’s history limit.
  • Approvals can be given ahead of time. vect plan --out plan.json, then vect approve plan.json, attests the plan’s requests, and vect apply plan.json honors them when the change digest still matches. (plan_approvals controls whether plan-file approvals count.) Saved plans are covered under vect plan in the CLI reference, and this is the only part of them you need here.
A default you can swap: both unattended and timed-out requests fail closed. unattended_approvals and approval_timeout live on the workspace in the server configuration, and --auto-approve and --approval-timeout override them for one run. Approving is a permission (approval.approve:<kind>), separate from starting the run, so on a shared server the person who starts an apply need not be the one who can approve its destroys. See RUN-APPROVAL-1 to RUN-APPROVAL-12.

Cancelling a run

A run that is waiting for something can be cancelled, and so can one that is busy. The cancel project has one node that sleeps for five minutes. We start it, then press Ctrl-C (or send SIGINT) after a few seconds (we leave out the engine’s own lines and the Temporal client’s log lines):

terraform.slow: running
vect: interrupted; canceling run 01a1e773-5cf1-7f07-8152-d21454e2f428
terraform.slow: running
…
terraform.slow: failed canceled
apply of workspace "dev": canceled (run 01a1e773-5cf1-7f07-8152-d21454e2f428)
  terraform.slow  failed  canceled
the run was canceled
deployment pointer 1 appended

On cancel, no new node starts, and running activities get the cancellation through their context. Temporal delivers it on the activity’s next heartbeat, so a busy node stops within about 24 seconds at worst. The run waits for the activity to end first, so its lease is released and it has recorded where it stopped before the run closes in the canceled state. The command exits 1.

What cancellation doesn’t do is undo anything. The node was mid-change, so the next plan finds its in-flight marker and says so (we show only the lines that mention it):

terraform.slow: planned will update, destroying 1 resource (see approval requests) (in-flight from run 01a1e773-5cf1-7f07-8152-d21454e2f428)
  terraform.slow  update  will update, destroying 1 resource (see approval requests) (in-flight from run 01a1e773-5cf1-7f07-8152-d21454e2f428)

That’s the same recovery path as a failure: plan again from the real state, then apply.

For a terraform node, stopping is done in two steps. The engine runs in its own process group, and vect sends the group SIGINT, the same as Ctrl-C in a terminal, so the engine can stop and write its state; its log shows Interrupt received. and the local-exec child (sleep here) goes with it. If the group is still running after server.engine_stop_grace (30 seconds by default, never longer than the shutdown drain), vect kills it. The node’s lease stays held until the engine has exited. A lost lease is the exception: the worker can no longer say the node is its own, so it kills the group at once, with no grace. On Linux an engine also dies with its worker. On other systems an engine orphaned by a dying worker is not stopped. Spec: EXEC-TF-14; the setting is in the server reference.

A word on where Ctrl-C works. A CLI-direct run forwards the cancellation for you, as above. When you’re watching a run on a server (--api-url), Ctrl-C just stops watching. The run keeps going on the server, which is usually what you want from a client that can lose its connection. The API has a CancelRun call for that, but the CLI doesn’t expose it as a command today. A run waiting on an approval ends when the request is rejected or times out.

What a run leaves behind

Every run is kept. vect runs lists them, and vect runs <run id> shows one run in full: its digests, metadata, each node’s result and reason, and the approval decisions. The newest run is first:

vect runs
RUN                                   KIND   STATE      SCOPE  STARTED               FINISHED              METADATA
01a1e773-46a0-7a6f-9dfc-5f441beb48e5  apply  succeeded  full   2026-11-17T03:00:54Z  2026-11-17T03:00:54Z  ticket=OPS-142
01a1e773-3eb4-7b8d-a8a7-76549ff554cb  plan   succeeded  full   2026-11-17T03:00:52Z  2026-11-17T03:00:52Z  -
01a1e773-348a-7bd7-a266-5018cde6a880  apply  failed     full   2026-11-17T03:00:49Z  2026-11-17T03:00:50Z  -
01a1e773-26c4-73b9-b508-949af06c9f4d  plan   succeeded  full   2026-11-17T03:00:45Z  2026-11-17T03:00:48Z  -

--metadata key=value, repeatable, tags a run (ticket=OPS-142 above) and the tag shows up in this listing, on the plan file, and on the deployment pointer. vect never interprets it, so use it to tie a run to a change ticket, a pipeline, or a commit.

vect logs --run <id> prints what the engine said, per node and per attempt, and --node <address> narrows it to one node. --follow tails a run that is still going. The logs are captured while the run executes and stored encrypted, so you can read them after the run has finished and from a machine that wasn’t watching.

Running part of the graph

--target <address> limits a run to one executor (or stack). You’ll know the idea from Terraform, but vect needs one more answer: what about the nodes your target depends on that are now out of scope?

By default it doesn’t run them. It reads their outputs from Inventory, as of the last time they converged, and pins those with the run’s facts. Here we changed only service, adding /v2 to its endpoint:

roots/service/main.tf
resource "terraform_data" "service" {
  input = "https://${var.schema}.example.test/v2"
}

and targeted it (leaving out the engine’s own lines):

vect plan --target terraform.service
terraform.service: running
terraform.service: planned will update
plan of workspace "dev": succeeded (run 01a1e773-8fd2-74d6-ba30-3a903c5eb767)
scope: --target terraform.service (upstream: inventory)
  terraform.service  update  will update

migrate’s schema came from Inventory, so nothing upstream ran. If you’d rather run what the target needs too, add --upstream=include (the engine’s own lines are left out again):

vect plan --target terraform.migrate --upstream=include
terraform.database: running
terraform.database: planned inputs unchanged
terraform.migrate: running
terraform.migrate: planned no changes
plan of workspace "dev": succeeded (run 01a1e773-9673-7622-b1c7-8c79284d5d71)
scope: --target terraform.migrate (upstream: include)
  terraform.database  no-op  inputs unchanged
  terraform.migrate   no-op  no changes

If an upstream output isn’t in Inventory yet, the run is refused up front with a message that suggests --upstream=include, rather than failing in the middle. A target of the form <address>::<internal> also works: vect includes <address> and passes <internal> to the executor, which for terraform becomes -target= on the engine.

CLI-direct and served runs

All of this has run CLI-direct so far, and that’s the default: the vect command starts an in-process server, with Temporal workers in it, does the work, and exits. There is no separate “local mode” with different behavior. The CLI always calls the same API, and an in-process server answers when you don’t point it at another.

To share runs between people and machines, run vect serve and point commands at it:

serve.vect
server {
  name              = "runs"
  default_workspace = "dev"
  listen            = "127.0.0.1:8443"
}

authenticator "oidc" "sso" {
  issuer   = "https://idp.example.com"
  audience = "vect"
}

policy {
  rule {
    effect     = "allow"
    actions    = ["bundle.upload", "run.plan", "run.apply", "run.destroy", "run.cancel", "inventory.read"]
    workspaces = ["*"]
  }

  rule {
    effect     = "allow"
    actions    = ["approval.approve:*"]
    workspaces = ["dev"]
    principal  = { subject = "bob@example.com" }
  }
}

(The rest of the file is the defaults and workspace blocks you already have. The transcripts in this guide were captured against a throwaway local OIDC issuer instead of idp.example.com.) Then:

export VECT_API_URL=http://127.0.0.1:8443
export VECT_TOKEN_FILE=token
vect apply

Same commands, same output, because the CLI is the same client either way. What changes is who runs the work: workers inside vect serve do, and they keep going after your command is gone.

A served runner authenticates every request, and authorization denies whatever no rule allows. The second rule above lets only one person approve, which shows in the output: when the other principal tries, the call fails.

To see it, rename the docs resource so that the next apply has to destroy the old one, then start that apply as the principal who is not allowed to approve. It raises an approval request and waits, so leave it running:

perl -pi -e 's/"docs" \{/"docs2" {/' roots/docs/main.tf
vect apply
…
terraform.docs: awaiting_approval awaiting approval
  approval request apr_78f5367fee85b621b7f71445  destroy  terraform.docs::terraform_data.docs  destroy terraform.docs::terraform_data.docs
  answer with: vect approve --run 01a1e773-a3a4-73f3-8663-c3bd3fbb885e --workspace dev <request-id>  (or reject)
…

From another terminal, the same principal tries to answer it:

vect approve --run 01a1e773-a3a4-73f3-8663-c3bd3fbb885e apr_78f5367fee85b621b7f71445
vect: permission_denied: permission denied: approval.approve:destroy on "terraform.docs::terraform_data.docs"
A default you can swap: CLI-direct needs no authentication because the caller is the server. The local principal exists only on that in-process connection and is rejected on any network listener. For a served runner, authenticator accepts oidc and aws_web_identity, and policy rules match actions, workspaces, targets, and principal attributes, including subject and issuer. See server configuration and API-AUTHZ-1 to API-AUTHZ-14.

One caution. Every runner of a workspace polls the same Temporal task queue, and whichever worker is free picks up the work. Two runners with different local stores, such as a CLI-direct command in one folder while a vect serve for the same workspace runs from another, can take each other’s tasks and fail with errors like Unknown bundle. With shared stores (S3 and DynamoDB, as in the storage callouts of Part 3) this isn’t a problem. For local use, pick one or the other per workspace.

Task queues and where nodes run

By default every node’s activities run on the workspace’s default_task_queue, polled by the runner. But some nodes can only run somewhere special: inside a private network, in an account with certain credentials, on a machine with a licensed tool. For these, vect has capabilities.

A node can provide a capability (it creates the place where work can run) and another can require it:

main.hcl
executor "terraform" "runner" {
  source             = "./roots/runner"
  provides_execution = "runner"
}

executor "terraform" "deploy" {
  source             = "./roots/deploy"
  requires_execution = "runner"
}

The server configuration maps each capability name to a Temporal task queue, and gives each queue and executor type a concurrency budget:

server.vect
workspace "dev" {
  …
  execution {
    default_task_queue = "vect"

    capability "runner" {
      task_queue = "vect-runner"
    }

    slots          = { "vect" = 10, "vect-runner" = 4 }
    executor_slots = { terraform = 2 }
  }
}

A node with requires_execution = "runner" has its plan and apply activities scheduled on vect-runner. Anything else uses the default queue. A worker for that queue is a separate process that reads the same server configuration:

vect worker --workspace dev --capability runner
vect: worker for capability "runner" polling task queue "vect-runner"

Planning shows the dependency between the two nodes. Until a node that provides runner has been applied, the node that requires it can’t have a worker to run on, so its plan is deferred (again without the engine’s own lines):

terraform.runner: running
terraform.runner: planned will run
terraform.deploy: running
terraform.deploy: deferred capability not bootstrapped
plan of workspace "dev": succeeded (run 01a1e773-c2ad-7b08-9754-92f3125b89c2)
  terraform.deploy  defer  capability not bootstrapped
  terraform.runner  run    will run

An apply puts the provider first, so the worker is online before anything that needs it plans. You can see that in Temporal’s own history for the run, which lists each activity with the queue it was scheduled on:

Apply the project, which prints the run ID:

vect apply
…
apply of workspace "dev": succeeded (run 01a1e773-c880-7510-92d1-a243d313fdd3)
  terraform.deploy  applied (run)  will run
  terraform.runner  applied (run)  will run
deployment pointer 1 appended

Then ask Temporal for that run’s history:

temporal workflow show -w vect/dev/01a1e773-c880-7510-92d1-a243d313fdd3 --output json |
  jq -r '.events[] | select(.eventType=="EVENT_TYPE_ACTIVITY_TASK_SCHEDULED")
         | .activityTaskScheduledEventAttributes
         | "\(.activityType.name)  \(.taskQueue.name)"'
vect.Evaluate.v1  vect
vect.Evaluate.v1  vect
vect.AcquireLease.v1  vect
vect.PlanNode.v1  vect
vect.ApplyNode.v1  vect
vect.Evaluate.v1  vect
vect.AcquireLease.v1  vect
vect.PlanNode.v1  vect-runner
vect.ApplyNode.v1  vect-runner
vect.Finish.v1  vect

The first PlanNode and ApplyNode pair is runner’s and uses the default queue. The second pair is deploy’s and runs on vect-runner. Evaluation, leases, and the run’s own bookkeeping always stay on the default queue.

Two limits keep a worker from being overwhelmed. slots caps how many activities one worker process runs at once per queue (10 by default), and executor_slots caps how many of one executor type run at once across all queues in that process. Both count per worker process, so adding a worker adds capacity. Waiting for things (convergence, approvals) doesn’t take a slot, so a run parked on an approval doesn’t starve the real work.

Leases keep two runs off the same node

Because runs are separate and can overlap, something has to stop two of them from changing the same thing. That’s the node lease. A run takes a lease on a node before it plans it and holds it until the change commits to Inventory, and no other run can lease that node meanwhile. A run that needs a node someone else holds waits for it, and reports who holds it, instead of failing.

Leases expire so that a dead holder can’t block a node forever. A lease lasts five minutes and the worker keeps renewing it while the node’s activity runs, and the run renews the ones it holds between activities. If the holder is gone, the next run to lease the node recovers it through the in-flight marker, the same way it recovers from a cancel. A plan-only run takes no leases at all. Leases are what loops, triggers, and affected runs build on, and they get their own treatment in a later guide.

Deploying new versions of vect

A workflow can run for a long time, longer than the gap between two releases of vect. Temporal solves this by pinning a workflow to the worker build that started it, so that its code never changes in the middle of a run. vect uses pinned workflows: every run is tied to the build it began on.

That would strand a run if the old build disappeared, for example when you roll out a new vect serve. So when vect serve starts (only then: not a CLI-direct command, and not vect worker), it moves each unfinished run of its workspaces that another build left pinned onto its own build, which keeps the runs, their pending approvals, and their leases alive across a deploy. We saw it happen: a destroy waiting on approval under one build, then vect serve started from a second build, which logged re-pinned a run to this build workspace=dev run=vect/dev/01a1dff4-... from=0.1.0+103af22cff6a to=0.1.0+dev. This is safe only if the workflow code is replay-compatible, so every change to it is gated by a version marker and checked by a replay test against recorded histories. In practice you can upgrade vect serve while a run waits on approval.

Shutting down is similar. When vect serve or vect worker gets SIGTERM, it stops accepting new work and lets running activities finish for up to server.drain_timeout, which is 15 minutes by default, before cancelling what’s left. A terraform engine that is cancelled then gets SIGINT and server.engine_stop_grace to exit (see cancelling). A node mid-apply is never retried automatically, so cancelling it on every restart would fail it for no reason. Draining first means a rolling restart doesn’t cut a tofu apply in half. The HTTP listener, though, closes at once on SIGTERM, so a client that was watching a run sees vect: internal: protocol error: incomplete envelope: unexpected EOF. The run itself is unaffected; watch it again with vect logs --follow or vect runs.

Where to go from here

You now have the shape of a run: a Temporal workflow that pins its facts, plans and applies a graph of nodes under leases, pauses for approvals, survives restarts, and leaves a record behind. Two things sit on top of it and get their own guides: continuous reconciliation with loops, triggers, and affected runs, and reusing saved plans. What each store holds, and how Inventory and the value store keep the records a run depends on, is in Inventory, bundles, and storage. The requirement-level detail for everything above is in the runtime and fleet scheduling specifications.