Skip to content
Workers and routing

How to run workers and route work to them

By the end of this guide a run’s nodes will execute on separate vect worker processes. One node will run only on a worker that serves the edge queue, another only on one that serves vault, and the rest will run in the CLI process as usual. You’ll start the two workers on one machine, watch where each node lands, see what a missing worker looks like, and stop a worker without cutting a change in half.

You need a vect checkout built or installed (Installation), the temporal CLI, and the habits from First steps. The example is docs/site/examples/howto/workers/routing/ in the vect repository. It uses only terraform_data, so it needs no cloud account. For why nodes have task queues at all, read Task queues and where nodes run; this page is only the how.

1. Start Temporal and copy the example

In one terminal, start a local Temporal dev server and leave it running:

temporal server start-dev

In a second terminal, from the repository checkout, copy the example somewhere you can write to and work from there (REPO remembers where the checkout is, for the other copies below). Every command below runs from this folder, because relative paths in the server configuration, such as the key file, resolve against the current working directory:

REPO=$PWD
cp -R docs/site/examples/howto/workers/routing ~/routing
cd ~/routing
export VECT_CONFIG=server.vect

The project has three nodes in a chain:

main.hcl
# Runs on the default task queue, wherever the runner is.
executor "terraform" "app" {
  source = "./roots/app"
}

# Runs only on a worker that serves the "edge" capability.
executor "terraform" "edge" {
  source             = "./roots/edge"
  requires_execution = "edge"
  inputs = {
    app = terraform.app.name
  }
}

# Runs only on a worker that serves the "vault" capability.
executor "terraform" "vault" {
  source             = "./roots/vault"
  requires_execution = "vault"
  inputs = {
    edge = terraform.edge.name
  }
}

requires_execution takes one capability name (a string matching [a-z][a-z0-9_-]*). It says “this node’s plan and apply must run where that capability lives”. A node without it runs on the workspace’s default queue.

2. Map capabilities to task queues

The mapping from capability to Temporal task queue is server configuration, not part of the project, so it can differ between environments without touching main.hcl:

server.vect
workspace "dev" {
  …

  execution {
    default_task_queue = "vect"

    capability "edge" {
      task_queue = "vect-edge"
    }

    capability "vault" {
      task_queue = "vect-vault"
    }

    slots          = { "vect" = 10, "vect-edge" = 4, "vect-vault" = 2 }
    executor_slots = { terraform = 2 }
  }
}

Leave endpoint = "localhost:7233" as shipped if that is where your dev server listens. The full attribute list is in the execution reference.

Every requires_execution name must appear here. Otherwise the run is refused before any node starts:

To see it, copy the broken variant of the project, which differs from the first one only in main.hcl:

cp -R $REPO/docs/site/examples/howto/workers/invalid-unmapped ~/invalid-unmapped
cd ~/invalid-unmapped
main.hcl
executor "terraform" "gpu" {
  source             = "./roots/app"
  requires_execution = "gpu"
}
vect validate
Success!
  (root)  1 block

vect validate passes on that project, because the mapping lives in the server configuration, which validate doesn’t read. The check happens when a run is admitted:

vect plan
vect: failed_precondition: the run cannot start: capability "gpu" required by terraform.gpu: capability has no ExecutionConfig mapping

Go back to the first project for the rest of this guide:

cd ~/routing

3. Start one worker per queue

Each vect worker serves one capability’s queue, so two queues need two processes. Start them in two more terminals (or in the background, as here), with the same configuration and working folder:

vect worker --capability edge  > edge.log  2>&1 &
EDGE=$!
vect worker --capability vault > vault.log 2>&1 &
VAULT=$!

($! is the process ID of the job you just started; the variables save it for later.)

Each prints a single line and then polls until it is interrupted:

cat edge.log vault.log
vect: worker for capability "edge" polling task queue "vect-edge"
vect: worker for capability "vault" polling task queue "vect-vault"

The same process IDs show up in the next section:

pgrep -fl "vect worker"
10385 vect worker --capability edge
10386 vect worker --capability vault

Two things to know about these workers. They run only node activities (plan and apply of the executors routed to their queue), never the run’s own workflow, and they read the same stores as the runner, so they must be able to reach whatever stores the workspace names. Here that’s the local .vect/ folder, which is why everything runs on one machine. A worker elsewhere needs shared stores such as S3 and DynamoDB (Inventory, bundles, and storage).

The default queue, vect, has no vect worker: it is served by the runner, which here is the CLI process itself, or vect serve if you run one.

4. Run, and see where each node landed

vect apply 2>/dev/null
apply of workspace "dev": succeeded (run 01a1e2b4-307f-7e11-9c94-2465939b3801)
  terraform.app    applied (run)  will run
  terraform.edge   applied (run)  will run
  terraform.vault  applied (run)  will run
deployment pointer 1 appended

(2>/dev/null leaves out the progress lines and engine output, which go to standard error; your run ID will differ.) The CLI output doesn’t say which process ran what. Neither do the worker logs: a worker prints its start-up line and nothing more. vect logs --run <id> --node terraform.vault shows the node’s engine output, but not where it ran either. The place to look is Temporal, which records the queue each activity was scheduled on and the process that started it.

notes/where.sh in the example joins those two facts with jq:

TEMPORAL_ADDRESS=localhost:7233 sh notes/where.sh 01a1e2b4-307f-7e11-9c94-2465939b3801
vect.PlanNode.v1  vect  pid 10407
vect.ApplyNode.v1  vect  pid 10407
vect.PlanNode.v1  vect-edge  pid 10385
vect.ApplyNode.v1  vect-edge  pid 10385
vect.PlanNode.v1  vect-vault  pid 10386
vect.ApplyNode.v1  vect-vault  pid 10386

app ran in pid 10407, the vect apply process itself. edge ran in 10385 and vault in 10386, the two workers from the previous step. Evaluation, leases, and the run’s bookkeeping always stay on the default queue, which is why they aren’t in the list. The same history is in the Temporal UI (http://localhost:8233 for the dev server) under the workflow vect/dev/<run id>.

If you ever need only the queue, temporal workflow show -w vect/dev/<run id> --output json lists every ACTIVITY_TASK_SCHEDULED event with its taskQueue.name.

Coming from Terraform: there’s no Terraform equivalent of “run this root on that machine”; you’d pick a CI runner for the whole job. In vect the unit of placement is the node, and it is chosen by a capability name in the project and a queue name in the server configuration, so a project can move between environments without edits.

5. When no worker serves a queue

Stop the vault worker and change that node so there is something to apply:

kill -TERM $VAULT
sed -i.orig 's/vault after/vault v2 after/' roots/vault/main.tf

Now apply:

vect apply
terraform.app: running
terraform.app: unchanged inputs unchanged
terraform.edge: running
terraform.edge: unchanged inputs unchanged
terraform.vault: running

Then nothing. There is no error and no timeout: the node’s plan activity is scheduled on vect-vault, nobody polls that queue, and the run waits. The node’s timeout doesn’t help, because it bounds an activity that has started. The signs are all in Temporal. In another terminal:

temporal task-queue describe --task-queue vect-vault
Task Queue Statistics:
    BuildID    TaskQueueType  ApproximateBacklogCount  ApproximateBacklogAge  BacklogIncreaseRate  TasksAddRate  TasksDispatchRate
…
Pollers:
…

The activity row of the statistics (trimmed here, because its numbers change every second) shows a backlog of at least one task whose age keeps growing, and the pollers’ LastAccessTime keeps growing too. That is a queue with no worker. (Temporal remembers a poller for a while after it exits, and may even count tasks as dispatched to it; trust the backlog and the access time, not the list or the rates.) temporal workflow describe -w vect/dev/<run id> shows the same thing from the run’s side: a pending vect.PlanNode.v1 activity in state Scheduled.

The fix is to start a worker for the queue, from the same folder and configuration. The waiting run picks up where it was:

vect worker --capability vault > vault.log 2>&1 &
VAULT=$!
…
  terraform.app    unchanged         inputs unchanged
  terraform.edge   unchanged         inputs unchanged
  terraform.vault  applied (update)  will update
deployment pointer 2 appended

(Trimmed to the end of the run, without the engine’s own output; the apply you left waiting prints it.) If you would rather give up instead, Ctrl-C in a CLI-direct run cancels it.

6. Stopping a worker without cutting a change in half

A node that is mid-apply is never retried automatically, so interrupting it would fail the node and leave it for a recovery plan. A worker that gets SIGTERM therefore stops taking new work and lets its running activities finish, for up to server.drain_timeout (15 minutes unless you set it), before it cancels what is left. To see it, add a node that takes about 20 seconds, on the vault queue:

cp drain/slow.hcl .
cp -R drain/roots/slow roots/
slow.hcl
# A node that takes a while, so there is something to drain.
executor "terraform" "slow" {
  source             = "./roots/slow"
  requires_execution = "vault"
}

Start vect apply, and when its log shows the node’s local-exec provisioner starting, send SIGTERM to the vault worker:

vect apply
kill -TERM $VAULT
date +%T; sleep 6; pgrep -fl "vect worker"
21:53:54
10385 vect worker --capability edge
10653 vect worker --capability vault

Six seconds later the vault worker is still alive, finishing the change. Twenty seconds after the signal it exits on its own, and the apply, in its own terminal, finishes normally:

…
  terraform.app    unchanged      inputs unchanged
  terraform.edge   unchanged      inputs unchanged
  terraform.slow   applied (run)  will run
  terraform.vault  unchanged      inputs unchanged
deployment pointer 3 appended

(Trimmed to the end of the run, without the engine’s own output.) The worker prints nothing as it stops. Draining only protects the activity that was already running. If the signal arrives between a node’s plan and its apply, the plan finishes, the worker exits, and the apply activity then waits for a worker, exactly as in the previous section. For a rolling restart, start the replacement worker first, then signal the old one.

A default you can swap: the drain window is drain_timeout in the server block. A very long one lets a restart wait for a long tofu apply; a short one trades that for a faster shutdown and may cancel a change mid-apply, which the next plan then has to recover.

7. Size the workers

Two numbers in execution bound a worker’s concurrency, and both are per worker process, read when the worker starts:

  • slots is a map from queue name to how many activities that queue runs at once in one process (default 10 per queue).
  • executor_slots caps how many activities of one executor type, such as terraform, run at once in one process, across all its queues.

A second vect worker on the same capability, on another machine, doubles that queue’s capacity, because each process brings its own limits. The slots/ example has two independent nodes that each sleep 15 seconds on the vault queue. First stop the workers from the earlier sections, because they serve the same vect-vault queue from different stores and would take this project’s tasks:

kill -TERM $EDGE $VAULT

Then copy the example, start a vault worker, and time vect apply:

cp -R $REPO/docs/site/examples/howto/workers/slots ~/slots
cd ~/slots
vect worker --capability vault > vault.log 2>&1 &
time vect apply >/dev/null 2>&1
kill $!
real	0m17.808s
user	0m0.973s
sys	0m0.396s

That is the shipped setting: slots for vect-vault is 2 and executor_slots is terraform = 2. Repeating it in a fresh copy of the example with other settings (edit the two numbers in server.vect) gives the table below. The table gives times relative to the baseline row, because yours will differ by a few seconds (each node also spends a moment initializing OpenTofu):

slots for vect-vaultexecutor_slotsvect apply takes
1terraform = 2about twice as long
2terraform = 2the baseline: one 15-second sleep plus start-up
2terraform = 1about twice as long

With one slot the two nodes run one after the other. With two they overlap. With two slots but one terraform permit they are serialized again, because executor_slots counts across queues.

Things to know

  • A worker serves exactly one capability. There is no --capability default, and the default queue is the runner’s. To serve two capabilities on one machine, start two workers, as above.
  • A capability name no node provides is fine. requires_execution doesn’t need a node with provides_execution in the same project; the mapping in execution is enough. Add provides_execution when the worker’s place is created by the graph itself (a VPC, say). The consumer then plans after its provider has applied, and vect plan shows it as deferred capability not bootstrapped until then. Runs shows that case.
  • Different stores, different problems. Workers and the runner that polls the same queue must share stores. Two runners with different local stores can take each other’s tasks and fail with Unknown bundle (Runs).
  • Known limitation: vect approve --run and vect reject --run query the run’s workflow and can fail with context deadline exceeded when no runner is polling the default queue; see issue #92.

See also