Agent Runners & Services

The Rescile Controller supports executing actions directly via the REST API or passing them to the Action Queue, where connected rescile-runner instances process them.

The rescile-runner enables you to execute workloads (like scripts, Terraform, or background services) securely within isolated network segments (e.g., inside a VPC, on-premise datacenter, or a DMZ).

Runner Execution Modes

To accommodate various networking architectures, runners connect to the Controller in agent mode (a persistent, outbound connection), alongside manual one-shot execution.

1. Agent Mode (Push)

In Agent Mode, runners connect outward to the Rescile Controller using a continuous gRPC multiplexed stream over the standard HTTP/2 endpoint. This is ideal for crossing NAT boundaries and executing workloads without opening inbound firewall ports.

To start a rescile-runner in agent mode, specify the --mode agent flag and provide the Controller URL and an API token.

rescile-runner \
    --mode agent \
    --controller-url https://controller.rescile.local:7600 \
    --token "$RESCILE_API_TOKEN" \
    --agent-labels "prod,dmz"

The agent immediately connects, registers its profile, and begins listening for dispatched jobs. Execution logs and status updates are pushed in real-time back to the Controller.

Capability-Based Routing

When an agent runner connects, it registers a Capability Profile containing:

• System OS and Architecture (e.g., linux, x86_64)
• Hostname as hostname:<HOSTNAME>, where HOSTNAME is read from the runner’s environment. Match it with constraints like hostname:rescile-runner-01.
• Dynamic Capabilities: Available tools like docker, nix, podman, or wasi. The rescile-ce runner always reports system when it can execute host commands.
• Custom Labels: User-defined tags for workload targeting (e.g., gpu, pci-dss, dmz).

Constraints can also reference module parameters ({{ params.runner_hostname }}) and the origin resource ({{ origin_resource.name }}), allowing a single action template to target the correct runner at dispatch time.

Matching Semantics

The Controller queues incoming execution requests (Actions) and continuously matches them against connected runners based on their required capabilities. The assignment is only dispatched if a runner satisfies all constraints. A runner satisfies a queued action when:

• every constraint matches a runner capability set to true (capabilities["<constraint>"] == true) or a runner label (labels contains the constraint), and
• every required runtime matches a runner capability set to true — labels do not satisfy runtime requirements.

The required_runtime tokens are normalized from the action’s runtime.engine:

Observing Runners and Demand via the API

Two questions, two endpoints:

• Which runners are available? GET /api/runners lists every registered runner with its capability profile, labels, and last_seen heartbeat timestamp. A runner whose last_seen grows stale has stopped heartbeating and will not receive dispatches.
• What is waiting for a runner? GET /api/action-queue items carry constraints and required_runtime, so a pending item’s demand can be compared against the runner list with the matching rules above. An item is servable when at least one connected runner satisfies both fields.

To answer which runners are required to run all actions (not just what is queued right now), aggregate GET /api/actions: group the actions by their (required_runtime, static constraints) shape and check each shape against GET /api/runners. Keep two caveats in mind:

• Constraint templates (server:{{ origin_resource.name }}) only resolve at execution time. They can further restrict which runner may take an action, but never loosen it — so a statically covered action can still wait on a runner that never matches the rendered constraint.
• api-engine actions need no runner at all; their required_runtime is empty.

Jobs vs. Services

Actions define a run_mode which instructs the runner on how to manage the process lifecycle:

• Job (Default): The runner executes the process, waits for it to exit, and reports the exit code (Success/Failure) back to the Controller. Suitable for automation scripts, IaC deployments, or finite data processing tasks.
• Service: The runner spawns the process in the background and continuously monitors it. If the Controller sends an updated configuration for the action, the runner gracefully restarts the service (Zero-Downtime reconciliation). Suitable for background daemons, web servers, or long-running APIs.

HTTP access to service actions

An application route with target = "action://<module>/<action>/<path>" reaches one running service action. Launch its HTTP server with action exec and declare its listener with [runtime.expose]. Set auto_execute = true when service should start automatically. HTTP requests go to that action’s TCP/Unix listener over runner’s outbound gRPC stream; no inbound runner port is required. Action can proxy onward to other backends. No request may select its own runner-side destination. The service must be running and unambiguous before the first request; incoming HTTP does not start it.

Controller and runner advertise action-http-tunnel-v1, action-http-tunnel-v2 (typed failures/cancellation), and action-http-tunnel-binary-headers-v1 independently. This permits staggered rollout: older peers retain text-only header behavior and may omit non-UTF-8 values; capable peers carry allowed header values as bytes. If a tunnel fails after dispatch, controller does not retry via direct HTTP. Older runners keep direct-endpoint behavior where already supported. Synchronize protobuf contracts in both builds; smoke-test /apps/<module>/... after rollout.

Limits: 1 MiB request/response body, 16 KiB/64 header fields each way, 32 simultaneous HTTP requests per runner, 30-second controller deadline including enqueue, and 25-second backend timeout. 503 means service unavailable or busy; 504 means timeout. WebSockets, SSE, and large uploads are not tunneled. See Action-owned service routes for header, cookie, URL rewrite, and filter behavior.

Security: CE does not authenticate the runner gRPC listener or module app routes; isolate CE’s runner endpoint and do not expose sensitive apps without a trusted boundary. EE requires a runner-* API key in [api_keys] for standalone runners (ordinary UI/API keys cannot register) and applies configured app authentication to /apps/.... Use HTTPS for remote gRPC so runner token is not sent in cleartext.

Managing the Action Queue

The Rescile Portal UI provides a Services & Runners dashboard where administrators can:

• View all actively connected Agent Runners, their hostnames, IP addresses, and capability labels.
• Monitor the live Action Queue to see Pending, Assigned, Running, Completed, or Failed workloads.
• Manually intervene by approving pending jobs, forcefully executing them, or cancelling blocked workloads.
• Trigger actions immediately via Run Action.