Runtime environments (RTE)

What an RTE is, how it runs a playbook, and what it may touch

  1. What an RTE is
  2. How the RTE runs
  3. The security boundary

For administrators: understand the RTE before you run one, size it, or move it to another network.

Page What it covers
Images The three RTE images, and building your own with the collections your playbooks use
Run an RTE Its settings, and a docker run example
Several RTEs Capacity, replicas behind one address, and what happens when an RTE stops or dies
Another datacenter A runbook: add an RTE in a second datacenter, from a bare host to its first job

What an RTE is

AnsibleForms 7 runs no playbook in the app container. The job of an ansible form is handed to a runner of type rte, a runtime environment: the AnsibleForms server started with AF_ROLE=rte, next to python and ansible. It is a small API without a web interface. It claims the job in the database, runs ansible-playbook, and writes the output and the final status back to the database, where the browser reads them. The app resolves the job’s credentials and hands them to the RTE sealed for that RTE and that job only. The RTE never reads the credentials table nor a secret store, and a playbook does not inherit AnsibleForms’ own environment variables (ENCRYPTION_SECRET, DB_*, RTE_TOKEN and the others): pass such a value as an extravar or a credential.

How the RTE runs

The RTE is a container you run yourself, next to the app: AnsibleForms never creates one. This walks through where it runs, what it shares with the app, and what happens inside it from the moment a form is launched until the playbook’s process is gone:

How the RTE runs a playbook On a container host or in a Kubernetes pod, both run by the operator, the app container and one or more RTE containers run side by side. 1: the browser launches a form on the app. 2: the app stores the job in the database. 3: the app's runner seals the job's secrets and posts the job to the RTE API. 4: the RTE claims the job in the database. 5: the RTE starts ansible-playbook as a child process, one per job, up to RTE_MAX_JOBS. 6: the playbook reaches the managed hosts. 7: the RTE writes the output and the final status to the database. 8: the app reads them and the browser shows them. The RTE reads the playbooks, inventories, repositories and ssh key from the app's volume, mounted in it. Container host or Kubernetes pod Docker, Podman or Kubernetes: installed and run by you Browser forms, job page App container web app, API, worker Runners seal the secrets RTE container AF_ROLE=rte, uid 1000 RTE API /rte/v1, with the token ansible-playbook ansible-playbook up to RTE_MAX_JOBS one process per job Managed hosts ssh, WinRM, APIs App's volume playbooks, .ssh key, inventories, repos MySQL / MariaDB jobs, output, heartbeats, runners 1 2 3 4 5 6 7 8 8

Who starts it. An RTE is a long-running container, started by whatever runs your other containers. It waits for jobs and keeps running between them:

How you run AnsibleForms Who starts the RTE
docker run or podman run You, with a command like the one in Run an RTE, on any host that reaches the database
Docker Compose The rte service of docker-compose.yml, next to app and mysqldb. Its config seed declares it as the default runner rte, at http://rte:8000
Helm chart Kubernetes, as a second container named rte in the app’s pod (containers.rte). The app reaches it on 127.0.0.1, and the chart’s seed declares it as the default runner

The container engine is yours: Docker or Podman on the host, or the container runtime of your Kubernetes cluster, installed and kept up to date by you. AnsibleForms never talks to it. Neither the app nor the RTE calls Docker or Podman, mounts their socket, or starts a container: the app only calls the RTE over HTTP, at the address of its runner.

What is inside. One container, with what it takes to run a playbook and nothing more:

  • The server : the same Node.js server as the app, started with AF_ROLE=rte (set in the image) under tini
  • Left out : the web interface, the login, the scheduler and the worker, which this role does not load
  • What it listens on : its PORT (8000): the API under /rte/v1, /live and /ready for probes, /metrics for Prometheus
  • Python and ansible : brought by the image layer you pick (see The RTE images)
  • The user : the server and every playbook run as node (uid 1000), never as root

What it shares with the app. The RTE reads the job from the database and the playbook from disk, so it needs the same database and the same files as the app:

Shared with the app What the RTE does with it
The database (DB_*) Reads the job, claims it, writes its output and final status, writes its heartbeat and its own runner row
RTE_TOKEN Checks it on every call. The app holds the same value on the runner row, and the job’s secrets are sealed with it
ENCRYPTION_SECRET Stores its token encrypted when it registers itself; not needed with RTE_REGISTER=0
The persistent volume, /app/dist/persistent The playbooks folder (ANSIBLE_PATH), or the playbooks repository the worker cloned under REPO_PATH, the inventories (INVENTORY_PATH and the inventory repositories) and the uploads. The RTE writes the job’s extravars files into the playbook folder, so mount it read-write
The ssh folder, /home/node/.ssh The private key and the known_hosts the playbook’s ssh connections use, the ones managed on the app’s settings pages

The RTE clones nothing: a repository is on its disk only because the volume it shares with the app holds it. Collections, roles and ansible.cfg come from the image, from the folders /etc/ansible/collections, /etc/ansible/roles and /etc/ansible/ansible.cfg when you mount them there (the Docker Compose project does), or from an ansible.cfg next to the playbooks.

The life of a job. The numbers below are the numbers in the diagram, for a playbook form launched from the browser or the API:

  1. Launch : the app checks the form and its values, builds the extravars and stores the job with status running. A form with an approval stops here until an approver accepts it, and reaches no RTE before that
  2. Choose the runner : the app takes the RTE the form names with runner: <name>, else the default runner of type RTE, and stores its name on the job. It resolves the job’s credentials from the database or a secret store, its ansible login and its vault password, and seals them with AES-256-GCM under a key derived from that runner’s token and the job id
  3. Hand-over : the app writes Running on RTE <name> (<address>) in the output and calls POST <address>/rte/v1/jobs with the token as a bearer token, the job id, its contract number and the sealed secrets. The RTE refuses another contract (409) and a bundle that does not open, and answers 503 while it starts, stops or is full
  4. Claim : the RTE checks that the job exists, is running and is a playbook, then writes its own name on it in the database: only one RTE ever claims a job, and a repeated call is harmless. It answers 202 and runs the job in the background
  5. Start : the RTE reads the extravars from the job row and resolves a named inventory to its folder. In the playbook folder (plus the form’s playbookSubPath, which must exist) it writes two files: extravars_<id>.json with the form’s values and the credentials, and a hidden one with the ansible login. Then it starts ansible-playbook as its own child process, without a shell, in a process group of its own, with -e @ those files, one -i per inventory, and the tags, limit, check, diff and verbosity of the form. The vault password goes in on stdin, never in a file or on the command line, and the process runs as node with a filtered environment (see below)
  6. Run : the playbook connects to the managed hosts from the RTE container, over ssh with the key and the known_hosts of /home/node/.ssh, with sshpass for a password login, or over the APIs its modules call. The RTE, not the app, needs to reach them
  7. Output and end : the RTE stores stdout and stderr in the database in batches, every half second or 64 KB, with the credentials masked. Past PROCESS_MAX_BUFFER bytes on a stream (50 MB by default) it stores no more, and the playbook goes on. A job log the playbook writes to .joblogs/job_log_<id>.log is copied to the job every 2 seconds, and what set_stats gathers is kept as the job’s artifacts. When the process exits, the RTE writes the last line and the final status (success, failed or aborted), and the form’s notifications go out
  8. Follow : the app waits on the job row, once a second, until its status is final, and asks the RTE about the job every 30 seconds. The browser never talks to the RTE: the form page re-reads the job through the app every 2 seconds, and the Jobs page follows it live

Abort. An abort sets the job’s abort flag in the database and calls POST /rte/v1/jobs/<id>/cancel on its RTE. The RTE sends SIGTERM to the playbook’s process group, so ansible-playbook and every fork it started stop together, and the job ends aborted. When the cancel call does not get through, the RTE still sees the flag, which it checks every 2 seconds while the playbook runs.

Clean-up. As soon as the process exits, whatever the outcome, the RTE removes both extravars files (a form with keepExtravars keeps the visible one, never the hidden one), the artifacts file in /tmp and the job log file. It clears the process id and its name from the job. The end is written once: a second writer, such as the app failing a job the RTE already ended, changes nothing.

One process per job, not one container per job. AWX and AAP start a new pod or container from an execution environment image for every job. AnsibleForms starts nothing: the RTE container is already running, and each job is a child process inside it. A job starts without pulling an image or scheduling a pod, and AnsibleForms needs no rights on your container engine or cluster. In return, the playbooks running on one RTE share its CPU, memory, /tmp, home folder and python: give the container enough resources for RTE_MAX_JOBS playbooks with their forks. To change what a playbook can use, change the image, not the job.

The security boundary

The security boundary. The RTE runs code that form authors wrote, so it is built to hold as little as possible:

  • A token on every call : the API answers 401 to any call without the runner’s token. It has no users, no login and no web pages, and it can only start a playbook job that already exists and is running, report on it, cancel it and report its health. It cannot create a job, return output or hand out credentials. Nothing else restricts who calls it, so keep it on a private network
  • Sealed secrets : to run a job, the RTE never reads the credentials table nor a secret store. It gets the secrets of the jobs it runs, sealed for its token and that job id only, so a bundle meant for another RTE or another job does not open, and a changed byte is refused. They travel safely over plain HTTP inside your network
  • No inherited environment : ansible-playbook gets the RTE’s environment without every variable AnsibleForms reads itself (ENCRYPTION_SECRET, DB_PASSWORD, RTE_TOKEN, VAULT_TOKEN, ACCESS_TOKEN_SECRET, the MYSQL_* ones and the others), so lookup('env', ...) cannot reach them. What you set for playbooks (PATH, proxies, ANSIBLE_*) passes as it is
  • No templates from users : a form value that holds {{, {% or {# reaches the playbook as an unsafe string that ansible never templates, unless the form sets allowJinjaInExtravars, and no shell ever runs the command line
  • Not root : the RTE and its playbooks run as node (uid 1000), so a playbook can do what that user can in the container, such as reading the mounted volume and the ssh key, and no more

The RTE still holds the database password, and ENCRYPTION_SECRET when it registers itself: treat its container, its image and its environment with the same care as the app’s.