Runtime environments (RTE)

What an RTE is, how it runs a playbook, and what it may touch

  1. What an RTE is
  2. How the RTE runs
  3. The security boundary
  4. Database mode (deprecated)

For administrators: understand the RTE before you run one, size it, or move it to another network.

Page What it covers
Images The three RTE images, and building your own with the collections your playbooks use
Modes Outbound, inbound or database: who opens the connection, the spool, the job’s files
Run an RTE The settings of each mode, and a docker run example of each
Several RTEs Capacity, replicas behind one address, and what happens when an RTE stops or dies
Another datacenter A runbook: add an RTE in a second datacenter, from a bare host to its first job

What an RTE is

AnsibleForms 7 runs no playbook in the app container. The job of an ansible form is handed to a runner of type rte, a runtime environment: the AnsibleForms server started with AF_ROLE=rte, next to python and ansible, without a web interface. It talks to the app over one WebSocket and holds no database credentials (RTE_MODE outbound or inbound, see Modes): the app sends it the job, the RTE runs ansible-playbook, and the app stores the output and the final status it sends back, where the browser reads them.

The app resolves the job’s credentials and hands them to the RTE sealed for that RTE and that job only. The RTE never reads the credentials table nor a secret store, and a playbook does not inherit AnsibleForms’ own environment variables (ENCRYPTION_SECRET, DB_*, RTE_TOKEN and the others): pass such a value as an extravar or a credential. The RTE of 7.0 to 7.2, which read and wrote the database itself, still runs in 7.x as the deprecated database mode (see Database mode).

How the RTE runs

The RTE is a container you run yourself, next to the app or in another network: AnsibleForms never creates one. This walks through where it runs, what it shares with the app, and what happens inside it from the moment a form is launched until the playbook’s process is gone:

How the RTE runs a playbook On a container host or in a Kubernetes cluster, both run by the operator, the app container and one or more RTE containers each hold one WebSocket between them, opened by the app (inbound) or by the RTE (outbound). Only the app talks to the database. 1: the browser launches a form on the app. 2: the app stores the job in the database, chooses the runner and seals the job's secrets for it. 3: the app sends the job over the socket. 4: the RTE claims it and gets the job's files, from its cache or from the app. 5: the RTE starts ansible-playbook as a child process, one per job, up to RTE_MAX_JOBS. 6: the playbook reaches the managed hosts. 7: the RTE writes the output, the events and the end to its spool and sends them over the socket; the app stores them in the database and acknowledges them. 8: the browser follows the job through the app. The app's volume is mounted in the RTE only with RTE_FILES=local. Container host or Kubernetes cluster run by you; an outbound RTE may sit in another network Browser forms, job page App container web app, API, worker Runners seal the secrets RTE hub stores, then acks RTE container AF_ROLE=rte, uid 1000 Job files cached by hash ansible-playbook ansible-playbook up to RTE_MAX_JOBS Spool kept until acked one Web- Socket Managed hosts ssh, WinRM, APIs App's volume playbooks, inventories RTE_FILES=local only MySQL / MariaDB the app's only 1 2 3 4 5 6 7 7 8

The diagram shows one socket per RTE whichever side opened it: the app dials an inbound RTE, an outbound RTE dials the app (see Modes). The same messages go both ways once it is open, and only the app reads and writes the database.

Who starts it. An RTE is a long-running container, started by whatever runs your other containers. It waits for jobs and keeps running between them:

How you run AnsibleForms Who starts the RTE
docker run or podman run You, with a command like the one in Run an RTE, on any host that reaches the app, or that the app reaches
Docker Compose The rte service of docker-compose.yml, next to app and mysqldb. Its config seed declares it as the default runner rte, inbound, at http://rte:8000
Helm chart Kubernetes, as a second container named rte in the app’s pod (containers.rte). The app dials it on 127.0.0.1, inbound, and the chart’s seed declares it as the default runner
The installer Docker Compose or the Helm chart as above, or a systemd service without containers; a Runner of its own, which asks to join and waits for an admin’s approval

The container engine is yours: Docker or Podman on the host, or the container runtime of your Kubernetes cluster, installed and kept up to date by you. AnsibleForms never talks to it. Neither the app nor the RTE calls Docker or Podman, mounts their socket, or starts a container: the app only talks to the RTE over its socket.

What is inside. One container, with what it takes to run a playbook and nothing more:

  • The server : the same Node.js server as the app, started with AF_ROLE=rte (set in the image) under tini
  • Left out : the web interface, the login, the scheduler, the worker and every database module, which this role does not load
  • What it listens on : its PORT (8000): /live and /ready for probes and /metrics for Prometheus, and inbound the socket the app dials, /rte/v1/socket. An outbound RTE takes no connection from the app
  • Python and ansible : brought by the image layer you pick (see The RTE images)
  • The user : the server and every playbook run as node (uid 1000), never as root

What it shares with the app. Its token, and with RTE_FILES=local the app’s files. With RTE_FILES=transfer, the default, the playbooks and inventories come with each job, and the RTE needs no shared disk:

Shared with the app What the RTE does with it
RTE_TOKEN Presents it when it dials the app (outbound), or checks it when the app dials it (inbound). The app holds the same value in the runner’s RTE credential, and the job’s secrets are sealed with it
The persistent volume, /app/dist/persistent, with RTE_FILES=local only The playbooks folder (ANSIBLE_PATH), or the playbooks repository the worker cloned under REPO_PATH, the inventories (INVENTORY_PATH and the inventory repositories) and the uploads. The RTE writes the job’s extravars files into the playbook folder, so mount it read-write
The ssh folder, /home/node/.ssh The private key and the known_hosts the playbook’s ssh connections use: mount the app’s, managed on its settings pages, or give the RTE its own

The RTE clones nothing: a repository reaches it in a job’s files, or on the volume it shares with the app. Collections, roles and ansible.cfg come from the image, from the folders /etc/ansible/collections, /etc/ansible/roles and /etc/ansible/ansible.cfg when you mount them there (the Docker Compose project does), or from an ansible.cfg next to the playbooks.

The life of a job. The numbers below are the numbers in the diagram, for a playbook form launched from the browser or the API:

  1. Launch : the app checks the form and its values, builds the extravars and stores the job with status running. A form with an approval stops here until an approver accepts it, and reaches no RTE before that
  2. Choose the runner : the app takes the RTE the form names with runner: <name>, else the default runner of type RTE, and stores its name on the job. It resolves the job’s credentials from the database or a secret store, its ansible login and its vault password, and seals them with AES-256-GCM under a key derived from that runner’s token and the job id
  3. Hand-over : the app node that holds the RTE’s socket sends it the job: the extravars, the form’s options, the sealed secrets and the hashes of the job’s files. Any app node may launch the job; the one holding the socket sends it within two seconds. A full RTE, or one not connected, makes the job wait, up to RTE_QUEUE_MINUTES
  4. Claim : the RTE answers claimed, or refused when it is full, stopping, cannot open the secrets, or the job comes with another contract than its own. With RTE_FILES=transfer it takes the playbooks and the inventories from its cache by their hash, asks the app for a hash it does not have, and unpacks them in a folder of the job’s own
  5. Start : in the playbook folder (plus the form’s playbookSubPath, which must exist) it writes two files: extravars_<id>.json with the form’s values and the credentials, and a hidden one with the ansible login. Then it starts ansible-playbook as its own child process, without a shell, in a process group of its own, with -e @ those files, one -i per inventory, and the tags, limit, check, diff and verbosity of the form. The vault password goes in on stdin, never in a file or on the command line, and the process runs as node with a filtered environment (see below)
  6. Run : the playbook connects to the managed hosts from the RTE container, over ssh with the key and the known_hosts of /home/node/.ssh, with sshpass for a password login, or over the APIs its modules call. The RTE, not the app, needs to reach them
  7. Output and end : the RTE batches stdout and stderr every half second or 64 KB, with the credentials masked, and numbers each batch. It writes every item to the job’s spool on its disk before it sends it: the output, the events, the job log of .joblogs/job_log_<id>.log (every 2 seconds), the start and the end with the final status (success, failed or aborted). The app stores each item once and acknowledges it, and the RTE drops the spool once the end is stored. Past PROCESS_MAX_BUFFER bytes on a stream (50 MB by default) it sends no more, and the playbook goes on. The app then sends the form’s notifications
  8. Follow : the browser never talks to the RTE: the form page re-reads the job through the app every 2 seconds, and the Jobs page follows it live as the app stores each item

A dropped socket. The playbook goes on. The RTE redials (outbound), or an app node dials it again (inbound), and sends again what the app had not acknowledged: the app skips what it stored already, so no line is lost or stored twice. That holds while the socket is back within two minutes: the app writes the RTE’s heartbeat only while its socket is open, and once that heartbeat is two minutes old the worker ends the RTE’s running jobs as abandoned, even if their playbooks still run. Keep the spool (RTE_SPOOL_PATH) on a volume, so a restarted RTE still sends what it had kept. A playbook that was running when its RTE restarted cannot go on: the app ends its job as abandoned when the RTE comes back without it.

Abort. An abort sets the job’s abort flag, and the app node that holds the RTE’s socket sends cancel within two seconds. The RTE sends SIGTERM to the playbook’s process group, so ansible-playbook and every fork it started stop together, and the job ends aborted. A cancel that arrives before its job is kept for a minute, and the job then starts aborted.

Clean-up. As soon as the process exits, whatever the outcome, the RTE removes both extravars files (a form with keepExtravars keeps the visible one, never the hidden one), the artifacts file in /tmp, the job log file and the job’s own folder of files. The end is stored once: a second writer, such as the app failing a job the RTE already ended, changes nothing.

One process per job, not one container per job. AWX and AAP start a new pod or container from an execution environment image for every job. AnsibleForms starts nothing: the RTE container is already running, and each job is a child process inside it. A job starts without pulling an image or scheduling a pod, and AnsibleForms needs no rights on your container engine or cluster. In return, the playbooks running on one RTE share its CPU, memory, /tmp, home folder and python: give the container enough resources for RTE_MAX_JOBS playbooks with their forks. To change what a playbook can use, change the image, not the job.

The security boundary

The RTE runs code that form authors wrote, so it is built to hold as little as possible:

  • Its token only : the RTE holds no database credentials and no ENCRYPTION_SECRET. It takes no call but the app’s socket, with its token, and has no users, no login and no web pages. Over the socket it can only run the jobs the app sends it, report on them and ask for their files: it cannot create a job, read another job or hand out credentials
  • Sealed secrets : to run a job, the RTE never reads the credentials table nor a secret store. It gets the secrets of the jobs it runs, sealed for its token and that job id only, so a bundle meant for another RTE or another job does not open, and a changed byte is refused
  • A checked connection : outbound, the RTE checks the app’s certificate. Inbound over HTTPS, the app pins the RTE’s own certificate when it joins, and dials it only when it presents that one (see Modes). An inbound RTE over plain HTTP belongs on a private network, such as the Docker Compose project’s
  • No inherited environment : ansible-playbook gets the RTE’s environment without every variable AnsibleForms reads itself (ENCRYPTION_SECRET, DB_PASSWORD, RTE_TOKEN, VAULT_TOKEN, ACCESS_TOKEN_SECRET, the MYSQL_* ones and the others), so lookup('env', ...) cannot reach them. What you set for playbooks (PATH, proxies, ANSIBLE_*) passes as it is
  • No templates from users : a form value that holds {{, {% or {# reaches the playbook as an unsafe string that ansible never templates, unless the form sets allowJinjaInExtravars, and no shell ever runs the command line
  • Not root : the RTE and its playbooks run as node (uid 1000), so a playbook can do what that user can in the container, such as reading its files and the ssh key, and no more

Treat its container, its image and its environment with the same care as the app’s.

Database mode (deprecated)

The RTE of 7.0 to 7.2, without RTE_MODE, runs all through 7.x and is removed in 8.0.0. It differs from the steps above in four ways:

  • It holds the database : its DB_* settings, and ENCRYPTION_SECRET when it registers itself (RTE_REGISTER)
  • The hand-over is an HTTP call : the app calls POST <address>/rte/v1/jobs with the token and the sealed secrets, and the RTE claims the job by writing its name on it in the database (steps 3 and 4)
  • It writes the output itself : in the database, in batches, with no spool (step 7). An abort reaches it through the job’s abort flag, which it reads every 2 seconds, and a /cancel call
  • It reads the files from the app’s volume : it needs the same disk as the app, as with RTE_FILES=local

Its runner page has none of the RTE settings: it reads its own variables. To move it to a socket mode, see Modes.