Automating administration
The scope of Ansible and the repository layout, PowerShell scripts run from Task Scheduler, and fulfilling requests through AWX or Ansible Automation Platform.
Automation of administration most often fails not for lack of tools, but for lack of boundaries and scaffolding: a playbook takes over tasks Ansible is not meant for, a script without a log and exit code shows success in the scheduler despite an error, and repetitive requests wait days for manual handling. This document covers three related elements: the scope of Ansible and the repository layout, the structure of an unattended PowerShell script, and the flow from a ticket in an ITSM system to a completed task in AWX or Ansible Automation Platform.
The scope of Ansible
Ansible connects to a machine over SSH, or to Windows hosts over WinRM or PSRP, runs a list of tasks and finishes. This model determines where it fits.
| Task | Ansible | Better tool or note |
|---|---|---|
| Packages, configuration files, services, accounts, firewall rules | Yes | The core use; modules are idempotent |
| Operational tasks across many machines: certificate rotation, updates in a fixed order | Yes | serial limits batch size, order sets the sequence, --limit the scope |
| API integrations: image registry, ticketing system, CI platform | Yes | The ansible.builtin.uri module or a dedicated collection |
| Windows Server where GPO does not reach | Yes | WinRM or PSRP and the ansible.windows collection; workstation settings stay in GPO |
| Maintaining state over time and correcting drift | No | A controller that continuously reconciles state |
| Day-to-day management of Kubernetes resources | No | Manifests in Git and a GitOps controller |
| Creating machines, networks and volumes | Usually not | Ansible has modules for it, but Terraform or OpenTofu provides a plan of changes and state |
| Complex logic and data transformation | No | A Python script called as a single task |
Ansible can prepare a cluster or install a GitOps controller in it, but does not run it day to day: the kubernetes.core.k8s module applies state idempotently, but does not reconcile it continuously and leaves no change history in the repository. A playbook with loops inside loops, conditions that depend on the results of earlier tasks and multi-line Jinja2 filters is a program written in the wrong language, which can be neither read nor tested. Such logic moves into a Python script with its own tests, and the playbook calls it as a single task and checks the result.
Repository layout and roles
The directory structure is the first piece of documentation, so the layout is fixed for every project.
| Path | Contents |
|---|---|
ansible.cfg | default inventory and roles_path |
requirements.txt | pinned versions of ansible-core and ansible-lint |
collections/requirements.yml | collections with pinned versions |
inventories/prod/, inventories/test/ | hosts.yml with machines and groups, and group_vars/ |
inventories/<environment>/group_vars/all/vars.yml | shared variables |
inventories/<environment>/group_vars/all/vault.yml | secrets, encrypted |
inventories/<environment>/group_vars/<group>.yml | per-group differences, e.g. web.yml |
playbooks/ | thin playbooks: target group and list of roles, e.g. web.yml |
roles/<role>/ | defaults/main.yml, tasks/main.yml, handlers/main.yml, templates/, README.md |
README.md | how to run, where the secrets are |
A playbook contains no tasks, only the target group (hosts: web) and a list of roles; the logic lives in roles. In ansible.cfg the default inventory points to the test environment (inventory = inventories/test/hosts.yml) and roles_path = roles to the roles directory. A test inventory as the default is a cheap safeguard: a command run without -i hits test machines, and production requires a deliberate -i inventories/prod/hosts.yml. The roles_path entry is needed because playbooks sit in a subdirectory, and without it Ansible looks for roles in a roles/ directory next to the playbook file and in the default paths.
A role installs and configures one thing: a web server, a monitoring agent, time synchronisation. An “application server” role that does twenty things is reusable and testable in neither respect. Every role variable has a default value and a comment in defaults/main.yml (e.g. nginx_worker_processes: auto); this is the role’s contract, because using it requires reading one file. The role’s tasks install a package, deploy a template and start the service, and a handler reloads it only when the template has actually changed. Fully qualified module names (ansible.builtin.template instead of template) remove any doubt about which collection a module comes from.
Role variables begin with the role’s name: nginx_worker_processes, not worker_processes. Without a prefix two roles will sooner or later use the same name for different purposes and the value with higher precedence wins; the ansible-lint rule var-naming[no-role-prefix] enforces the convention.
Variables, secrets and idempotency
Ansible has 22 levels of variable precedence, so in practice three places are enough: defaults/main.yml in a role, and group_vars/ and host_vars/ in the inventory. Passwords and keys do not sit in the repository as plain text: Ansible Vault is the baseline (an encrypted group_vars/all/vault.yml with variables prefixed vault_), and when the same secrets are needed in several tools, an external secrets manager.
A playbook run twice in a row reports zero changes the second time, and every changed in the second run is a task to fix. The package, template and service modules guarantee this by themselves. The problem comes from command and shell, which Ansible always treats as a change:
createswith the path of a marker file skips the task when the file already exists (initialising an application’s data directory),changed_when: falsetogether withcheck_mode: falsemarks a read-only task that also runs in--checkmode (reading the schema migration status),whenon a registered result (register) runs the changing task only when the read showed a need, e.g. pending migrations.
--check --diff simulates a run without changes and shows what would change in files, preferably with --limit to a single server. Modules without dry-run support report nothing, so this is a prediction, not a guarantee. ansible-inventory --host <host> prints the variables from group_vars and host_vars after merging, which answers the question of which value won.
The minimum that can be maintained without a separate test environment: on every change in the repository, ansible-playbook --syntax-check and ansible-lint run, with versions pinned in requirements.txt. In GitLab CI this is a single job on a Python image: pip install -r requirements.txt, ansible-galaxy collection install -r collections/requirements.yml, then both check commands against the test inventory. The basic profile (set in .ansible-lint as profile: basic) covers typical coding errors and standard style without burying an older repository under hundreds of findings. Once the code is tidy, it is raised to moderate and then to production.
A PowerShell script run from Task Scheduler
Task Scheduler shows 0x0, that is success, even when the script did nothing it was supposed to do, for example after a drive letter changed and the error was printed where nobody sees it. An unattended script is a production service and needs the same as a service: declared requirements, validated input, a dry-run mode, a log, a readable result and no passwords in the code. The script passes through four stages in turn: parameters with validation attributes, -WhatIf mode (ShouldProcess), the action in a try block, and a log with an exit code. An error from any stage goes to catch, and every outcome, including a failure, ends with a log entry and a non-zero exit code.
Script skeleton
The script logic is deliberately simple; what matters is the scaffolding, which can be copied into every script run from a scheduler.
#Requires -Version 5.1
<#
.SYNOPSIS
Deletes export files older than the given number of days.
.DESCRIPTION
Run from Task Scheduler. Exit codes: 0 success,
1 critical error, 2 some files were not deleted.
.EXAMPLE
.\Remove-OldExports.ps1 -Path D:\Export -OlderThanDays 30 -WhatIf
#>
[CmdletBinding(SupportsShouldProcess)]
param(
[Parameter(Mandatory)]
[ValidateScript({ Test-Path -LiteralPath $_ -PathType Container })]
[string]$Path,
[ValidateRange(1, 3650)]
[int]$OlderThanDays = 30,
[ValidateNotNullOrEmpty()]
[string]$LogDirectory = 'C:\ProgramData\Scripts\Logs'
)
$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest
# The log is written in -WhatIf mode too
New-Item -ItemType Directory -Path $LogDirectory -Force -WhatIf:$false | Out-Null
$logFile = Join-Path $LogDirectory ('Remove-OldExports_{0:yyyyMMdd}.log' -f (Get-Date))
function Write-Log {
param(
[string]$Message,
[ValidateSet('INFO', 'WARN', 'ERROR')][string]$Level = 'INFO'
)
$line = '{0:yyyy-MM-dd HH:mm:ss} [{1}] {2}' -f (Get-Date), $Level, $Message
Add-Content -LiteralPath $logFile -Value $line -Encoding UTF8 -WhatIf:$false
Write-Verbose $line
}
$exitCode = 0
try {
Write-Log "Start: $Path, older than $OlderThanDays days, WhatIf: $WhatIfPreference"
$cutoff = (Get-Date).AddDays(-$OlderThanDays)
$files = @(Get-ChildItem -LiteralPath $Path -File | Where-Object LastWriteTime -lt $cutoff)
Write-Log "Files to delete: $($files.Count)"
foreach ($file in $files) {
if ($PSCmdlet.ShouldProcess($file.FullName, 'Delete file')) {
try {
Remove-Item -LiteralPath $file.FullName
Write-Log "Deleted: $($file.FullName)"
}
catch {
Write-Log "Not deleted: $($file.FullName): $($_.Exception.Message)" -Level WARN
$exitCode = 2
}
}
}
}
catch {
Write-Log "Error: $($_.Exception.Message)" -Level ERROR
$exitCode = 1
}
finally {
Write-Log "End, exit code: $exitCode"
}
exit $exitCode
Header and parameters with validation
#Requires checks conditions before the first line runs: -Version 5.1, -Modules ActiveDirectory, -RunAsAdministrator. The script then ends with one clear message instead of an error halfway through. The help block with .SYNOPSIS, .DESCRIPTION and .EXAMPLE is documentation that does not drift away from the code (Get-Help .\Remove-OldExports.ps1 -Full), and the description of exit codes answers the first question on seeing 0x2 in the scheduler.
Validation attributes work before the script does anything: a wrong value aborts the run with a message that names the parameter.
| Attribute | What it checks | Example use |
|---|---|---|
[Parameter(Mandatory)] | the parameter must be supplied; in -NonInteractive mode its absence ends in an error | directory path |
[ValidateScript({ … })] | any condition returning true or false | Test-Path -LiteralPath $_ -PathType Container |
[ValidateRange(1, 3650)] | numeric range | number of days |
[ValidateSet('Archive', 'Delete')] | a value from a list, with completion in the console | operating mode |
[ValidatePattern('^srv-[a-z]+-\d{2}$')] | regular expression | server name |
[ValidateNotNullOrEmpty()] | no empty value | log directory |
The -WhatIf mode
[CmdletBinding(SupportsShouldProcess)] adds the -WhatIf and -Confirm parameters to the script. Every change is wrapped in $PSCmdlet.ShouldProcess(target, operation), which in -WhatIf mode returns false and prints what would be done. What the script would do in production can therefore be checked before it is allowed to do it. The Ansible counterpart is --check mode.
Errors and the log
The default value of $ErrorActionPreference is Continue. Many command errors, for example Get-ChildItem on a non-existent path, are then non-terminating: PowerShell prints a message and runs the following lines, and the catch block does not run. Hence the 0x0 result despite the error. $ErrorActionPreference = 'Stop' turns every error into an exception that reaches catch. Set-StrictMode -Version Latest adds a second safeguard: a reference to a non-existent variable is an error, so a typo does not silently turn into an empty value.
Start-Transcript -Path <file> -Append is the simplest route for an existing script, but it produces a verbose log that is hard to search. A custom function such as Write-Log in the skeleton writes one line per event with a date and a level, which is easy to compare across days and to send to a central log system. Write-Verbose is for details visible on a manual run with -Verbose.
Exit codes for the scheduler
Task Scheduler does not read the log. It sees only the code with which the powershell.exe process ended, and shows it in hexadecimal.
| Code in the script | Meaning | Result in the scheduler |
|---|---|---|
exit 0 | everything done | 0x0 |
exit 1 | critical error, work aborted | 0x1 |
exit 2 | partly done, details in the log | 0x2 |
| none (set by the scheduler) | the task has not run yet | 0x41303 |
The code reaches the scheduler when the script is run with -File; with -Command it can be flattened to 0 or 1. The task is registered with New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-NoProfile -NonInteractive -File ...', New-ScheduledTaskTrigger and New-ScheduledTaskPrincipal with a gMSA account (-LogonType Password), followed by Register-ScheduledTask. -NonInteractive makes every prompt for a missing parameter end in an error instead of hanging the task forever.
Secrets in a script
When run from the scheduler, the script runs under a group managed service account (gMSA), e.g. gmsa-scripts$, which itself holds the permissions to the share or directory objects (for a manual run Get-Credential is enough). Nobody knows the password, and the domain changes it automatically, by default every 30 days. The account is created with New-ADServiceAccount (this requires a KDS root key in the domain), and on the server it has to be installed (Install-ADServiceAccount) and verified (Test-ADServiceAccount). The account needs the “Log on as a batch job” right, and the server must be listed in PrincipalsAllowedToRetrieveManagedPassword.
Tokens and keys for external APIs come from the Microsoft.PowerShell.SecretManagement module with a vault extension, e.g. Microsoft.PowerShell.SecretStore, or from an external vault such as Azure Key Vault. Only a reference stays in the script (Get-Secret -Name 'ExportApiToken' -Vault 'Automation' -AsPlainText). SecretStore is tied to a user, so it is configured in the context of the account that runs the task, with Interaction set to None. The SecretManagement and SecretStore modules are considered complete and receive only security fixes.
ConvertTo-SecureString 'Haslo123' -AsPlainText -Force in the body of a script does not protect the password, it only changes its type: the password sits in plain text in the file, in the repository and in every backup. The same applies to passwords in the arguments of a scheduled task.
Quality and signing
PSScriptAnalyzer (Invoke-ScriptAnalyzer) catches aliases, plain-text passwords and state-changing functions without ShouldProcess, so code passes through it before deployment. The execution policy is not a security boundary, but with AllSigned a signature (Set-AuthenticodeSignature, a code-signing certificate trusted on the servers) works as change control: editing the file on the server invalidates the signature, so fixes have to go through the repository.
Fulfilling requests through automation
A repetitive request, for example read-only access to a share with reports, is a few minutes of work and days of waiting in a queue. The waiting time does not come from the difficulty of the task, but from the fact that a person has to notice it, read it and handle it by hand. This step is removed by connecting the ticketing system (ITSM) to an automation platform such as AWX or Ansible Automation Platform. The pattern has five steps:
- A form in the ticketing system with fixed fields: a list of shares, a user picker from the directory, a read or modify toggle, not a “describe what you need” field.
- Approval in the same system, by a manager or the resource owner, according to existing rules.
- An API call to the automation platform after approval, with the form content as variables.
- The automation, that is the playbook, checks the data, makes the change and collects the result.
- The result written back into the ticket: what was done, or what error occurred. A successful ticket closes itself; a failed one returns to the queue for a person.
A person appears in two places: at approval and on error.
The form
Every form field becomes a variable in the playbook, so from the start it is designed together with its validation rule.
| Form field | Variable | Form type | Check in the playbook |
|---|---|---|---|
| Ticket number | request_id | filled in automatically | matches a pattern, e.g. REQ- and six digits |
| Share | share_name | selection list | name is on the list of permitted shares |
| User | user_sam | picked from the directory | permitted characters and account name length |
| Access level | access_level | toggle | only read or modify |
| Dry run | dry_run | hidden, set by IT | boolean |
The list of permitted shares lives in the project repository, in a file next to the playbook (allowed_shares.yml, e.g. Finance, HR, Projects), not in the form or in variables: the form helps to choose, but what is permitted is decided by the automation. Access is granted through group membership, not by adding the account to the share’s permissions, so the automation needs the right to change group membership, not to change ACLs on the file server.
Launching a job template through the API
In AWX and Ansible Automation Platform a playbook is launched through a job template, which ties together the playbook, inventory and credentials. After approval the ticketing system sends a single request, POST /api/v2/job_templates/<id>/launch/, with an Authorization: Bearer <token> header, the application/json type and the body {"extra_vars": {"request_id": "REQ-004211", "share_name": "Finance", "user_sam": "jkowalska", "access_level": "read", "dry_run": false}}. The response contains the identifier of the launched job (the job field), which the ticketing system stores with the request. The job state is read from GET /api/v2/jobs/<id>/, and the status field takes values including successful, failed, error and canceled. In Ansible Automation Platform 2.5 and later the same controller API is available under the path /api/controller/v2/.
The template accepts extra_vars from the request only when it has prompting for variables on launch (ask_variables_on_launch) enabled or a survey with those fields. Otherwise the platform launches the job without them and lists the omitted fields in ignored_fields. At the same time extra_vars have the highest precedence in Ansible and override any variable defined in the playbook, inventory or role. The safe configuration is a survey with the form fields (survey_enabled: true) with ask_variables_on_launch disabled: only the survey fields get through. Both settings at false mean that variables from the request are ignored.
Playbook: validation, change, result
The form can be changed or bypassed, or the API called directly, so the playbook checks every field itself. The whole request is one block: after validation and the change, the playbook writes the result into the ticket with the ansible.builtin.uri module (a note and a status change to resolved, or open in dry-run mode), delegated to localhost because the module runs only on a POSIX platform; for a Windows host the equivalent is ansible.windows.win_uri. If any step fails, the rescue section writes the error into the ticket, passes it to the queue for people and fails the job through ansible.builtin.fail. The ticketing system’s API address and the field names in body depend on the specific product; the awx_job_id variable in the note ties the ticket to the full job log.
- name: Share access based on a ticket
hosts: ad_management
gather_facts: false
tasks:
- name: Handle the request
block:
- name: Validate the form data
ansible.builtin.assert:
that:
- request_id is match('^REQ-[0-9]{6}$')
# list from a file, not a variable: extra_vars cannot override it
- share_name in (lookup('ansible.builtin.file', 'allowed_shares.yml') | from_yaml)
- access_level in ['read', 'modify']
- user_sam is match('^[a-z][a-z0-9.]{1,19}$')
fail_msg: The request data did not pass validation
quiet: true
- name: Add the user to the access group
microsoft.ad.group:
name: "GG-Share-{{ share_name }}-{{ access_level }}"
members:
add:
- "{{ user_sam }}"
state: present
check_mode: "{{ dry_run | bool }}"
register: share_access_result
- name: Write the result into the ticket
ansible.builtin.uri:
url: "https://itsm.example.com/api/requests/{{ request_id }}/notes"
method: POST
headers:
Authorization: "Bearer {{ itsm_token }}"
body_format: json
body:
status: "{{ 'open' if dry_run | bool else 'resolved' }}"
note: "{{ 'Dry run, access was not granted' if dry_run | bool else 'Access granted' }}, AWX job {{ awx_job_id }}"
delegate_to: localhost
no_log: true
rescue:
- name: Write the error into the ticket and pass it to people
ansible.builtin.uri:
url: "https://itsm.example.com/api/requests/{{ request_id }}/notes"
method: POST
headers:
Authorization: "Bearer {{ itsm_token }}"
body_format: json
body:
status: open
assigned_group: it-service-desk
note: "Error: {{ ansible_failed_result.msg | default('unknown') }}, AWX job {{ awx_job_id }}"
when: request_id | default('') is match('^REQ-[0-9]{6}$')
delegate_to: localhost
no_log: true
- name: Fail the job
ansible.builtin.fail:
msg: The request was not fulfilled, details in the ticket
Decisions visible in the example:
assertcomes before every change, so invalid data stops the block before the automation touches the directory. In therescuesection the ticket number goes into the URL only if it passed validation, because it is external data too. With an invalidrequest_idthe only signal is the job status in AWX, which the ticketing system polls and uses to mark the request as needing a person.- The
microsoft.ad.groupmodule withmembers.addis idempotent: if the user is already in the group, the task ends with no change, so re-running the same request is not an error and creates no duplicates. - The list of permitted shares is read from a file directly in the condition, with no variable name that could be substituted through
extra_vars.
Dry run, permissions and measures
For the first weeks a new automation runs with dry_run: true: it validates the data, checks in check_mode what it would change and records this in the ticket, while a person fulfils the request and compares the report with their own action. The exit criterion (e.g. twenty consecutive matching requests) is set before the start.
Three identities take part in the flow, and each gets only what it needs:
- the ticketing system’s token in AWX: a separate user with the Execute role on a single template and a
writescope (launching a job is a write operation), with no access to other templates, inventories or credentials, - the account the playbook uses in Active Directory: the “Modify the membership of a group” delegation on the organisational unit holding the share access groups, not a domain administrator,
- the ticketing system’s API token used by the playbook: adding notes and changing status, with no administrative permissions.
The first automation handles the request with the highest volume and the least costly errors, usually access to a share or a distribution group, not the most complicated one. Two numbers are measured: the time from submitting a request to its fulfilment, and the share of requests of a given type completed with no human involvement. The first shows the effect for users, the second where the automation still fails and why.
Decisions and alternatives
| Approach | When it makes sense | Limitation |
|---|---|---|
| Ansible for configuration and multi-machine tasks | packages, services, accounts, rotations, API integrations | does not maintain state over time; --check is a prediction, not a guarantee |
| Terraform or OpenTofu | creating machines, networks and volumes | a plan of changes and state, but not configuration inside the system |
| GitOps controller | day-to-day management of Kubernetes resources | requires manifests in Git; Ansible at most installs the controller |
| Python script called as a single task | complex logic and data transformation | needs its own tests outside the playbook |
| gMSA account for a scheduled task | a script with access to a share or directory | requires a KDS key, the batch logon right and the server listed in PrincipalsAllowedToRetrieveManagedPassword |
| SecretManagement with SecretStore | tokens and keys for external APIs | store tied to a user; configured in the context of the task account |
Template survey with ask_variables_on_launch: false | launching from the ticketing system | only the survey fields get through |
ask_variables_on_launch: true | arbitrary variables, trusted caller | the caller overrides any playbook variable, validation included |
Script signing (AllSigned) | scripts on servers under change control | requires a code-signing certificate trusted on the servers |
Implementation
- Define the boundaries: Ansible for configuration and multi-machine tasks, Terraform for creating resources, a GitOps controller for Kubernetes, complex logic in a Python script. Check of effects: playbooks contain no
commandtasks with Jinja2 logic and noshell: kubectl apply. - Repository layout according to the path table, with the test inventory as the default and secrets in Vault. Check: every Vault file encrypted (
grep -L '^\$ANSIBLE_VAULT'on thevault.ymlfiles). - A CI job with
ansible-playbook --syntax-checkandansible-linton thebasicprofile, raised gradually toproduction. Check: a second run of the playbook reports nochanged. - Scheduler scripts built on the skeleton with validation,
-WhatIf, a log and an exit code, run with-Fileunder a gMSA account. Check: tasks outside the Microsoft folder (Get-ScheduledTaskwithGet-ScheduledTaskInfo) without error codes inLastTaskResult(SCHED_S_ codes are states, not errors), running on schedule, under SYSTEM or a gMSA, with no personal passwords stored in tasks. - An AWX job template with a survey and
ask_variables_on_launch: false, a separate user with the Execute role and a token with thewritescope. Check: reading the template configuration throughGET /api/v2/job_templates/<id>/(a token with thereadscope is enough) showstrueandfalsein thesurvey_enabledandask_variables_on_launchfields. - The first request type with
dry_run: trueuntil the exit criterion is met, then switching to execution and measuring fulfilment time and the share of requests handled without a person.
Limitations and pitfalls
-WhatIfwithout a log.-WhatIfpropagates to every command in the script that supports it, includingNew-ItemandAdd-Content, which write the log. Without-WhatIf:$falseon those two commands a dry run leaves no log, and the log is its result.- Exit codes of external programs.
$ErrorActionPreferencedoes not cover the exit codes of programs, so after callingrobocopy, for example,$LASTEXITCODEhas to be checked; for robocopy values of 8 and above mean an error (if ($LASTEXITCODE -ge 8) { throw "robocopy: code $LASTEXITCODE" }). In PowerShell 7.4 and later$PSNativeCommandUseErrorActionPreferenceset to$trueturns non-zero codes into errors, but robocopy uses them for success too, and Windows PowerShell 5.1 does not have this variable. - Script file encoding. Windows PowerShell 5.1 reads a file without a BOM in the system code page and turns non-ASCII characters in messages into random symbols; scripts containing such characters are saved as UTF-8 with BOM.
- Validation with the list in a variable. Validation in the form is a convenience for the user, not a safeguard: anyone holding a token can launch the template with arbitrary values. A list of permitted values stored as a playbook variable is overridden by
extra_varswhenask_variables_on_launch: true, along with all validation, so it is read from a file and input data is treated as untrusted. rescuewithoutfail. A block whose error was handled ends the job with thesuccessfulstatus, so a ticketing system that checks only the status will not notice the problem. A finalansible.builtin.failinrescuerestores the failure status.