Automating administration

The scope of Ansible and the repository layout, PowerShell scripts run from Task Scheduler, and fulfilling requests through AWX or Ansible Automation Platform.

AutomationWindows Server

Automation of administration most often fails not for lack of tools, but for lack of boundaries and scaffolding: a playbook takes over tasks Ansible is not meant for, a script without a log and exit code shows success in the scheduler despite an error, and repetitive requests wait days for manual handling. This document covers three related elements: the scope of Ansible and the repository layout, the structure of an unattended PowerShell script, and the flow from a ticket in an ITSM system to a completed task in AWX or Ansible Automation Platform.

The scope of Ansible

Ansible connects to a machine over SSH, or to Windows hosts over WinRM or PSRP, runs a list of tasks and finishes. This model determines where it fits.

TaskAnsibleBetter tool or note
Packages, configuration files, services, accounts, firewall rulesYesThe core use; modules are idempotent
Operational tasks across many machines: certificate rotation, updates in a fixed orderYesserial limits batch size, order sets the sequence, --limit the scope
API integrations: image registry, ticketing system, CI platformYesThe ansible.builtin.uri module or a dedicated collection
Windows Server where GPO does not reachYesWinRM or PSRP and the ansible.windows collection; workstation settings stay in GPO
Maintaining state over time and correcting driftNoA controller that continuously reconciles state
Day-to-day management of Kubernetes resourcesNoManifests in Git and a GitOps controller
Creating machines, networks and volumesUsually notAnsible has modules for it, but Terraform or OpenTofu provides a plan of changes and state
Complex logic and data transformationNoA Python script called as a single task

Ansible can prepare a cluster or install a GitOps controller in it, but does not run it day to day: the kubernetes.core.k8s module applies state idempotently, but does not reconcile it continuously and leaves no change history in the repository. A playbook with loops inside loops, conditions that depend on the results of earlier tasks and multi-line Jinja2 filters is a program written in the wrong language, which can be neither read nor tested. Such logic moves into a Python script with its own tests, and the playbook calls it as a single task and checks the result.

Repository layout and roles

The directory structure is the first piece of documentation, so the layout is fixed for every project.

PathContents
ansible.cfgdefault inventory and roles_path
requirements.txtpinned versions of ansible-core and ansible-lint
collections/requirements.ymlcollections with pinned versions
inventories/prod/, inventories/test/hosts.yml with machines and groups, and group_vars/
inventories/<environment>/group_vars/all/vars.ymlshared variables
inventories/<environment>/group_vars/all/vault.ymlsecrets, encrypted
inventories/<environment>/group_vars/<group>.ymlper-group differences, e.g. web.yml
playbooks/thin playbooks: target group and list of roles, e.g. web.yml
roles/<role>/defaults/main.yml, tasks/main.yml, handlers/main.yml, templates/, README.md
README.mdhow to run, where the secrets are

A playbook contains no tasks, only the target group (hosts: web) and a list of roles; the logic lives in roles. In ansible.cfg the default inventory points to the test environment (inventory = inventories/test/hosts.yml) and roles_path = roles to the roles directory. A test inventory as the default is a cheap safeguard: a command run without -i hits test machines, and production requires a deliberate -i inventories/prod/hosts.yml. The roles_path entry is needed because playbooks sit in a subdirectory, and without it Ansible looks for roles in a roles/ directory next to the playbook file and in the default paths.

A role installs and configures one thing: a web server, a monitoring agent, time synchronisation. An “application server” role that does twenty things is reusable and testable in neither respect. Every role variable has a default value and a comment in defaults/main.yml (e.g. nginx_worker_processes: auto); this is the role’s contract, because using it requires reading one file. The role’s tasks install a package, deploy a template and start the service, and a handler reloads it only when the template has actually changed. Fully qualified module names (ansible.builtin.template instead of template) remove any doubt about which collection a module comes from.

Role variables begin with the role’s name: nginx_worker_processes, not worker_processes. Without a prefix two roles will sooner or later use the same name for different purposes and the value with higher precedence wins; the ansible-lint rule var-naming[no-role-prefix] enforces the convention.

Variables, secrets and idempotency

Ansible has 22 levels of variable precedence, so in practice three places are enough: defaults/main.yml in a role, and group_vars/ and host_vars/ in the inventory. Passwords and keys do not sit in the repository as plain text: Ansible Vault is the baseline (an encrypted group_vars/all/vault.yml with variables prefixed vault_), and when the same secrets are needed in several tools, an external secrets manager.

A playbook run twice in a row reports zero changes the second time, and every changed in the second run is a task to fix. The package, template and service modules guarantee this by themselves. The problem comes from command and shell, which Ansible always treats as a change:

  • creates with the path of a marker file skips the task when the file already exists (initialising an application’s data directory),
  • changed_when: false together with check_mode: false marks a read-only task that also runs in --check mode (reading the schema migration status),
  • when on a registered result (register) runs the changing task only when the read showed a need, e.g. pending migrations.

--check --diff simulates a run without changes and shows what would change in files, preferably with --limit to a single server. Modules without dry-run support report nothing, so this is a prediction, not a guarantee. ansible-inventory --host <host> prints the variables from group_vars and host_vars after merging, which answers the question of which value won.

The minimum that can be maintained without a separate test environment: on every change in the repository, ansible-playbook --syntax-check and ansible-lint run, with versions pinned in requirements.txt. In GitLab CI this is a single job on a Python image: pip install -r requirements.txt, ansible-galaxy collection install -r collections/requirements.yml, then both check commands against the test inventory. The basic profile (set in .ansible-lint as profile: basic) covers typical coding errors and standard style without burying an older repository under hundreds of findings. Once the code is tidy, it is raised to moderate and then to production.

A PowerShell script run from Task Scheduler

Task Scheduler shows 0x0, that is success, even when the script did nothing it was supposed to do, for example after a drive letter changed and the error was printed where nobody sees it. An unattended script is a production service and needs the same as a service: declared requirements, validated input, a dry-run mode, a log, a readable result and no passwords in the code. The script passes through four stages in turn: parameters with validation attributes, -WhatIf mode (ShouldProcess), the action in a try block, and a log with an exit code. An error from any stage goes to catch, and every outcome, including a failure, ends with a log entry and a non-zero exit code.

Script skeleton

The script logic is deliberately simple; what matters is the scaffolding, which can be copied into every script run from a scheduler.

#Requires -Version 5.1
<#
.SYNOPSIS
    Deletes export files older than the given number of days.
.DESCRIPTION
    Run from Task Scheduler. Exit codes: 0 success,
    1 critical error, 2 some files were not deleted.
.EXAMPLE
    .\Remove-OldExports.ps1 -Path D:\Export -OlderThanDays 30 -WhatIf
#>
[CmdletBinding(SupportsShouldProcess)]
param(
    [Parameter(Mandatory)]
    [ValidateScript({ Test-Path -LiteralPath $_ -PathType Container })]
    [string]$Path,

    [ValidateRange(1, 3650)]
    [int]$OlderThanDays = 30,

    [ValidateNotNullOrEmpty()]
    [string]$LogDirectory = 'C:\ProgramData\Scripts\Logs'
)

$ErrorActionPreference = 'Stop'
Set-StrictMode -Version Latest

# The log is written in -WhatIf mode too
New-Item -ItemType Directory -Path $LogDirectory -Force -WhatIf:$false | Out-Null
$logFile = Join-Path $LogDirectory ('Remove-OldExports_{0:yyyyMMdd}.log' -f (Get-Date))

function Write-Log {
    param(
        [string]$Message,
        [ValidateSet('INFO', 'WARN', 'ERROR')][string]$Level = 'INFO'
    )
    $line = '{0:yyyy-MM-dd HH:mm:ss} [{1}] {2}' -f (Get-Date), $Level, $Message
    Add-Content -LiteralPath $logFile -Value $line -Encoding UTF8 -WhatIf:$false
    Write-Verbose $line
}

$exitCode = 0
try {
    Write-Log "Start: $Path, older than $OlderThanDays days, WhatIf: $WhatIfPreference"
    $cutoff = (Get-Date).AddDays(-$OlderThanDays)
    $files = @(Get-ChildItem -LiteralPath $Path -File | Where-Object LastWriteTime -lt $cutoff)
    Write-Log "Files to delete: $($files.Count)"

    foreach ($file in $files) {
        if ($PSCmdlet.ShouldProcess($file.FullName, 'Delete file')) {
            try {
                Remove-Item -LiteralPath $file.FullName
                Write-Log "Deleted: $($file.FullName)"
            }
            catch {
                Write-Log "Not deleted: $($file.FullName): $($_.Exception.Message)" -Level WARN
                $exitCode = 2
            }
        }
    }
}
catch {
    Write-Log "Error: $($_.Exception.Message)" -Level ERROR
    $exitCode = 1
}
finally {
    Write-Log "End, exit code: $exitCode"
}
exit $exitCode

Header and parameters with validation

#Requires checks conditions before the first line runs: -Version 5.1, -Modules ActiveDirectory, -RunAsAdministrator. The script then ends with one clear message instead of an error halfway through. The help block with .SYNOPSIS, .DESCRIPTION and .EXAMPLE is documentation that does not drift away from the code (Get-Help .\Remove-OldExports.ps1 -Full), and the description of exit codes answers the first question on seeing 0x2 in the scheduler.

Validation attributes work before the script does anything: a wrong value aborts the run with a message that names the parameter.

AttributeWhat it checksExample use
[Parameter(Mandatory)]the parameter must be supplied; in -NonInteractive mode its absence ends in an errordirectory path
[ValidateScript({ … })]any condition returning true or falseTest-Path -LiteralPath $_ -PathType Container
[ValidateRange(1, 3650)]numeric rangenumber of days
[ValidateSet('Archive', 'Delete')]a value from a list, with completion in the consoleoperating mode
[ValidatePattern('^srv-[a-z]+-\d{2}$')]regular expressionserver name
[ValidateNotNullOrEmpty()]no empty valuelog directory

The -WhatIf mode

[CmdletBinding(SupportsShouldProcess)] adds the -WhatIf and -Confirm parameters to the script. Every change is wrapped in $PSCmdlet.ShouldProcess(target, operation), which in -WhatIf mode returns false and prints what would be done. What the script would do in production can therefore be checked before it is allowed to do it. The Ansible counterpart is --check mode.

Errors and the log

The default value of $ErrorActionPreference is Continue. Many command errors, for example Get-ChildItem on a non-existent path, are then non-terminating: PowerShell prints a message and runs the following lines, and the catch block does not run. Hence the 0x0 result despite the error. $ErrorActionPreference = 'Stop' turns every error into an exception that reaches catch. Set-StrictMode -Version Latest adds a second safeguard: a reference to a non-existent variable is an error, so a typo does not silently turn into an empty value.

Start-Transcript -Path <file> -Append is the simplest route for an existing script, but it produces a verbose log that is hard to search. A custom function such as Write-Log in the skeleton writes one line per event with a date and a level, which is easy to compare across days and to send to a central log system. Write-Verbose is for details visible on a manual run with -Verbose.

Exit codes for the scheduler

Task Scheduler does not read the log. It sees only the code with which the powershell.exe process ended, and shows it in hexadecimal.

Code in the scriptMeaningResult in the scheduler
exit 0everything done0x0
exit 1critical error, work aborted0x1
exit 2partly done, details in the log0x2
none (set by the scheduler)the task has not run yet0x41303

The code reaches the scheduler when the script is run with -File; with -Command it can be flattened to 0 or 1. The task is registered with New-ScheduledTaskAction -Execute 'powershell.exe' -Argument '-NoProfile -NonInteractive -File ...', New-ScheduledTaskTrigger and New-ScheduledTaskPrincipal with a gMSA account (-LogonType Password), followed by Register-ScheduledTask. -NonInteractive makes every prompt for a missing parameter end in an error instead of hanging the task forever.

Secrets in a script

When run from the scheduler, the script runs under a group managed service account (gMSA), e.g. gmsa-scripts$, which itself holds the permissions to the share or directory objects (for a manual run Get-Credential is enough). Nobody knows the password, and the domain changes it automatically, by default every 30 days. The account is created with New-ADServiceAccount (this requires a KDS root key in the domain), and on the server it has to be installed (Install-ADServiceAccount) and verified (Test-ADServiceAccount). The account needs the “Log on as a batch job” right, and the server must be listed in PrincipalsAllowedToRetrieveManagedPassword.

Tokens and keys for external APIs come from the Microsoft.PowerShell.SecretManagement module with a vault extension, e.g. Microsoft.PowerShell.SecretStore, or from an external vault such as Azure Key Vault. Only a reference stays in the script (Get-Secret -Name 'ExportApiToken' -Vault 'Automation' -AsPlainText). SecretStore is tied to a user, so it is configured in the context of the account that runs the task, with Interaction set to None. The SecretManagement and SecretStore modules are considered complete and receive only security fixes.

ConvertTo-SecureString 'Haslo123' -AsPlainText -Force in the body of a script does not protect the password, it only changes its type: the password sits in plain text in the file, in the repository and in every backup. The same applies to passwords in the arguments of a scheduled task.

Quality and signing

PSScriptAnalyzer (Invoke-ScriptAnalyzer) catches aliases, plain-text passwords and state-changing functions without ShouldProcess, so code passes through it before deployment. The execution policy is not a security boundary, but with AllSigned a signature (Set-AuthenticodeSignature, a code-signing certificate trusted on the servers) works as change control: editing the file on the server invalidates the signature, so fixes have to go through the repository.

Fulfilling requests through automation

A repetitive request, for example read-only access to a share with reports, is a few minutes of work and days of waiting in a queue. The waiting time does not come from the difficulty of the task, but from the fact that a person has to notice it, read it and handle it by hand. This step is removed by connecting the ticketing system (ITSM) to an automation platform such as AWX or Ansible Automation Platform. The pattern has five steps:

  1. A form in the ticketing system with fixed fields: a list of shares, a user picker from the directory, a read or modify toggle, not a “describe what you need” field.
  2. Approval in the same system, by a manager or the resource owner, according to existing rules.
  3. An API call to the automation platform after approval, with the form content as variables.
  4. The automation, that is the playbook, checks the data, makes the change and collects the result.
  5. The result written back into the ticket: what was done, or what error occurred. A successful ticket closes itself; a failed one returns to the queue for a person.

A person appears in two places: at approval and on error.

Fig. 1. People submit and approve the request, the automation does the rest; an error does not close the ticket but returns it to the queue with the full message.

The form

Every form field becomes a variable in the playbook, so from the start it is designed together with its validation rule.

Form fieldVariableForm typeCheck in the playbook
Ticket numberrequest_idfilled in automaticallymatches a pattern, e.g. REQ- and six digits
Shareshare_nameselection listname is on the list of permitted shares
Useruser_sampicked from the directorypermitted characters and account name length
Access levelaccess_leveltoggleonly read or modify
Dry rundry_runhidden, set by ITboolean

The list of permitted shares lives in the project repository, in a file next to the playbook (allowed_shares.yml, e.g. Finance, HR, Projects), not in the form or in variables: the form helps to choose, but what is permitted is decided by the automation. Access is granted through group membership, not by adding the account to the share’s permissions, so the automation needs the right to change group membership, not to change ACLs on the file server.

Launching a job template through the API

In AWX and Ansible Automation Platform a playbook is launched through a job template, which ties together the playbook, inventory and credentials. After approval the ticketing system sends a single request, POST /api/v2/job_templates/<id>/launch/, with an Authorization: Bearer <token> header, the application/json type and the body {"extra_vars": {"request_id": "REQ-004211", "share_name": "Finance", "user_sam": "jkowalska", "access_level": "read", "dry_run": false}}. The response contains the identifier of the launched job (the job field), which the ticketing system stores with the request. The job state is read from GET /api/v2/jobs/<id>/, and the status field takes values including successful, failed, error and canceled. In Ansible Automation Platform 2.5 and later the same controller API is available under the path /api/controller/v2/.

The template accepts extra_vars from the request only when it has prompting for variables on launch (ask_variables_on_launch) enabled or a survey with those fields. Otherwise the platform launches the job without them and lists the omitted fields in ignored_fields. At the same time extra_vars have the highest precedence in Ansible and override any variable defined in the playbook, inventory or role. The safe configuration is a survey with the form fields (survey_enabled: true) with ask_variables_on_launch disabled: only the survey fields get through. Both settings at false mean that variables from the request are ignored.

Playbook: validation, change, result

The form can be changed or bypassed, or the API called directly, so the playbook checks every field itself. The whole request is one block: after validation and the change, the playbook writes the result into the ticket with the ansible.builtin.uri module (a note and a status change to resolved, or open in dry-run mode), delegated to localhost because the module runs only on a POSIX platform; for a Windows host the equivalent is ansible.windows.win_uri. If any step fails, the rescue section writes the error into the ticket, passes it to the queue for people and fails the job through ansible.builtin.fail. The ticketing system’s API address and the field names in body depend on the specific product; the awx_job_id variable in the note ties the ticket to the full job log.

- name: Share access based on a ticket
  hosts: ad_management
  gather_facts: false
  tasks:
    - name: Handle the request
      block:
        - name: Validate the form data
          ansible.builtin.assert:
            that:
              - request_id is match('^REQ-[0-9]{6}$')
              # list from a file, not a variable: extra_vars cannot override it
              - share_name in (lookup('ansible.builtin.file', 'allowed_shares.yml') | from_yaml)
              - access_level in ['read', 'modify']
              - user_sam is match('^[a-z][a-z0-9.]{1,19}$')
            fail_msg: The request data did not pass validation
            quiet: true

        - name: Add the user to the access group
          microsoft.ad.group:
            name: "GG-Share-{{ share_name }}-{{ access_level }}"
            members:
              add:
                - "{{ user_sam }}"
            state: present
          check_mode: "{{ dry_run | bool }}"
          register: share_access_result

        - name: Write the result into the ticket
          ansible.builtin.uri:
            url: "https://itsm.example.com/api/requests/{{ request_id }}/notes"
            method: POST
            headers:
              Authorization: "Bearer {{ itsm_token }}"
            body_format: json
            body:
              status: "{{ 'open' if dry_run | bool else 'resolved' }}"
              note: "{{ 'Dry run, access was not granted' if dry_run | bool else 'Access granted' }}, AWX job {{ awx_job_id }}"
          delegate_to: localhost
          no_log: true

      rescue:
        - name: Write the error into the ticket and pass it to people
          ansible.builtin.uri:
            url: "https://itsm.example.com/api/requests/{{ request_id }}/notes"
            method: POST
            headers:
              Authorization: "Bearer {{ itsm_token }}"
            body_format: json
            body:
              status: open
              assigned_group: it-service-desk
              note: "Error: {{ ansible_failed_result.msg | default('unknown') }}, AWX job {{ awx_job_id }}"
          when: request_id | default('') is match('^REQ-[0-9]{6}$')
          delegate_to: localhost
          no_log: true

        - name: Fail the job
          ansible.builtin.fail:
            msg: The request was not fulfilled, details in the ticket

Decisions visible in the example:

  • assert comes before every change, so invalid data stops the block before the automation touches the directory. In the rescue section the ticket number goes into the URL only if it passed validation, because it is external data too. With an invalid request_id the only signal is the job status in AWX, which the ticketing system polls and uses to mark the request as needing a person.
  • The microsoft.ad.group module with members.add is idempotent: if the user is already in the group, the task ends with no change, so re-running the same request is not an error and creates no duplicates.
  • The list of permitted shares is read from a file directly in the condition, with no variable name that could be substituted through extra_vars.

Dry run, permissions and measures

For the first weeks a new automation runs with dry_run: true: it validates the data, checks in check_mode what it would change and records this in the ticket, while a person fulfils the request and compares the report with their own action. The exit criterion (e.g. twenty consecutive matching requests) is set before the start.

Three identities take part in the flow, and each gets only what it needs:

  • the ticketing system’s token in AWX: a separate user with the Execute role on a single template and a write scope (launching a job is a write operation), with no access to other templates, inventories or credentials,
  • the account the playbook uses in Active Directory: the “Modify the membership of a group” delegation on the organisational unit holding the share access groups, not a domain administrator,
  • the ticketing system’s API token used by the playbook: adding notes and changing status, with no administrative permissions.

The first automation handles the request with the highest volume and the least costly errors, usually access to a share or a distribution group, not the most complicated one. Two numbers are measured: the time from submitting a request to its fulfilment, and the share of requests of a given type completed with no human involvement. The first shows the effect for users, the second where the automation still fails and why.

Decisions and alternatives

ApproachWhen it makes senseLimitation
Ansible for configuration and multi-machine taskspackages, services, accounts, rotations, API integrationsdoes not maintain state over time; --check is a prediction, not a guarantee
Terraform or OpenTofucreating machines, networks and volumesa plan of changes and state, but not configuration inside the system
GitOps controllerday-to-day management of Kubernetes resourcesrequires manifests in Git; Ansible at most installs the controller
Python script called as a single taskcomplex logic and data transformationneeds its own tests outside the playbook
gMSA account for a scheduled taska script with access to a share or directoryrequires a KDS key, the batch logon right and the server listed in PrincipalsAllowedToRetrieveManagedPassword
SecretManagement with SecretStoretokens and keys for external APIsstore tied to a user; configured in the context of the task account
Template survey with ask_variables_on_launch: falselaunching from the ticketing systemonly the survey fields get through
ask_variables_on_launch: truearbitrary variables, trusted callerthe caller overrides any playbook variable, validation included
Script signing (AllSigned)scripts on servers under change controlrequires a code-signing certificate trusted on the servers

Implementation

  1. Define the boundaries: Ansible for configuration and multi-machine tasks, Terraform for creating resources, a GitOps controller for Kubernetes, complex logic in a Python script. Check of effects: playbooks contain no command tasks with Jinja2 logic and no shell: kubectl apply.
  2. Repository layout according to the path table, with the test inventory as the default and secrets in Vault. Check: every Vault file encrypted (grep -L '^\$ANSIBLE_VAULT' on the vault.yml files).
  3. A CI job with ansible-playbook --syntax-check and ansible-lint on the basic profile, raised gradually to production. Check: a second run of the playbook reports no changed.
  4. Scheduler scripts built on the skeleton with validation, -WhatIf, a log and an exit code, run with -File under a gMSA account. Check: tasks outside the Microsoft folder (Get-ScheduledTask with Get-ScheduledTaskInfo) without error codes in LastTaskResult (SCHED_S_ codes are states, not errors), running on schedule, under SYSTEM or a gMSA, with no personal passwords stored in tasks.
  5. An AWX job template with a survey and ask_variables_on_launch: false, a separate user with the Execute role and a token with the write scope. Check: reading the template configuration through GET /api/v2/job_templates/<id>/ (a token with the read scope is enough) shows true and false in the survey_enabled and ask_variables_on_launch fields.
  6. The first request type with dry_run: true until the exit criterion is met, then switching to execution and measuring fulfilment time and the share of requests handled without a person.

Limitations and pitfalls

  • -WhatIf without a log. -WhatIf propagates to every command in the script that supports it, including New-Item and Add-Content, which write the log. Without -WhatIf:$false on those two commands a dry run leaves no log, and the log is its result.
  • Exit codes of external programs. $ErrorActionPreference does not cover the exit codes of programs, so after calling robocopy, for example, $LASTEXITCODE has to be checked; for robocopy values of 8 and above mean an error (if ($LASTEXITCODE -ge 8) { throw "robocopy: code $LASTEXITCODE" }). In PowerShell 7.4 and later $PSNativeCommandUseErrorActionPreference set to $true turns non-zero codes into errors, but robocopy uses them for success too, and Windows PowerShell 5.1 does not have this variable.
  • Script file encoding. Windows PowerShell 5.1 reads a file without a BOM in the system code page and turns non-ASCII characters in messages into random symbols; scripts containing such characters are saved as UTF-8 with BOM.
  • Validation with the list in a variable. Validation in the form is a convenience for the user, not a safeguard: anyone holding a token can launch the template with arbitrary values. A list of permitted values stored as a playbook variable is overridden by extra_vars when ask_variables_on_launch: true, along with all validation, so it is read from a file and input data is treated as untrusted.
  • rescue without fail. A block whose error was handled ends the job with the successful status, so a ticketing system that checks only the status will not notice the problem. A final ansible.builtin.fail in rescue restores the failure status.

Sources