Maintaining and recovering Active Directory
FSMO roles and functional levels, system state backup and four ways to restore the directory, forest recovery, and signing tools in with a domain account over LDAPS or OIDC.
Active Directory needs three kinds of maintenance that become visible only during a failure: correct placement of the FSMO roles and a plan for raising functional levels, a backup from which the directory can be rebuilt, and the dependency of tools on the directory for sign-in. This document covers the FSMO roles, domain controller replacement and functional levels, system state backup with four ways of restoring and forest recovery, and signing administrative tools in with a domain account over LDAPS and OIDC.
FSMO roles and functional levels
The five roles and their scope
Active Directory is a multi-master directory: writes are possible on any controller. Operations that need a single arbiter are assigned to five roles: two exist once per forest, three once per domain. When a controller is simply switched off, roles do not move on their own; only a graceful demotion moves them. After the holder fails the domain keeps running, but the functions tied to the role expire at different speeds, from hours (PDC Emulator) to months (Schema Master).
| Role | Scope | What it does | When the holder fails |
|---|---|---|---|
| Schema Master | forest | accepts schema changes | no effect until the schema has to be extended |
| Domain Naming Master | forest | adds and removes domains and application partitions | no effect until the forest structure changes |
| PDC Emulator | domain | time source, password changes, account lockouts, GPO editing | within hours: time drift and Kerberos errors |
| RID Master | domain | hands out pools of RIDs to controllers | when the pool runs out no new objects can be created |
| Infrastructure Master | domain | updates references to objects from other domains | no effect in a single-domain forest |
The Infrastructure Master matters only in a multi-domain forest and has no work to do when the AD Recycle Bin is enabled or every domain controller in the domain is a global catalog; otherwise it should not be on a global catalog controller. A sufficient layout is the forest roles, the PDC Emulator and the RID Master on one controller, the Infrastructure Master on a second, and a third controller without roles as a reserve to seize from. Only the PDC Emulator of the forest root domain, the top of the time hierarchy, should take time from an external source.
Transfer and seizure
A transfer is a planned operation: both sides are alive and hand the state of the role over. It is done before the controller is switched off with Move-ADDirectoryServerOperationMasterRole -Identity dc02 -OperationMasterRole PDCEmulator, RIDMaster, InfrastructureMaster, SchemaMaster, DomainNamingMaster.
A seizure is an emergency operation for when the current holder is unreachable. It is the same cmdlet with -Force: it first tries a transfer and seizes the role only when that is not possible. A seizure requires three conditions:
- The current holder will definitely not return.
- The seizing controller is replicated with the others:
repadmin /replsummarywithout errors, ideally alsodcdiag /test:knowsofroleholders. - The role is needed now; for the Schema Master and Domain Naming Master it is often wiser to wait.
After the seizure, dcdiag /test:ridmanager confirms that the RID Master is reachable, and the dead controller’s object is removed together with its metadata (ntdsutil, “metadata cleanup”) and its records in DNS.
Replacing a domain controller
Replacing a controller follows four steps.
- New controller and replication. Promotion into the existing domain with DNS and, if needed, the global catalog. SYSVOL must already replicate through DFSR, because Windows Server 2016 is the last version that supports FRS (
dfsrmig /getglobalstate). Before the transfer,repadmin /replsummaryanddcdiagclean. - Role transfer, checked with
netdom query fsmo. The PDC Emulator of the forest root domain takes time from an external source, so the time configuration is changed on both the new and the previous PDC Emulator. - Companion services. The new controller’s address in DHCP, on servers with static addresses and in DNS forwarders, and in services that refer to the old one by name or address (LDAP, NPS, printers).
- Demoting the old one and clean-up.
Uninstall-ADDSDomainController, thenUninstall-WindowsFeature AD-Domain-Servicesand switching the server off; after an emergency shutdown the metadata is removed by hand, as after a seizure. Then entries in DNS, sites and subnets, and in scripts and documentation.
Functional levels
The domain and forest functional level declares the oldest system running on the controllers and does not rise by itself when a newer controller is added.
| Level | What it unlocks |
|---|---|
| Windows Server 2016 | Privileged Access Management (forest level) |
| Windows Server 2025 | optional 32k database pages; requires all controllers on 2025 with a 32k-capable database (an in-place upgrade stays on 8k) |
Windows Server 2019 and 2022 introduced no new levels, so their controllers run at level 2016. A 2025 controller does not run at level 2012 R2, so an older domain must first reach 2016, which requires SYSVOL over DFSR (as of October 2026).
A level is raised when all controllers run the required version (no forgotten old one), replication is clean and a system state backup from that day exists. All domains are raised first, then the forest, because the forest level cannot be higher than the level of any domain: Set-ADDomainMode -Identity corp.example.com -DomainMode Windows2016Domain, followed by Set-ADForestMode -Identity corp.example.com -ForestMode Windows2016Forest.
Backing up and recovering Active Directory
Why a checkpoint is not enough
Every controller numbers its changes with an update sequence number (USN) and exchanges them with its partners. A controller rolled back to yesterday’s state without the directory knowing has an old number while its partners remember a higher one, so its new changes may never replicate. Since Windows Server 2012 the VM-GenerationID mechanism protects against this where the hypervisor exposes it, but the checkpoint still sits on the same storage and will not bring back a deleted object. The reference point remains a system state backup taken with a directory-aware tool; a host-level backup of the whole machine is acceptable only if it is application-aware (VSS) and regularly tested.
System state backup
A controller’s system state backup includes the directory database (ntds.dit), SYSVOL, the registry and the boot files. The built-in tool needs its feature installed once (Install-WindowsFeature Windows-Server-Backup), and the backup to a dedicated volume is taken with wbadmin start systemstatebackup -backupTarget:E: -quiet. Retention is handled by wbadmin delete systemstatebackup -keepVersions:14 -quiet, which keeps the 14 most recent versions and deletes older ones.
The command runs daily from Task Scheduler on at least two controllers in every domain, and the result is copied to storage beyond the reach of domain accounts; a backup that lives only on the controller’s own volume dies with it. The directory itself records when each partition was last backed up: repadmin /showbackup shows the time of the last backup of each partition (the domain, Configuration, Schema and the DNS zones) in the Org.Time/Date column. When a partition has not been backed up for longer than half of the deleted object lifetime (the default threshold), the controller writes event 2089 to the Directory Service log.
The deleted object lifetime and the Recycle Bin
A backup older than the deleted object lifetime (tombstoneLifetime: 180 days by default, and 60 days in forests created on Windows 2000, Windows Server 2003 or 2003 R2 where nobody has raised the value) cannot be used for a restore: the other controllers have long forgotten the deletions the restored controller would need to receive from them, and lingering objects appear in the directory.
An Active Directory Recycle Bin enabled in advance restores a deleted object with all its attributes and group memberships, without restarting the controller and without a backup. It is enabled once for the whole forest, irreversibly (Enable-ADOptionalFeature), at a forest functional level of at least Windows Server 2008 R2; objects deleted before it was enabled cannot be restored with it. A deleted account is restored with Get-ADObject -Filter "sAMAccountName -eq 'jsmith'" -IncludeDeletedObjects | Restore-ADObject. For a deleted organisational unit, the unit itself is restored first and its contents afterwards, because an object returns to its parent container, which has to exist.
Four ways to restore
| Method | When | What happens | Typical time |
|---|---|---|---|
| AD Recycle Bin | a deleted object, within msDS-deletedObjectLifetime | the object returns with its attributes and memberships, no restart | minutes |
| Non-authoritative restore | one controller’s database is damaged, the others work | the controller returns to the backup state (wbadmin start systemstaterecovery in DSRM) and pulls newer changes from its partners | an hour or two |
| Authoritative restore | objects deleted or changed without the Recycle Bin, an error replicated to every controller | selected objects from the backup get higher version numbers (ntdsutil) and overwrite the state on partners | several hours |
| Forest recovery | no controller works or none can be trusted | one controller per domain from backup in isolation, clean-up, password resets, the rest built anew | from several hours to several days |
When a single controller fails and the others work, it is usually quicker and cleaner to remove its metadata, seize its roles and build a new controller than to restore it from backup.
An authoritative restore is performed in directory services restore mode (DSRM), signing in with the .\Administrator account and the DSRM password:
bcdedit /set safeboot dsrepair
REM restart, sign in as .\Administrator with the DSRM password
wbadmin get versions
wbadmin start systemstaterecovery -version:10/10/2026-22:00
REM still in DSRM, before returning to normal mode
ntdsutil "activate instance ntds" "authoritative restore" "restore subtree OU=Finance,DC=corp,DC=example,DC=com" quit quit
bcdedit /deletevalue safeboot
REM restart into normal mode
When wbadmin asks for a restart after the restore, the answer has to be no: the controller must stay in DSRM until ntdsutil finishes the authoritative restore. The .ldf files with back-links (such as group memberships from other domains) that ntdsutil writes along the way are imported after the restart with ldifde -i -f <file>.ldf.
The DSRM password
Every controller has a local DSRM administrator account whose password is set at promotion and rarely changed afterwards. Without it a controller started in DSRM is inaccessible and a restore from backup is impossible. The password is set anew in ntdsutil (set dsrm password, reset password on server null) or synchronised with a disabled domain account (sync from domain account <account>) whose password is kept in the vault. Synchronisation is a one-off copy, so after every change of the account’s password it has to be repeated on every controller, and the vault has to be available without a working domain.
Forest recovery in stages
Forest recovery is needed when no controller works or none can be trusted: ransomware on every controller, loss of the data centre, a compromised domain. The process has six stages.
- Decide and isolate. All controllers disconnected from the production network, recovery in an isolated network. A backup from before the incident is chosen.
- The first controller per domain. Restore system state in DSRM with an authoritative SYSVOL:
wbadmin start systemstaterecovery -version:<version> -authsysvol. - Roles and metadata. Seize all FSMO roles, remove the other controllers’ objects from the directory and their records from DNS, raise the RID pool by 100,000 and invalidate the current pool.
- Passwords and keys. Reset the krbtgt account password twice, at least 10 hours apart, and reset DSRM passwords; if the incident was a security breach, also reset inter-domain trust passwords and privileged account passwords.
- Verification. Logon, Kerberos ticket issuance, SRV records in DNS, no diagnostic errors.
- Replication and new controllers. Check the replication of the restored controllers in the isolated network, connect them to the production network, add the global catalog on a controller in the forest root domain, and only then promote further controllers.
The sequence above is a simplification; the details of each stage, including invalidating the RID pool and the order of actions in a forest with several domains, are described in Microsoft’s “Active Directory forest recovery guide”. An organisation’s own procedure is a shortened version of it with server names, commands and a note on where in the vault the passwords are kept.
Recovery rehearsal
Once a year a rehearsal is held in an isolated virtual network with no connection to production (on Hyper-V, a Private switch): one controller is restored from a production backup, the roles are seized, sign-in is verified and the time is measured. The rehearsal exposes missing DSRM passwords and outdated steps; a forest recovery that nobody has rehearsed takes days instead of hours. The result goes into a restore-test record.
Signing tools in through Active Directory
A CI system, a GitOps platform, monitoring and a wiki sign users in with a domain account in two ways: directly over LDAPS, or through an OIDC broker. In both, an account disabled in the directory stops working in every tool (in OIDC, with a delay until the issued token expires), so the local user lists that nobody remembers after an administrator leaves disappear. The difference lies in who sees the user’s password and where a second factor can be enforced.
Direct LDAPS
The tool takes the user’s login and password, connects to a domain controller and checks them with a bind operation. It reads group membership with an LDAP query and assigns roles accordingly. The advantages are simple configuration, no extra components and support in many tools. The disadvantages follow from the user’s password reaching the tool:
- every tool sees the passwords of everyone who signs in, so compromising one of them means leaking domain passwords,
- multi-factor authentication cannot be enforced, because an LDAP bind accepts only a password,
- every tool needs an account for group queries, and when its password expires, sign-in stops for everyone,
- session, lockout and logout policy differs in every tool.
OIDC through a broker
The tool redirects the user to an identity broker via OpenID Connect. The broker, connected to Active Directory, checks the password and returns a signed token with the user name and group list to the tool; the tool never sees the password.
Two open-source options are commonly found. Dex is a lightweight broker with no user database of its own: it translates LDAP to OIDC and stores no accounts. Keycloak is a full identity provider with user federation from LDAP, its own session policy and a second factor (one-time codes, WebAuthn keys). A third route is a cloud identity provider with accounts synchronised from the directory.
| Property | Direct LDAPS | OIDC through Dex | OIDC through Keycloak or Entra ID |
|---|---|---|---|
| Who sees the password | every tool | Dex only | the identity provider only |
| Second factor | no | no, the LDAP connector checks only the password | yes, enforced in one place |
| Directory bind accounts | one per tool | one | one, or none with synchronisation |
| Extra components | none | one service, with no user database of its own | a server with a database, or a cloud service |
Moving from LDAPS to OIDC requires no changes in the directory: the groups and naming convention stay the same, only the place where the password is checked changes.
The LDAP connector in Dex
The configuration connects Dex to the directory over LDAPS, admits only enabled user accounts and passes only groups prefixed with GG- to the tools, so the token does not carry hundreds of irrelevant groups.
connectors:
- type: ldap
id: ad
name: Active Directory
config:
# The domain name instead of a single controller address
host: corp.example.com:636
# The internal certificate authority's certificate
rootCA: /etc/dex/tls/corp-root-ca.pem
# Bind account, password from an environment variable
bindDN: CN=svc-dex-bind,OU=Service Accounts,DC=corp,DC=example,DC=com
bindPW: $LDAP_BIND_PASSWORD
usernamePrompt: Domain login
userSearch:
baseDN: OU=Users,DC=corp,DC=example,DC=com
# Enabled user accounts only
filter: "(&(objectCategory=person)(objectClass=user)(!(userAccountControl:1.2.840.113556.1.4.803:=2)))"
username: sAMAccountName
idAttr: DN
emailAttr: userPrincipalName
nameAttr: displayName
preferredUsernameAttr: sAMAccountName
groupSearch:
baseDN: OU=Groups,DC=corp,DC=example,DC=com
filter: "(&(objectClass=group)(cn=GG-*))"
userMatchers:
- userAttr: DN
groupAttr: member
nameAttr: cn
Some platforms, Argo CD for example, ship Dex built in: the connector configuration then goes into the dex.config key in argocd-cm, and the bind account password into the secret that bindPW refers to.
Permissions from groups
Whichever option is chosen, permissions come from directory group membership, not from manual assignments in the tool. A simple convention works: a separate group per role per tool, with a name that says what it grants, such as GG-GitOps-Admins, GG-GitOps-ReadOnly, GG-Grafana-Editors. Group membership is the only way to obtain a role, and an access review comes down to reviewing groups. In Argo CD, argocd-rbac-cm maps groups to roles (entries such as g, GG-GitOps-Admins, role:admin, with scopes: '[groups]'), and leaving out policy.default means people outside those groups have no access.
A configuration that matches on member returns only direct memberships. If roles should also come from nested groups, in Dex the recursionGroupAttr: member option in userMatchers enables it; in plain LDAP against AD the matching rule 1.2.840.113556.1.4.1941 does the same. A simpler approach is a rule that groups granting roles in tools contain user accounts only.
The bind account and local accounts
The broker needs an account to look up users and groups. An ordinary domain account is enough, because authenticated users can read the attributes needed for sign-in by default. The account belongs to no group other than Domain Users, is denied local and Remote Desktop logon, and its description names the owner and the date of the next password change. The password has at least 30 random characters and is changed on a plan, for example once a year, together with the secret in the broker. An account whose password expires on its own at an unexpected moment stops sign-in to every tool at once.
A tool’s local administrator stays, but it is not used for daily work and is secured as emergency access in case the directory or the broker is unavailable.
Decisions and alternatives
| Approach | When it makes sense | Limitation |
|---|---|---|
| Transferring an FSMO role | planned replacement or shutdown of a controller, both partners alive | requires a working current holder |
Seizing an FSMO role (-Force) | holder unreachable and certain never to return; for forest roles it is often better to wait | the previous holder must not return to the network in its earlier state |
| System state backup with a directory-aware tool | the reference point for every controller, at least two per domain | a backup older than the deleted object lifetime is useless |
| Host-level backup of the whole machine | only as an application-aware (VSS) backup with regular tests | without VSS and tests it does not guarantee a consistent restore |
| Virtual machine checkpoint | not a directory backup | sits on the same storage, will not bring back a deleted object |
| A new controller instead of a restore from backup | one controller fails, the others work | requires removing metadata and seizing roles |
| Direct LDAPS | one or two tools; an acceptable start with certificate verification and a dedicated bind account | every tool sees passwords, no second factor |
| OIDC through Dex | several tools, a Kubernetes platform | only Dex sees the password, but it cannot enforce a second factor |
| OIDC through Keycloak or Entra ID | MFA required, many tools | a server with a database, or a cloud service |
Baseline configuration
- Role placement according to the layout of one controller for the forest roles, PDC Emulator and RID Master, a second for the Infrastructure Master and a third as a reserve; time configuration on the PDC Emulator of the forest root domain.
- The Recycle Bin enabled before it is needed, and the DSRM password stored in a vault available without a working domain, set or synchronised on every controller.
- A daily system state backup from Task Scheduler on at least two controllers per domain, copied beyond the reach of domain accounts; event 2089 in the Directory Service log monitored.
- Tool sign-in: separate groups per role, a bind account with minimal rights, port 636 reachable on every controller (
Test-NetConnection -Port 636), and the trust chain and name verified the way the tool does it (openssl s_clientwith-verify_hostname).
Maintenance procedures
- A health check before any operation on roles and before a restore: replication without errors (
repadmin /replsummary), a backup of every partition no older than 24 hours (repadmin /showbackup *), roles on the expected controllers (netdom query fsmo). Replication errors are removed first, because a backup taken from a controller that has not replicated for weeks contains an outdated directory. - Controllers replaced in order: promotion, role transfer, companion services, demotion, with
dcdiag,repadmin /replsummaryandnetdom query fsmoafter each step. - Functional levels raised only after the last old controller has been replaced and after a fresh system state backup: domains first, then the forest.
- A yearly recovery rehearsal in an isolated network with the time measured, filling in missing passwords and steps in the procedure.
Limitations and pitfalls
- A controller returning after a role seizure. Two holders of the same role cause a conflict for the Schema Master, Domain Naming Master and RID Master; the PDC Emulator and Infrastructure Master are not affected. The previous holder is removed by reformatting the disk and reinstalling, or by forced demotion and metadata clean-up; Microsoft recommends rebuilding the controller rather than restoring it from a backup.
- Functional levels as a one-way operation. The
Set-ADDomainModedocumentation allows lowering a level provided no feature requiring the higher level is in use, while the page on upgrading domain controllers allows it only in a few listed cases, so a plan does not rely on a rollback. A fresh system state backup is needed before the change. - A backup taken after a security incident. With ransomware and a compromised administrator account, the latest backup may already contain the attacker’s changes: new accounts, memberships of privileged groups, altered Group Policy objects. Choosing a backup requires establishing since when the attacker was in the network, and after recovery the privileged accounts and Group Policy are reviewed before the forest returns to production.
- Disabled certificate verification and the name in the certificate. Many examples online contain
insecureSkipVerify: true, which turns off verification of the domain controller’s certificate: the connection is encrypted, but it is not known with whom, so anyone who intercepts traffic collects the passwords of everyone signing in. Instead,rootCApoints at the internal certificate authority’s certificate. LDAPS works only when the controller has a server authentication certificate, and Microsoft requires the controller’s FQDN in the Subject CN or in the SAN extension. The certificate usually contains the controller’s own FQDN, and the domain name only when it was issued from a template that puts it in the SAN; when addressing by domain name this has to be checked, and otherwise the controllers are pointed at by their FQDNs. - Delay in revoking access in OIDC. A token issued by the broker is valid until it expires, and in Dex the lifetime is set by
expiry.idTokens(24 hours by default). The Dex documentation recommends short lifetimes, for example 10 minutes; for tools with access to production, an hour or less is a sensible value.