Encryption at Rest Operations, Recovery and Troubleshooting
Table of Contents
Introduction
Once Encryption at Rest (EAR) has been deployed and configured, administrators are responsible for maintaining key availability, supporting infrastructure changes, and ensuring that encrypted data remains accessible throughout the lifecycle of the Endpoint Data Protection environment.
Operational activities such as adding or replacing servers, recovering from hardware failures, rotating encryption keys, and retiring obsolete infrastructure require careful handling of Encryption at Rest key material. Incorrect key management may impact the ability to decrypt previously protected information or perform recovery operations successfully.
This guide focuses on the operational aspects of Encryption at Rest and provides procedures for server onboarding, server migration, disaster recovery, server decommissioning, and ongoing maintenance activities. It also includes security recommendations, troubleshooting guidance, and recovery considerations to help administrators manage Encryption at Rest environments safely and consistently.
Before performing any operational or recovery procedure, ensure that current Encryption at Rest key backups are available and that the associated backup passwords can be retrieved. Key backups are a critical dependency for server migration, disaster recovery, and long-term access to encrypted data.
Rotating Encryption at Rest Keys
Key rotation is the process of replacing the key used for new encryption operations with a different key while maintaining access to previously encrypted data.
Typical reasons for key rotation include:
- Internal security policies
- Compliance requirements
- Periodic cryptographic hygiene
- Migration to organization-supplied key material
- Retirement of older keys
Encryption at Rest supports seamless key rotation by allowing multiple keys to coexist within the key store. Existing data remains accessible while new encryption operations use the newly designated default key.
Rotating a key does not automatically re-encrypt existing data. Previously encrypted information remains associated with the key that originally encrypted it.
Single Server Environment
In a single-server deployment, key rotation consists of four steps:
- Create or Import a New Key: Create a new key or import an existing key into the local key store.
- Verify the Key Backup: Validate the backup file before putting the key into service.
-
Make the New Key the Default Key: Activate the new key:
AdminTool.exe /encryptionAtRestMakeDefaultKey /encryptionAtRestThumbprint <thumbprint>- Verify the result using:
AdminTool.exe /encryptionAtRestListKeys
- The new key should display: DEFAULT: YES
- Revoke the Previous Key (Optional)
Once the new key has successfully been put into service, the previous key can be revoked if it is no longer required for future encryption operations. The retired key remains available for decrypting historical data.
Rotation Workflow:
Create or Import New Key
↓
Validate Backup
↓
Make New Key Default
↓
Verify Operation
↓
(Optional) Revoke Previous KeyMulti-Server Environment
In multi-server deployments, key rotation requires additional planning because every active server must be able to decrypt the same data. Before a key can become the default key, every active server must already hold the replacement key.
- Create a new key or import an existing key into the local key store on Server A.
- Distribute the Key: Import the same key backup onto every remaining server (Server B, Server C, Server D etc.)
- All servers must report the same thumbprint when listing available keys.
- Verify Key Availability on each server by executing AdminTool.exe /encryptionAtRestListKeys
- Confirm the following:
- The new key exists on every server.
- The thumbprint is identical on every server.
- No key synchronization warnings are displayed.
- Confirm the following:
- Make the New Key the Default Key: Designate the replacement key as the default key on each server.
- Important: If one or more active servers do not hold the replacement key, the operation will be refused.
- Verify the New Default Key
- Confirm that all servers report the replacement key as the active default key.
- Revoke the Previous Key
Once the new key has successfully been put into service on each Server, the previous key can be revoked if it is no longer required for future encryption operations. The retired key remains available for decrypting historical data.
Rotation Workflow:
Create New Key
↓
Export Key Backup
↓
Import Key on All Servers
↓
Verify Matching Thumbprints
↓
Make New Key Default on All Servers (Repeat the operation on each active server that should use the new key for encryption)
↓
Verify All Servers
↓
(Optional) Revoke Old Key on All ServersWhat Changes During Key Rotation?
After successful rotation:
- New secrets are encrypted using the new default key.
- Existing secrets remain unchanged.
- Previously encrypted data remains readable.
- Multiple keys may coexist within the key store.
- The previous key may be safely retired once no further encryption operations depend on it.
- Best Practice
Always retain a secure backup of both the new key and any previously used keys. Even after a successful rotation, historical data or database backups may still require the older key for decryption and recovery purposes.
Revoking an Encryption at Rest Key
A key can be revoked when it is no longer intended to be used for future encryption operations. Common reasons include key rotation, compliance requirements, or the planned retirement of older encryption keys. Revoking a key prevents the key from being designated as the default key in the future. However, the key remains in the key store to ensure that data encrypted with that key can still be decrypted when required.
Revoking a key does not delete the key and does not re-encrypt existing data. Any data previously encrypted with the revoked key remains accessible as long as the key remains available in the key store.
Before You Start
Before revoking a key:
- The key must exist in the local key store.
- The key must not already be revoked.
- The key thumbprint must be known.
- If the key is currently the default key, another key should already be available and designated as the default key before revocation.
- In multi-server environments,
- Ensure that all servers have transitioned to the replacement key before revoking the retired key.
- Revoke the retired key on all servers where it is present.
To obtain the thumbprint of an existing key, run:
AdminTool.exe /encryptionAtRestListKeysRevoking a Key
Adjust and execute the following command using the thumbprint of the key that should be retired.
AdminTool.exe /encryptionAtRestRevokeKey /encryptionAtRestThumbprint <thumbprint>In multi-server environments, ensure that the key revocation is performed consistently across all participating servers.
As a result, a successful revocation process looks like this:

After the revocation, review again all available encryption at rest keys
AdminTool.exe /encryptionAtRestListKeysThe revoked key should now show REVOKED YES:

After a key has been revoked:
- The key can no longer be designated as the default key.
- The key remains available for decryption of previously encrypted data.
- Existing encrypted secrets remain unchanged.
- No data is automatically re-encrypted.
- The key continues to be listed in the key store for historical and recovery purposes.
This behavior allows organizations to retire keys safely while maintaining access to data that was encrypted before the key was revoked.
Adding a Server to an Existing Encryption at Rest Environment
When a new Endpoint Data Protection server is added to an environment that already uses Encryption at Rest, the server must receive all Encryption at Rest keys required by the deployment before it can participate fully in encryption operations. This process ensures that the new server can decrypt existing protected data and remains consistent with the other servers in the environment.
Important: The new server must receive all keys used within the environment, including historical and revoked keys that may still be required to access previously encrypted data.
Before You Start
Before adding a server:
- The new server must be installed and connected to the same Endpoint Data Protection database.
- At least one existing server must already be using Encryption at Rest.
- Exported key backup files and the associated passwords must be available.
- The existing deployment must remain operational throughout the onboarding process.
- Import Existing Keys
Import Encryption at Rest Keys
Import all required Encryption at Rest keys onto the new server.
Example:
AdminTool.exe /encryptionAtRestImportKey /encryptionAtRestFile <backupfile>.json /encryptionAtRestPassword <password>AdminTool.exe /encryptionAtRestImportKey /encryptionAtRestKeyHex <64 hexadecimal characters>Repeat the process for every key used within the deployment.
Verify Imported Keys
List the available keys:
AdminTool.exe /encryptionAtRestListKeysVerify:
- All expected keys are present.
- Thumbprints match those on the existing servers.
Set Default Key
Once all keys have been imported successfully, designate the same default key used by the existing deployment.
AdminTool.exe /encryptionAtRestMakeDefaultKey /encryptionAtRestThumbprint <thumbprint>Verification
For verification, confirm the following:
- The new server lists all expected keys.
- The correct default key is configured.
- No missing key warnings are reported.
- Encryption operations function normally.
What Changed?
The new server now participates fully in the Encryption at Rest environment and can:
- Decrypt existing encrypted data
- Encrypt new data using the active default key
- Participate in future key rotations
- Participate in backup and recovery operations
Moving a Server to New Hardware
Encryption at Rest supports replacing an existing Endpoint Data Protection server without interrupting access to encrypted data. To successfully migrate a server to new hardware, the replacement server must receive all Encryption at Rest keys that are currently known to the server being replaced.
Important: All keys must be migrated, not only the current default key. Historical and revoked keys may still be required to decrypt previously encrypted data or restore older backups.
When to Use This Procedure
Use this procedure when:
- An existing Endpoint Data Protection server is being replaced with new hardware.
- The original server is still available and operational.
- Encryption at Rest is already active within the environment.
If the original server is no longer available, use the recovery procedure described in the Server Recovery section instead.
If Encryption at Rest is not yet in use, no key migration is required and the server can be replaced using the standard Endpoint Data Protection server replacement process.
Before You Start
Before beginning the migration:
- The new server must be prepared for installation and connected to the same Endpoint Data Protection database.
- The existing server must remain online until the migration has been completed successfully.
- Any required Encryption at Rest key backup files and passwords must be available.
Procedure
-
Identify Existing Keys: On the existing server, list all available Encryption at Rest keys:
AdminTool.exe /encryptionAtRestListKeys
-
Document all listed keys, including:
- Thumbprints
- Default key designation
- Revocation status
- Single-Server Environment: If no other server holds these keys, export and safely store backups of every listed key before proceeding.
- Install the Replacement Server: Install Endpoint Data Protection on the new hardware and connect it to the existing Endpoint Data Protection database.
-
Add the New Server to the Encryption at Rest Environment: Follow the procedure described in Adding a Server to an Existing Encryption at Rest Environment. During this step:
- Import all required Encryption at Rest keys.
- Verify that all expected thumbprints are present.
- Ensure the correct default key is configured.
-
Verify Key Consistency: On the new server, execute:
AdminTool.exe /encryptionAtRestListKeys- Verify that:
- The new server holds all keys previously held by the old server.
- The same key is configured as the default key.
- No missing key warnings are reported.
- Normal operations are functioning correctly.
- Do not continue until the replacement server matches the original server's Encryption at Rest configuration.
-
Remove the Old Server: Once the replacement server has been verified:
- Stop the Endpoint Data Protection services on the old server.
- Remove the old server from Encryption at Rest management by following the Decommissioning a Server procedure.
- Verify that the old server no longer participates in key management operations.
The migration is complete when:
- The replacement server appears in the environment.
- All keys have been migrated successfully.
- The same default key remains active.
- Encryption and decryption operations continue normally.
- The old server has been removed from Encryption at Rest management.
What Changed?
After the migration:
- The new server holds every Encryption at Rest key previously held by the original server.
- The same default key continues to be used for new encryption operations.
- Existing encrypted data remains accessible without modification.
- The old server no longer participates in Encryption at Rest key management.
Recovering from a Lost Server
If an Endpoint Data Protection server is permanently lost and can no longer be accessed, Encryption at Rest can be restored on a replacement server using previously exported key backups.
Typical scenarios include:
- Hardware failure
- Operating system corruption
- Virtual machine loss
- Disaster recovery situations
- Accidental server deletion
Recovery is only possible if the required Encryption at Rest key backups are available. Without the corresponding key backup files and passwords, data protected by those keys may become inaccessible.
Before You Start
Before beginning recovery:
- A replacement server must be available.
- The Endpoint Data Protection database must still be accessible.
- Backup copies of all required Encryption at Rest keys must be available.
- The passwords protecting the exported key files must be known.
Recovering Procedure
-
Identify Required Keys: Determine which Encryption at Rest keys were used by the lost server. If additional servers remain operational, execute:
- AdminTool.exe /encryptionAtRestListKeys
- and note:
- All available thumbprints
- The current default key
- Any historical or revoked keys
- Important: Recover all listed keys, not only the current default key.
- Install a Replacement Server: Install Endpoint Data Protection on the replacement server and connect it to the existing Endpoint Data Protection database.
-
Import the Required Keys: Import all available key backup files.
- Example:
AdminTool.exe /encryptionAtRestImportKey /encryptionAtRestFile <backupfile>.json /encryptionAtRestPassword <password>AdminTool.exe /encryptionAtRestImportKey /encryptionAtRestKeyHex <64 hexadecimal characters> - Repeat the process until all required keys have been restored.
- Example:
-
Verify Key Availability: List the available keys:
AdminTool.exe /encryptionAtRestListKeys- Verify that:
- All expected keys are available.
- Thumbprints match the remaining servers.
- The correct default key is present.
-
Restore the Default Key Configuration
- If required, designate the appropriate key as the default key:
AdminTool.exe /encryptionAtRestMakeDefaultKey /encryptionAtRestThumbprint <thumbprint>
-
Remove References to the Lost Server
- After confirming that the replacement server operates correctly, remove the lost server from Encryption at Rest management by following the Decommissioning a Server procedure.
- This removes obsolete key metadata associated with the unavailable server and prevents future key consistency warnings.
Verification
Recovery is complete when:
- All required keys are available on the replacement server.
- The correct default key is configured.
- No missing key warnings are reported.
- Encryption and decryption operations function normally.
What Changed?
After recovery:
- The replacement server can access previously encrypted data.
- Existing encrypted secrets remain fully readable.
- New encryption operations continue using the current default key.
- The lost server no longer participates in Encryption at Rest key management.
Best Practice
Regularly export and validate Encryption at Rest key backups. Losing both a server and its associated key backups may result in permanent loss of access to encrypted data.
Decommissioning a Server
When a server is permanently removed from the Endpoint Data Protection environment, it must also be removed from Encryption at Rest (EAR) management. As long as a server remains registered within EAR, it continues to participate in key validation checks and may prevent key management operations from being completed successfully.
Decommissioning a server removes its EAR-related metadata from the database. This procedure does not delete or transfer any encryption keys and does not affect encrypted data stored within the environment. Decommissioning should only be performed after all required keys have been verified on the remaining servers.
Before You Start
Decommissioning removes only the server's EAR metadata from the database. Encryption keys are not deleted and encrypted data remains unchanged. The procedure simply informs EAR that the server is no longer part of the managed environment, allowing future key management operations to proceed without checking that server. Before decommissioning a server, ensure that:
- At least one remaining server has access to the current default EAR key.
- The target server has been shut down or is no longer participating in the environment.
- All registered names associated with the target server have been identified using:
AdminTool.exe /showServers- The name is the part before the bracket:
CMD: Servers list PR NAME[IP] TYPE VERSION ----------------------------------------------------------------- 01 SRV-EDP-01[192.168.10.21] MGMT 26.1.0.0 02 SRV-EDP-02[192.168.10.22] MGMT 26.1.0.0- A Server with more than one network address can appear under more than one name. Note all of them.
- If /showServers lists other Servers but not the target, it is already deregistered.
Procedure
Deregister the target Server
Remove each registered server name from the server list:
- Deregister the target Server. Run this once for every name it appears under:
AdminTool.exe /deleteServer <target server name>
- Each run prints:
CMD: Delete server SRV-EDP-02 Result: done
Remove the target Server's key metadata
Get the Server UID
You need first to have the Server UID. You can get it by listing the encryption keys.
AdminTool.exe /encryptionAtRestListKeysDecommission the Server
For total decommissioning, then also run:
AdminTool.exe /encryptionAtRestDecommissionServer /encryptionAtRestServerUid <identifier>It prints:
CMD: Decommission encryption at rest key metadata for server b6d09e52-1c84-47af-9e30-5b7c2f8a01d3
Result: success
Rows deleted: 2Check the result
- Confirm the result with List Keys
- It should no longer lists the target Server.
- A row count of 0 in is a valid result — it means there was nothing left to remove.
Security Considerations
While the Encryption at Rest architecture provides strong cryptographic protection for sensitive data, the overall security of the deployment also depends on how encryption keys, backups, and recovery information are managed throughout their lifecycle.
The recommendations in this section are intended to help administrators maintain a secure Encryption at Rest deployment, reduce the risk of accidental key loss, and ensure that encrypted data remains recoverable during server migrations, disaster recovery scenarios, and long-term operational use.
Failure to properly protect and retain encryption keys and key backups may result in the permanent loss of access to encrypted data.
- Protect Key Backups: Encryption at Rest key backups should be stored separately from the Endpoint Data Protection servers.
- Protect Backup Passwords: Backup passwords should be stored within an approved password vault.
-
Never Copy KeyRing Files:
- Do not directly copy files from %ProgramData%\EgoSecure\KeyRing\
- Use Export Key and Import Key instead.
- Retain Historical Keys: Revoked keys may still be required to access historical data, restore older database backups, or support disaster recovery scenarios. If all encrypted data has been successfully migrated to a newer key and no retained backups depend on the previous key, the key may no longer be required. Evaluate backup retention and recovery requirements before removing historical encryption keys.
- Verify backups immediately after creation: Verify newly created key backups by performing a test import before relying on them for disaster recovery.
- Regularly Validate Backups: Regularly verify that exported keys can still be imported successfully.
Troubleshooting
| Message | Possible Cause | Resolution |
|---|---|---|
| No encryption at rest keys found. | Encryption at rest is not yet in use in this database — or the Server could not read the key records | Check the Server log before acting. If it reports a database error, resolve that first; otherwise start with creating or importing your first key. |
| Error: KeyRing is not loaded on the server. Check the server logs for details. | The Server cannot open its key store | Stop — contact support |
| Error: this server is not registered. Re-register it (reconnect to the database) before performing this action. | This Server has not registered in the database | Confirm the service is running and connected to the database, then retry. If it is, run the command from another Server and contact support |
| Result: failed. Error code: 0x... | The command did not complete, and the message carries no specific cause | Check the Server log, which usually records it. Stop — contact support, quoting the code |
| Result: timeout occured (operation is still in progress...) | The command stopped waiting; the operation may still be running on the Server | Confirm the outcome with procedure 1 before repeating the command |
| WARNING: Missing keys detected | One or more Servers do not hold the key in use | Add the Key to the missing servers |
| Error: /encryptionAtRestFile and /encryptionAtRestPassword must be given together to also export the new key, or neither. | Only one of the two was supplied | Supply both and repeat |
| Error: key generation or data protection (DPAPI) failed. | The Server could not generate or protect the key | Stop — contact support |
| Error: /encryptionAtRestKeyHex or /encryptionAtRestFile parameter is required. | Self-evident | Supply the parameter and repeat |
| Error: specify either /encryptionAtRestKeyHex or /encryptionAtRestFile, not both. | Both were supplied | Supply only /encryptionAtRestKeyHex and repeat |
| Error: invalid hex key or algorithm. | The value is not valid hexadecimal | Check the value and repeat |
| Error: a key with the same thumbprint already exists in the keyring. | This Server already holds that key | Nothing to import |
| Error: data protection (DPAPI) failed. | The Server could not protect the imported key | Stop — contact support |
| Error: this file was exported from a different database. It can only be imported on a server pointing at the same database it was exported from. | The file came from a Server on another database | Export again from a Server on this database |
| Error: failed to unwrap the export file -- wrong password, or a corrupted/tampered file. | Wrong password, or a damaged file | Check the password; if it is correct, export again |
| Error: malformed export file. | The file is not a key export, or has been altered | Export again from a Server that holds the key |
| Error: a key with the same thumbprint already exists in the keyring | This Server already holds that key | Nothing to import |
| Error: failed to open /encryptionAtRestFile for reading | The path does not exist, or is not readable from this machine | Check the path, and copy the file locally if it is on a network share |
| Error: /encryptionAtRestPassword parameter is required when importing from a file. | Self-evident | Supply the parameter and repeat |
| Error: /encryptionAtRestThumbprint, /encryptionAtRestFile or /encryptionAtRestPassword parameter is required | Self-evident | Supply the |
| Error: /encryptionAtRestFile already exists. | A file of that name is already there | Repeat with a new file name |
| Error: /encryptionAtRestFile is not a valid file path | The folder does not exist, or the path is malformed | Correct the path and repeat |
| Error: failed to write /encryptionAtRestFile | The path is valid but the file could not be written — disk space, folder permissions, or a path too long | Correct the cause or choose another location, then repeat. No partial file is left behind |
| Error: key not found in the keyring. | This Server does not hold that key | Re-check with Verify Existing Key and run this on a Server that holds the key |
| Error: key wrapping (Argon2id/AES-GCM) failed. | The Server could not protect the file's contents | Stop — contact support |
| Result: failed. Error code: 0x..., and List Keys shows a WARNING: Missing keys detected: section | A Server does not hold the key in use, so the migration was refused before it started — nothing was changed | Review the missing-key warning, import the required key on the affected server, or decommission obsolete server metadata, then repeat the operation. |
| Result: failed. Error code: 0x..., with no warning section | The migration could not start, or stopped part way | Check the Server log, which usually records the cause. Stop — contact support, quoting the code |
| Error: cannot revoke the current default key. Set another key as default first. | The key is the one currently in use | Put another key into use first — Add a key to a Server, then Make as Default Key |
| Error: /encryptionAtRestThumbprint parameter is required | Self-evident | Supply the parameter and repeat |
| Result: key created, export failed | The key exists but the backup file was not written; the message names the cause | Repeat back up an encryption key before going any further |
Recovery Checklist
| Verification | Expected Result |
|---|---|
| List Keys | All expected keys are displayed. |
| Default Key | The correct key is marked as DEFAULT = YES. |
| Revoked Keys | Only retired keys are marked as REVOKED = YES. |
| Multi-Server Environment | All servers report identical thumbprints. |
| Key Backups | Backup files are available and readable. |
| Migration Status | Existing data has been migrated where required. |
| Server Inventory | Obsolete servers have been removed from key management. |
Operational Best Practices
- Export and verify every newly created key.
- Store backup files and passwords separately.
- Keep backups of all keys, including revoked keys.
- Run List Keys before and after every key operation.
- Create a database backup before running a migration.
- Clear command history after handling sensitive key material. Encryption keys, passwords, or hexadecimal key values entered on the command line may be retained in command history or administrative log