Recovery Locations
Recovery Locations, the host running the recovery service and processing data restores, can be added, edited, and deleted in the Recovery Locations page of the Management Console. They can be added prior to adding devices to the Standby Image plan, or on the first step of the Enable Standby Image to Hyper-V, Enable Standby Image to Azure and Enable Standby Image to ESXi wizards.
The Recovery Locations page can be found on the Management Console under the Continuity section:
The page displays several pieces of information relating to previously created recovery locations.
The columns displayed are:
- Recovery location name
- Customer
- Recovery location type
- Azure
- Hyper-V
- VMware ESXi
- Host availability
- Offline
- Online
- Requires a storage drive
Drive is required for Standby Image and Recovery Testing locations as this is the location in the file system where the new Virtual Machine files will be created. This is not required for Azure and ESXi locations
- Storage location
For Hyper-V, this can be added by clicking Add storage location, entering the drive letter or local path in the box and clicking Save
- Assigned devices
- Host storage
- Host memory capacity
- Host CPU
- Host OS
- Version
Minimum Requirements
For Standby Image and Recovery Testing, Windows Admin Center must be installed on the management machine, available here.
This is not required for One-Time Restore.
Default Hardware Configuration
By default, each Recovery Location is configured to run 5 restores in parallel with a target VM size of 4 CPU cores and 4 GB of RAM. The following minimal configuration is recommended depending on the restore target:
- Hyper-V and ESXi
- CPU - 22 Cores or more
- RAM - 32 GB or more
- Restore Target Drive Capacity - more than two times the size of the selected size of all devices combined
- Local VHD and Azure
- CPU - 12 Cores or more
- RAM - 16 GB or more
- Restore Target Drive Capacity - more than two times the size of the selected size of all devices combined
Single device restore
To start Recovery Service on a Windows Server and run a single device restore we recommend the following hardware configuration, depending on the restore format:
- Hyper-V and ESXi
- CPU - 6 Cores or more
- RAM - 6 GB or more
- Restore Target Drive Capacity - more than two times the size of the backed up data
- Local VHD and Azure
- CPU - 4 Cores or more
- RAM - 5 GB or more
- Restore Target Drive Capacity - more than two times the size of the backed up data
Also, make sure that Recovery Location operating system conforms to the requirements stated here: Virtual Disaster Recovery Requirements
Azure Requirements
To install the Recovery Location's recovery service on an Azure VM, the following requirements are necessary:
- A user created for you in your Azure tenant. The user must:
- Have access to a subscription
- Have access to a resource group you want to use to keep the Recovery Location VM and restored (target) Azure VMs
- Be able to assign permissions on virtual machines within the resource group
Additional resources for more devices
The default number of parallel restores can be adjusted depending on the hardware you use for the Standby Image Recovery Location. When configuring each recovery location, it is important to do this in a way it is neither too big (as it might slow down the restore because the host will be overloaded) nor too low (as in this case you might not use the full capacity of your computing resources and hence receive a slower than ideal performance).
While network bandwidth and disk IOPS don't have any strict requirements to run the Standby Image Recovery Service it does directly affect the restore speed and low resources can cause poor performance.
CPU and RAM might be a blocker to adding more restores in parallel. If you do not have enough CPU and/or RAM it is possible to see performance degradation, failing restores, or Virtual Machine boot issues due to “out of memory“ errors.
We recommend reserving the following additional resources for each extra device when configuring a number of parallel restores:
- Hyper-V and ESXi
- CPU - 4 cores
- RAM - 4 GB
- Local VHD and Azure
- CPU - 2 cores
- RAM - 2 GB
The requirements mentioned above are recommendations. You may set the number of cores and memory at your discretion, however, taking into account the recommendations above.
If you have already had experience using Recovery Console and have a suitable configuration, it is possible to use the same configuration for Standby Image when taking into account the recovery of the same number of devices of a similar configuration.
Recommendations for Maximum Performance
Hardware
When running multiple restores in parallel, we recommend the following to increase the performance of each machine:
- Use SSD disks with higher IOPs
- A fast network connection with good bandwidth
Both the target disk and system disk should have enough performance as the system disk may be used by system services and Hyper-V.
Hypervisor Configuration
While required to effectively mitigate certain classes of vulnerabilities, the core scheduler (enabled by default for Windows Server 2019 and newer) may also potentially reduce performance. We recommend changing the scheduler to Classic to increase performance.
This action should be done along with applying appropriate security controls to mitigate risks raised by this change.
The free Hyper-V Server 2019 ISO can be found here.
Anti-Virus Recommended Exclusions
The following are recommended to add to the anti-virus exclusions list to allow for successful restores:
Recovery Service Exclusions
AuthTool.exe [file] SYSTEM_DRIVE:\Program Files\Recovery Service\*\AuthTool.exe
unified_entry.exe [file]. SYSTEM_DRIVE:\Program Files\Recovery Service\*\unified_entry.exe
RecoveryFP.exe [file] SYSTEM_DRIVE:\Program Files\Recovery Service\*\BM\RecoveryFP.exe
VdrAgent.exe [file] SYSTEM_DRIVE:\Program Files\Recovery Service\*\BM\VdrAgent.exe
ProcessController.exe [file] SYSTEM_DRIVE:\Program Files\Recovery Service\*\BM\ProcessController.exe
ClientTool.exe [file] SYSTEM_DRIVE:\Program Files\Recovery Service\*\BM\ClientTool.exe
Virtual Machines Exclusions
*.vhd RESTORE_TARGET_DRIVE\StandbyImage\*\vm\*.vhd
*.vhdx RESTORE_TARGET_DRIVE\StandbyImage\*\vm\*.vhdx
Hyper-V Processes Exclusions
%ProgramData%\Microsoft\Windows\Hyper-V [folder]
Vmms [process]. %systemroot%\System32\Vmms.exe
Vmwp [process]. %systemroot%\System32\Vmwp.exe
Vmsp [process]. %systemroot%\System32\Vmsp.exe
Vmcompute [process]. %systemroot%\System32\Vmcompute.exe
DNS
The DNS should also be accepted if router or firewall rules are in place to allow full communication to all storage nodes:
*.cloudbackup.management
*.iaso.com
*.backup.management
*.mob.system-monitor.com
*.prd.bckp.io
System Network Configuration
When running a large number of parallel restores on the same recovery location, (around 50) a lot of network connections may be utilized. In order to improve the performance and stability of such heavily loaded systems it is recommend to adjust network configuration in the following way:
- Increase ephemeral ports count, set up new values:
start = 20000
number of ports = 45000
- Reduce
TcpTimedWaitDelay
to 5 seconds
See details here: Settings that can be Modified to Improve Network Performance - BizTalk Server