inner_banner

DR Testing Woes – Light at the End of the Tunnel

December 23, 2021 / superadmin
blog

Root Cause Analysis for Failed Restores

Kevin Pescatello
Information Security Engineer
12/11/2021

Problem

When you are using Synology Active Backup for Business  (ABB) and the import of a backed-up server fails in Hyper-V Backups.  This is what you need to do.

Synology

Requirements Summary

Ensure the following;

  • Group policy is made and assigned to perspective host servers with one policy only
  • The account used to manage the backup jobs is part of the Hyper-V Operators group and local administrator.
  • Make sure the Computer account, and account used to manage the hypervisor are allowed to login as a service. This will save you time in troubleshooting. Not currently listed in Synology’s Hyper-V Requirements.  New issue discovered DSM 6.2.2 and DSM 7.0

 

Problem Details

During the DR test that we do every year we noticed that servers were not able to be restored as they did two months ago. This led to a ticket being created with Synology on 10/25/2021 and lasted over 43 days with multiple exchanges. All settings were the same as in August when a successful restore was done for SERVER01 to preserve it since the time and effort in the test domain should ABC one day setup other domains for testing their web application. The only upgrade to the system occurred in June 2021 for the DSM upgrade. Change controls helped us focus on the problem since all changes are reviewed and monitored.

 

In troubleshooting we found out that the Full Restore option to a different location would not allow you to A) Browse to new server destination folder  B) Complete the restore Wizard Process. There is a work around where you have to select from the column menu burger button Cluster Resource. See image below

You end up selecting all the checkmarks then must deselect it on the menu bar. Then you can select other Hyper-V Hosts.

 

While doing this we discovered that only the instant restore would work as the  FULL and Chosen Host restore was still unsuccessful. It would run for 2 minutes and then fail.

 

As soon as we added the Computer Account or the Server Name of the target host in group policy of the policy applied to that server, to login as a service, it kept failing. You will get various error logs in Microsoft Hyper-V VMMS > Admin log that will point you in the right direction. See below.

Here is what I found in Hyper-V VMMs logs Admin

The last successful restore was Aug-17-2021 SERVER01 Here is the event ID on the cluster node1 to prove that. From Hyper-V-VMMs logs had the following event ID showing a good restore 18304

Next error was event ID 22042 Received invalid or corrupted data for a virtual machine migration. This led me to search on what accounts security wise would logon and work to restore the server. This is where the light bulb went off. See below screenshot

Solution

In order to fix my Backup and restore process I added the server name for the target host of the affected domain to logon as a service in the group policy applied to it, changed the account used for backups from local to domain account, changed the password of that account, did the work around for the server and folder viewing by selecting cluster on the first button, but then deselecting on the column heading, click next all the way through and its working.

…..and we’re back in business.

 

Conclusion

In order to get backups restored to a new destination server and not the original location, you need to add the destination computer to login as a service under Windows Settings > Security Settings > Local Security Policy > User Rights Assignment as well as make sure the service account connecting to the hypervisor is in the Hyper-V Operator Role and local admin role on the target machine.  Not the source machine, it does not matter for the restore. As the host taking the new restore guest VM needs to be able to logon as a service and run in those privilege levels to restore the server successfully.

 

Synology did not discover this work around Netwerk Guardian LLC did. Synology was gracious enough to send 2 sticks of memory for my efforts which I am grateful. I get to geek out more. If you are looking for a backup solution and cannot afford nor need any compliance reporting, Synology is your ticket.  It works, license free, just bring your hard drives and your data either VMs or files. You can stand up a VM anywhere as long as you can talk to the hypervisor. If they ever get compliance reporting, this will be a competitor for sure.