Restart Hyper-V VM on heartbeat lost

I have a Windows Server 2012R2 hyper-v host. It is running a number of VMs, including a few linux VMs. I have one linux VM that has been occationally, but somewhat regularly crashing somewhere in the kernel, resulting in the VM becoming completely unresponsive. I know hyper-v has a "heartbeat" facility, and it successfully detects the VM crashing as a loss-of-heartbeat.
However, I'm at a loss as to how to set up hyper-v so that it actually does anything with the heartbeat signal. Apparently at one point, there was the option to reset the VM if a heartbeat is lost for > 1 minute, but it's either been removed, or placed somewhere I cannot find. I only have this one server, so doing things like clustering or fail-over are not really viable (this is a home lab server). Before anyone comments, Yes, I am trying to track down the cause of the crash in the VM, but it's irregularity has been making it difficult.

• hyper-v
• windows-server-2012-r2

AFAIK, VM monitoring is only available for Windows Server 2012 and Windows Server 2012 R2 virtual machines.

@joeqwerty - The heartbeat monitoring works fine on linux. The "Heartbeat OK/Heartbeat Lost" status indicator is fully functional. The linux support of hyper-v is actually quite good. There are paravirtualized drivers for all the base functions, full guest support, and the latest kernel versions even support memory ballooning.

Unless I'm missing what you mean (it's just disabled for non Server 2012 VMs?), guest additions/heartbeat/memory balloning work on Windows 7 as well.

2 Answers 2

powershell is your rescue:

$VM = Get-VMIntegrationService -VMName test-vm -Name Heartbeat if ($VM.PrimaryStatusDescription -ne "OK")

just add this to task scheduler.

If You VM does not support heartbeat then modify this to ping the VM instead of checking the HB.

I needed a solution for monitoring the heartbeat of all Hyper-V virtual machines and automatically preforming a hard reset when the VM locked up. As you may already know. it seems as though you can do this using Hyper-V when you have a cluster setup, but with only a single Hyper-V host, that is not possible without some custom scripting.

I found a nice VBS the monitors the heartbeat of all systems, and even has a configurable retry count and grace period in the latest version. The original script and description can be found here. It'd be a good idea to schedule this to run with system startup (don't just place it in the startup folder, because that would require the Hyper-V Administrator to login to the system before the script would start running).

Usage: - Copy VMHeartBeat.vbs script to a location of your choice - Either double click on the script or open a command prompt,change to the folder where the script is copied and then run cscript.exe VMHeartBeat.vbs

Note: If you double click on the script wscript.exe engine or whichever is your default scripting engine will be used to run the script. When you use cscript.exe, you need to keep the command window always open.

This script will create a log file at the same location as the script and keep appending the success/failure events to it. Log file name is VMHeartBeatEvents.log

That is it. You are all set to detect the VM heart beat loss and then perform a hard reset.

To test if the script is working or not, you can download NotMyFault.exe and then copy it to the VM. Now, just run VMHearBeat.vbs on Hyper-V host and open up notmyfault.exe inside the VM, select a bug and click “Do Bug”. Based on what kind of a bug you select, VM may blue screen. At this point in time, VMHearBeat.vbs would have detected the heart beat loss and performed a hard reset of the virtual machine. You may check the log file, as mentioned earlier, to see if the hard reset was successful or not.

For the sake of preserving the script, I've also pasted the latest version of the .VBS code (with retry count and grace period) below.

UPDATE: The latest version of the script actually had some bugs, so the script below has been modified and I confirmed it is working on Server 2012R2. \v2 needed to be added to line #20 and 10 needed to be changed to 11 on line #4 of the VMHardReset function.

Option Explicit 'Define Event log constants here Const EVENT_INFO = 4 Const EVENT_ERROR = 1 Const RETRY_COUNT = 3 Const CHECK_EVERY_X_SECS = 6 Dim objSummaryInfo, objWMIService, objVMService, intValArray, objVMitems Dim InParam, OutParam, objItem, strQuery Dim Job, objShell Dim colArgs, strLog, dict, compName Set colArgs = WScript.Arguments.Named 'Get Command Line arguments in to some variables strLog = colArgs.Item("log") Set objShell = Wscript.CreateObject("Wscript.Shell") Set objWMIService = GetObject("winmgmts:\\" & "." & "\root\virtualization\v2") Set objVMService = objWMIService.ExecQuery("SELECT * FROM Msvm_VirtualSystemManagementService").ItemIndex(0) intValArray = Array(0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47,48,49,50,51,52,53,54,55,56,57,58,59,60,61,62,63,64,65,66,67,68,69,70,71,72,73,74,75,76,77,78,79,80,81,82,83,84,85,86,87,88,89,90,91,92,93,94,95,96,97,98,99,100,101,102,103,104,105,106,107,108,109,110,111,112,113,114,115,116,117,118,119,120,121,122,123,124,125,126,127,128,129,130,131,132,133,134,135,136,137,138,139,140,141,142,143,144,145,146,147,148,149,150,151,152,153,154,155,156,157,158,159,160,161,162,163,164,165,166,167,168,169,170,171,172,173,174,175,176,178,179,180,181,182,183,184,185,186,187,188,189,190,191,192,193,194,195,196,197,198,199) Set dict = CreateObject("Scripting.Dictionary") While (1) Wscript.echo(CStr(Now()) +": Testing Loop ") objVMItems = objVMService.GetSummaryInformation(NULL,intValArray,objSummaryInfo) for each objItem in objSummaryInfo compName = objItem.ElementName if (not dict.Exists(compName)) then dict.add compName,0 end if If (objItem.EnabledState=2 AND (objItem.HeartBeat = 6 OR objItem.HeartBeat = 13)) Then dict.Item(compName)=dict.Item(compName)+1 WriteLog(" **"+compName+"** missed heartbeat check; count "+CStr(dict.Item(compName))) if (dict.Item(compName) >= RETRY_COUNT ) then dict.Item(compName)= 0 Wscript.echo(CStr(Now()) +": Now performing a hard reset on "+compName) VMHardReset(compName) end if Else dict.Item(compName)= 0 End If Next Set objVMItems=Nothing Set objItem = Nothing Set objSummaryInfo = Nothing Wscript.Sleep CHECK_EVERY_X_SECS * 1000 Wend Set objWMIService = Nothing Set objVMService = Nothing Set InValArray = Nothing Function VMHardReset(vmElementName) Dim objvm, strLine Set objvm = GetComputerSystem(vmElementName) Set InParam = objvm.Methods_("RequestStateChange").InParameters.SpawnInstance_() InParam.RequestedState=11 Set OutParam = objvm.ExecMethod_("RequestStateChange",InParam) If (TrackJob(OutParam)) Then strLine = "Virtual Machine: " & vmElementName & " has been successfully recovered from a hearbeat failure" 'Write Success Event to text log WriteLog(strLine) Set strLine=Nothing 'Write a Windows Event Log objShell.LogEvent EVENT_INFO, strLine Else strLine = "Virtual Machine: " & vmElementName & " could not be recovered from a hearbeat failure" 'Write Failure Event to text log WriteLog(strLine) Set strLine=Nothing 'Write a Windows Event Log objShell.LogEvent EVENT_ERROR, strLine End If Wscript.echo(CStr(Now()) +": "+strLine) Set InParam = nothing Set objvm = nothing Set vmElementName = Nothing End Function Function GetComputerSystem(vmElementName) strQuery = "select * from Msvm_ComputerSystem where ElementName = '" & vmElementName & "'" set GetComputerSystem = objWMIService.ExecQuery(strQuery).ItemIndex(0) 'Set GetComputerSystem = Nothing Set strQuery = Nothing End Function Function TrackJob(OutParam) Dim Job If (OutParam.ReturnValue = 0) Then TrackJob=True ElseIf (OutParam.ReturnValue <> 4096) Then TrackJob=False Else Set Job = objWMIService.Get(OutParam.Job) While (Job.JobState = 3) or (Job.JobState = 4) Set Job = objWMIService.Get(OutParam.Job) Wend If (Job.JobState <> 7) Then TrackJob=False Else TrackJob = True End If End If Set OutParam = Nothing Set Job = Nothing End Function Sub WriteLog(line) Dim objFileStream, objFileSystem Set objFileSystem = CreateObject("Scripting.FileSystemObject") Set objFileStream = objFileSystem.OpenTextFile("VMHeartBeatEvents.log", 8, true, -1) objfileStream.WriteLine CStr(Now()) +": "+line objfileStream.Close Set objFileSystem = Nothing End Sub