Using Netmon to figure out the source of high CPU in WMIprvse.exe

Using Netmon to figure out the source of high CPU in WMIprvse.exe

  • Comments 5

Recently I was working with a customer who had a file server experiencing high CPU in the WMIprvse.exe process. We received multiple user dumps and noted that someone or something was running the same query again and again. We needed to figure out what was running the query in a tight loop, causing the high CPU.

image

Figure 1 - Task Manager on FileServer

Before we get into the exact troubleshooting steps, let me provide some background on WMI.  Winmgmt is the WMI service within a SVCHOST process running under the LocalSystem account.  In all cases, the WMI service automatically starts when the first management application or script requests connection to a WMI namespace. For more information, see Starting and Stopping the WMI Service. To avoid stopping all the services when a provider fails, each provider is loaded into a separate host process named "Wmiprvse.exe". This also allows each instance of Wmiprvse to run under a different account with varying security. For more details you can look at the MSDN documentation on WMI.

 

I dumped out all the services in the various svchost.exe processes. You can do this from a command prompt by running the tasklist /svc command. In my instance, I found that the WinMgmt service was running in svchost, PID  452 (PID number will vary). Someone had to be making RPC calls to this svchost.exe process to run the WMI queries. It could be some local process on the machine; it could even be a process on a remote machine.

 

At this point I requested user dumps of PID 452 from the customer.  This would allow me to determine who was making the RPC calls to svchost.exe to run the WMI queries.  While the customer was uploading the dumps, we decided to get a Network Monitor trace to see if the RPC calls were coming over the network.

 

Immediately, I could see a lot of RPC traffic to the svchost.exe process(PID=452).

 

image

Figure 2 - Network Monitor Output from the FileServer. Notice the Source and destination ports and IP addresses. IP addresses are hidden by the aliases

Looking at the RPC payload, I could see the text of the WMI query. You can see this in the Hex Details Pane. The query that was running in a loop was Select * from Win32_Process. Looks like I found the source of the WMI queries.

 

At this point, we got the source IP for the RPC packets. We logged into the machine, and brought up the Task Manager.

 

image

Figure 3 - Task Manager on Remote Machine(Machine1)

Immediately we saw that there was some script running inside a Wscript.exe process.  At this point I was pretty sure that this script was the culprit. The customer was not sure what this was, and was not comfortable terminating the process.  To prove my suspicion, I had him open a command prompt and run the following command, netstat –ano.

 

image

Figure 4 - Netstat output from Remote Machine

 

From the output in Fig. 4, I could see a TCP connection created by PID 3532 (wscript.exe). Looking at the local and foreign addresses from the above output, they matched up exactly to what we were seeing in the Network Monitor trace.

 

In the above case, we already had our suspicions on the wscript.exe process; however, sometimes it might not be that easy. In that case, we could have used the netstat output to look at all connections to the file server (157.59.123.121). If there were multiple connections, then we can also narrow it down by the port number. Based on that, we could have found the PID responsible.

 

The customer called me later in the day, and told me that they had recently updated their scripts. One of their scripts had a bug which was running WMI scripts in a tight loop. Fixing the script caused the problem to go away.

 

Had the query being coming from a local process, I would have had to debug the svchost.exe process, and figure out who was making the WMI calls. However, since we could see the traffic on netmon, we didn’t need to use the debugger. Interesting way to get to the root of the problem without using a debugger!

 

 

Share this post :
Leave a Comment
  • Please add 7 and 4 and type the answer here:
  • Post
  • Netmon has helped me to solve the problem with delays in starting of MS Office Word. It appeared to be, that somehow this application or Windows Vista itself wanted to find the home network printer, which was of course not accessible in the customer's network. Removing this printer solved the problem.

  • This is pretty interesting technique-- but, "user dumps of PID 452"?   What exactly does that mean?

    [ Good question. We're saying after you determine the process ID (PID) use a tool like ADPlus to take some dumps of the process
     
    -Ron]
  • It was mentioned at the end that a debugger wasn't used, so no user dumps were needed to resolve it.

    If you use Netmon 3.4 and enable the Windows parsers, then you don't need to find the PID of Svchost (winmgmt) and just add "WMI" as your Display Filter. And that also lets you see the WMI query as the StrQuery field in Frame Details instead of just in Hex Details. Also "-anob" instead of "-ano" for Netstat will include the process name in addition to the PID. And "wmic process get commandline" lets you see the name and location of the script file that Wscript/Cscript is running.

    Step-by-step:

    1. Computer is slow so you sort Task Manager by CPU and see WmiPrvSE.exe is spiking.

    2. Download Network Monitor 3.4 and take a capture while Wmiprvse is spiking.

    3. In Netmon select "Parser Profiles", then "NetworkMonitor Parsers" then "Windows" to include the WMI parser.

    4. Type "WMI" for the Display Filter and click Apply to view just WMI traffic.

    5. The frame with description "WMI:IWbemServices:ExecQuery Request" shows the IP address of the remote computer making the WMI query (Ipv4: Src = <IP address> in Frame Details).

    6. Expand "WMI: IWbemServices:ExecQuery Request" in Frame Details and look at the StrQuery field for the specific WMI query being made (Select * From SomeWMIClass).

    7. Go to the remote machine with that source address and sort Task Manager by CPU to see if Wscript.exe or Cscript.exe is running (other processes can make WMI queries but VBS scripts run in either Wscript or Cscript, and they are a common source of WMI queries).

    8. "Netstat -anob" shows which processes are connecting to a foreign address matching the IP address of the computer where Wmiprvse.exe is spiking.

    9. "Wmic process get commandline" shows the command line for running processes, which for Wscript\Cscript would include the name and location of the script, for example: "C:\WINDOWS\System32\CScript.exe" "C:\Scratch\BadScript.vbs"

    Note that if the WMI authentication level is packet privacy (pktPrivacy), you will still see the source IP address but the WMI query string (Select * From Whatever) will be encrypted and not visible in the network capture. WBEMTEST defaults to just "packet" when connecting, but the WMIC command-line tool defaults to pktprivacy. For scripts, it would depend on how they are making the connection to WMI, for example, this Vbscript code would result in encrypted query strings because pktPrivacy is specified:

    Set objWMIService = GetObject("winmgmts:{authenticationLevel=pktPrivacy}!root\cimv2")

  • We are seeing wmiprvse.exe with high CPU cycle running on our windows 2008r2 servers. We have tried many of the hot fixes but still no issue. We used process monitor and see that it is tied to tzres.dll and HKCU.  Please let us know what you suggest as we have ran out of ideas. Thank you for you help ahead of time.

    [Hi szhosting.  Unfortunately we are not able to provide one on one support through this blog.  If the information in this article is not sufficient to help solve your problem, you can request support at http://support.microsoft.com/.]

  • For me it was the McAfee Outlook plugin causing the issue, the largest contributing factor anyhow.  kc.mcafee.com/.../index

    To fix you must uninstall McAfee then reinstall using a command line either manually or by adding it to your EPO push from the server tasks.

    Hope that helps someone else!

    James

Page 1 of 1 (5 items)