We are seeing a few customers report this error in SCCM. Unfortunately, the sinvproc.log is not very specific about what the problem is. This does prevent the client's software inventory data from getting inserted into the database, causing the .sic file to be placed into the badsinv folder and a resync generated (creating a cycle of these errors followed by resyncs). The software inventory scan does complete successfully on the client, this is a server side error.
As of today, we are seeing this caused by a file called ~0000.exe with a time stamped year of 1601. This is a file used by Symantec's Ghost product.
The path to this file may be:
C:\Documents and Settings\All Users\Application Data\Symantec\Ghost\AutoInstall\Installed Applications\~0000.exe
There are 2 known workarounds:
1. Delete the file on the client and run a full inventory.
2. Exclude the folder that this file lives in from being scanned and run a full inventory.
The sinvproc.log will indicate the client its processing when the error occurs:
Processing Inventory for Machine: CLIENT123
...
WARNING: Exception Detected: Unspecified error~~ !!
Make sure these 2 entries reference the same thread Id (the hex number at the end of the line, as sinvproc can by multithreaded):
WARNING: Exception Detected: Unspecified error~~ !! SMS_SOFTWARE_INVENTORY_PROCESSOR 1/9/2008 11:45:52 AM 13768 (0x35C8)
As of today, the ~0000.exe with a year time stamp of 1601 is the only know file that is causing this error. I will update the blog if I hear of any others.
Update (5/19/2008)
From Symantec
http://service1.symantec.com/SUPPORT/on-technology.nsf/docid/2008031311003660
Hotfix for SCCM:
http://support.microsoft.com/kb/950653/en-us
Sorry for the delay...
Before we look at specific MOM logs and what information they contain, let's talk about the types of logging available.
In part 1, we talked about the .mc8 logs and the registry keys that are used to tweak logging. There are a couple of other registry keys that we need to talk about.
Das Logging (Database Access Services) - this logging (under dllhost*.mc8 log files), records the DAS activity - data into the Onepoint database.
Its set using the following key:
HKLM \ Software \ Mission Critical Software \ DASServer \ LoggingFlags set to ffff. (0 is disabled)
Tracelevel (covered in Part 1) must be set to 9 for Das Logging to be enabled.
To activate it, you have to restart the MOM Das Com+ Application. These logs are written to the profile of the DAS account, under Local Settings \ Temp \ Microsoft Operations Manager.
The Das log is useful for troubleshooting SQL type errors, or data stream type issues, like seeing the performance counter data that is being inserted into Onepoint.
Das logging should not be enabled unless you are troubleshooting a database related issue - keeping Das Logging enabled can cause performance degradation in writing to the Onepoint database.
A second type of logging is dbconnector logging. Dbconnector logging is described in 901049:
http://support.microsoft.com/?kbid=901049
Basically, enabling this will cause all data items (events, performance data, alerts, and discovery data) that are inserted into the Onepoint database, to also be written to dbconnector.xml, under %windir% \Temp\Microsoft Operations Manager.
This file will grow very large very quickly, so do not leave this logging enabled unless you are trying to capture specific data from your agents.
Finally, DebugView from Sysinternals is a great utility for getting additional logging from MOM, especially during setup (in addition to the verbose MSI logs), and the MMPC (MOM to MOM Product Connector) execution. In fact, DebugView provides verbose DAS type logging for the MMPC, normal DAS logging does not capture any additional MMPC database activity.
In the 3rd part (which will be out much sooner that part 2), we will go through the other MOM 2005 logs and what they are used for.
This topic will be covered in a few different posts, but let's start with the basics of logging in MOM 2005.
Logging is set in the registry on both the management servers and clients in this key:
HKLM \ Software \ Mission Critical Software \
Values of note:
Tracelevel - this is the verbosity level of logging. Values are ffffffff (disabled), and 0 thru 9. The higher the number, the more verbose the logging. By default, tracelevel is set to 1. When troubleshooting an issue, tracelevel 6 is typically verbose enough to indicate an error condition. Tracelevel should only be set to 9 when debugging an issue; do not leave the value at 9; you could see a performance hit with MOM.
Tracecircularlines - this is the number of lines that are stored in a MOM log file. When this limit is reached, the current log is renamed and a new current log is started. The naming convention will be discussed shortly.
Traceinitseconds - this is the amount of time that will elapse when the service is started when the log file is renamed to an (init) log. There is no line limit for the (init) log, strictly a time limit. The default is 90 seconds.
Naming convention
Let's start with the basic .mc8 MOM logs.
Initially, the MOM Service log is called momservice.mc8 (stored in %windir%\temp\Microsoft Operations Manager\ by default). When the traceinitseconds value elapses (90 seconds by default), the momservice.mc8 is renamed to momservice(init).mc8. The current log then becomes momservice(b).mc8. After this, when the tracecircularlines value is met in the momservice(b).mc8, it is renamed to momservice(a).mc8 and the new current log is momservice(b).mc8.
In addition, when you start the MOM service, the existing momservice(init).mc8, momservice(b).mc8 and the momservice(a).mc8 are appended with a 1 to become momservice(init)1.mc8, momservice(b)1.mc8 and momservice(a)1.mc8. MOM will keep 4 copies of these (the 4 last runs of the MOM service - i.e. momservice(init)4.mc8).
The momservice*.mc8 logs should not be confused with the momservice.log (different extension) - the momservice.log contains computer discovery logging where as the momservice*.mc8 logs contain such things as agent communications, rule processing, notification activities, etc.).
The next post will list the different MOM 2005 logs and what they contain.
In MOM 2005 SP1, if the Management Server action account is set to the local system account, any attempt to install, uninstall, upgrade, or update agent settings will fail if you specify another account to install the agent with (in a scenario where the Management Server action account does not have local administrator rights on the target agent) will immediately fail - no details provided.
If you enable logging (tracelevel = 6), the momservice(b).log (on the management server) will contain the error:
Wrn:Failed to disconnect remote connection\\xx.xx.xx.xxx , error text = 2250:This network connection does not exist.
This looks a lot like the issue described in 912998 (http://support.microsoft.com/kb/912998/en-us) where the Network access: Do not allow storage of credentials or .NET Passports for network authentication group policy is enabled. Make sure this is not enabled on the Management Server. If it's not enabled, most likely your Management Server action account is set to Local System.
When you installed MOM initially, you had to specify an account for the Management server action account. You can set the Management Server action account to Local System by setting:
HKEY_LOCAL_MACHINE\SOFTWARE\Mission Critical Software\OnePoint\Configurations\ <configuration_group_name>\AA\ActionIdentityMode to 0
This is documented in 891602 (http://support.microsoft.com/kb/891602/en-us).
Restart the MOM Service for this change to take effect.
You can confirm what the Management Server action account is currenly set to using the setactionaccount.exe utility that is installed in the MOM folder (Program Files\Microsoft Operations Manager 2005 by default).
This command will give you the account:
setactionaccount <configuration_group_name> -query
If the Management Server action account is Local System, the response will be:
"Providers and responses run under the services process identity"
To change the Management Server action account back to a domain account, you need to do the following:
1. Set HKEY_LOCAL_MACHINE\SOFTWARE\Mission Critical Software\OnePoint\Configurations\ <configuration_group_name>\AA\ActionIdentityMode to 1
2. Run the setactionaccount.exe:
setactionaccount.exe <configuration_group_name> -set <domain> <password>
The utility will prompt you for the password twice. Once the action account is set back to a domain account, you will be able to upgrade, install, uninstall, and update agent settings specifying an account (other than the action account) with local administrator rights on the target agent.
