A few days back I worked on a very interesting case and when I searched on Internet I found that a lot of people are running in to the same problem which prompted me to write this blog entry.
You will run in to this issue only if you have Exchange/SMTP running on the machine.
You keep on getting these failure audits in your event viewer and you dont konw why they are coming. After some time the account listed in the failure audit just gets locked out and you have to go and unlock the account very frequently. In a lot of cases I saw this was happening in less than 30 seconds.
Event Type: Failure AuditEvent Source: SecurityEvent Category: Logon/Logoff Event ID: 529Date: 8/16/2007Time: 10:13:24 AMUser: NT AUTHORITY\SYSTEMComputer: <server>Description:Logon Failure: Reason: Unknown user name or bad password User Name: <USER> Domain: <Domain> Logon Type: 8 Logon Process: Advapi Authentication Package: Negotiate Workstation Name: <ServerNAme> Caller User Name: NETWORK SERVICE Caller Domain: NT AUTHORITY Caller Logon ID: (0x0,0x3E4) Caller Process ID: 2464 Transited Services: - Source Network Address: Source Port:
Proceed further only if you see the above text in bold in the event viewer entry.The process id 2464 is determined to be InetInfo. If yes then read further...If no you might be able to use some troubleshooting steps from this blog entry.
The interesting thing to note here is that the Logon Process is ADVAPI. ADVAPI is the DLL for advanced Windows api's and is used in a lot of OS related code. The function on which you can concentrate on for now are LogonUser, LogonUserA, LogonUserExW and LogonUserExA. The code which is generating these events is calling one of these functions for sure.
To find out the code, we can use the Debugging Tools For Windows - www.microsoft.com/whdc/devtools/debugging/default.mspx. Install them on your machine and after install just attach to InetInfo.exe (you can attach to a process by going to WinDBG and then selecting File -> Attach to Process. After that select InetInfo.exe from the list.
NOTE: The moment you do this you have stopped InetInfo and every execution is blocked. In other words what this means that InetInfo is waiting for you to do something and once you are done only then it will be able to proceed.
After that run the following commands one by one.
1) .symfix c:\symcache
2) bp ADVAPI32!LogonUserA "k 100;.time;g"
(You should be able to connect to Internet from the machine where you are Debugging as WinDBG goes to http://msdl.microsoft.com/downloads/symbols to download the PDB files for the DLL's. You will still be able to debug the process but the function names will not be correct)
After that wait for some time till the problem happens. Once you get the failure Audit in Event Viewer, scroll up in the WinDBG window to see the time when the problem happend and if you see a stack like the following it will just confirm that the failure is coming from exchange.
advapi32!LogonUserA+0x23 exps!CExchAuthContext::HrCheckClearTextLogin+0x1af exps!CExchAuthContext::HrServerNegotiateClearTextAuth+0xb6 exps!CExchAuthContext::HrServerNegotiateAuth+0x18 exps!CSessionContext::OnEXPSInNegotiate+0x14a exps!CSessionContext::OnSmtpInCallback+0x2ae smtpsvc!SMTP_CONNECTION::ProcessPeBlob+0xc1 smtpsvc!SMTP_CONNECTION::ProcessInputBuffer+0x12b smtpsvc!SMTP_CONNECTION::ProcessReadIO+0xb7 smtpsvc!SMTP_CONNECTION::ProcessClient+0x146 smtpsvc!SmtpCompletion+0x16 isatq!AtqpProcessContext+0x1db isatq!AtqPoolThread+0x1d1
(You might see the different functions if the symbols have not matched but exps.dll in the stack would be enough to point to this issue)
So why is Exchange doing that. From the call stack we can see that we are just trying to process a SMTP message that came to this server. Your next would be to check the SMTP message and get more details around it
Use Ethereal to capture a trace and after the problem has happened, stop the trace and analyze it using EtherealUse the following filter in Ethereal - smtp.rsp.parameter contains "Authentication unsuccessful"
and in the list of the packets, right click on one of them and say follow TCP Stream. Confirm that this failure for the same user (The user name and password are base64 decoded)...
So yes, this is the guy...
220 maine.anr.msu.edu Microsoft ESMTP MAIL Service, Version: 6.0.3790.3959 ready at Tue, 14 Aug 2007 14:46:08 -0400 EHLO CYF-162-WILKINS 250-maine.anr.msu.edu Hello [10.10.144.11] <---This is the guy sending the SMTP message 250-TURN 250-SIZE 250-ETRN 250-PIPELINING 250-DSN 250-ENHANCEDSTATUSCODES 250-8bitmime
250-BINARYMIME 250-CHUNKING 250-VRFY 250-X-EXPS GSSAPI NTLM LOGIN 250-X-EXPS=LOGIN 250-AUTH GSSAPI NTLM LOGIN 250-AUTH=LOGIN
250-X-LINK2STATE 250-XEXCH50 250 OK AUTH LOGIN 334 VXNlcm5hbWU6ZmFydXFp 334 UGFzc3dvcmQ6 535 5.7.3 Authentication unsuccessful.
Use a Base64 Decoder to Decode VXNlcm5hbWU6ZmFydXFp and it should out to be a user name and UGFzc3dvcmQ6 would be the password. In our case VXNlcm5hbWU6ZmFydXFp decodes (Base64 decoder) to "Username:faruqi" . Try to find out what is the IP Address 10.10.144.11 which is listed there and diagnose it further as to if it is an Internal IP or if someone is trying to HACK YOUR MACHINE.
wow, it resolved one of my customer's issues..thanks a ton!
I have a similar issue where a use account is getting locked
--------------------------Event Log from DC------------------------------------
A user account was locked out.
SYSTEM Account Name:DC01$
Account That Was Locked Out:
Caller Computer Name:Exch2 (Hub Cas Server)
--------------------------Event log from Exchange Server-----------------------
Further If I read the log from the hub cas server i get the below entry
An account failed to log on.
Logon Type: 3
Account For Which Logon Failed:
Security ID: NULL SID
Failure Reason:Account locked out.
Caller Process ID:0x1674
Caller Process Name:D:\E2k7\Bin\EdgeTransport.exe
Workstation Name:Exch2 (Hub Cas Server)
Source Network Address:-
Detailed Authentication Information:
Package Name (NTLM only):-
Unable to determine what is the possible cause which is calling the EdgeTransport.exe. Have you come across this scenario.
Fantastic, this helped so much
HI, Thank you very much for this :D I had been tackling an issue for months thinking it was remote web, RDP etc but this post showed me exactly what it was and also how the security on emails servers really should be better. So with that I am adding white-lists to all my other clients with SMTP to help protect against this. Thank you Once again.