As we in the US (or most of us) have just changed our clocks for daylight savings this is a great time to talk synchronization of server clocks. The first section talks about things running slow, and was also applicable to 2007. If this doesn’t persuade you then the second section deals with things stopping – which ought to be a good enough reason!
My Farm is Running Slow!
One possible reason for the farm, or more precisely the queue jobs, running slowly (apart from underpowered configuration) is clock synchronization between the application layer and the database (SQL Server) layer. The reason behind this is that the queue may timestamp a job that should be run immediately – but the time (Now!) is not actually now, but a few seconds or minutes into the future (or even hours!). This was also covered by a posting for 2007. The queue therefore delays each job by a little. May not make much difference if the time difference is small and the server is busy – but worth avoiding as the resolution is so easy. Please note – setting the Application Server clock ahead of the WFE will NOT speed up your farm by making the jobs start before they arrive :).
My Farm isn’t Authenticating me!
This one is new to Project Server 2010 and is related to the use of WCF. The typical symptom of this is ‘An unexpected error has occurred’ and when you track the GUID in the error message back to the ULS logs (typically on the Web Front End server) then the message will be something like:
WcfSendRequest: RemoteAddress: 'http://server:32843/6151221ab8334845843fc3c2bdf789ba/PSI/Authentication.svc' Channel: 'Microsoft.Office.Project.Server.Interfaces.IAuthentication' Action: 'http://schemas.microsoft.com/office/project/server/webservices/Authentication/GetUserByName' MessageId: 'urn:uuid:66949013-4d76-4216-b077-8c81d6d025fb'
System.ServiceModel.FaultException: An error occurred when verifying security for the message.
and you may also see a 500 internal server error logged from the web service (or in a netmon trace). Where the clock comes in to play is that the SOAP message sent to the web service has an expiry time, which is set for 5 minutes. If the Application Server happens to be outside this timeframe then it will not accept the request – and return 500, Internal Server Error, with the inner message ‘An error occurred when verifying security for the message’. If you Bing for the general WCF error you may find solutions suggesting extending the expiry – please don’t - just put the clock right!
These errors are from Project Server web services, and the basic SharePoint pages do not authenticate through the same pages, so other pages in the farm may be rendering just time.
So please make sure your servers all have the same time – and better still, the right time!