Cascade Skyline - with Microsoft Logo and Project Support header - author Brian Smith

Friends don't let friends delete their cache or cancel queue jobs

Friends don't let friends delete their cache or cancel queue jobs

  • Comments 42

Inspired by Brian Kennemer's e-mail tag line of "Friends don't let friends assign resources to summary tasks" I thought I would get back on my soapbox about the cache and queue.  I do appreciate that there are some early bugs around custom field display that require the occasional local cache deletion - and there are a couple of rare scenarios that will leave the queue in a bad way and things need canceling.  But generally many of the situations our customers run into can be resolved without recourse to either of these actions - which can both lead to DATA LOSS!

A couple of examples from the queue:-

Project Save from Project Professional - Getting Queued

This means the data is flowing from the client cache to the server queue - and once it is all in the queue it will then be loaded into the Project Server database tables.  If the client goes away while this is happening (and this can be our fault as we don't handle Project closing very well) or the network goes down, or you hibernate your laptop as you race out of Starbucks, then the queue will just sit in this state.  If you cancel the queue job then the good data in the client cache will never see the light of day.  The correct approach is to identify from the queue where the save is coming from (the owner will display from the queue) and then get that person to re-connect their client and the getting queued should continue.  In some cases you will see the original save show as canceled but if you look in the ULS logs it will have a message along the lines of:-

PWA:http://server1/PWA, SSP:SharedServices1, User:DOMAIN\username, PSI: WinProj.PreSaveProject [T:abf8f56f-e3d1-4139-9355-55ef33aa1378][U:079d778a-2a14-455a-a52e-3141b57e75ea][S:6521e25f-5c1c-41d3-a224-7a868e161c42][D:CLIENT1\ProjConf 2][J:abf8f56f-e3d1-4139-9355-55ef33aa1378][PS_AC][3] Cancelling correlation 2439f848-3966-44b7-a645-1ff7b6914f10 as it has 1 send incomplete winproj save jobs.

which indicates the original save hadn't got very far so it cancels it from the server and starts again.  This was in fact the project that should have demonstrated this recovery at the project conference - but I didn't leave Project Professional connected to that profile for long enough (my fault - trying to present 3 hours of stuff in 75 minutes).  Another interesting tip from this queue job - CLIENT1\ProjConf 2 is the server name and the Project Server account (not user account but the "profile") used on that machine to make this queue request.

So the queue shows something like this:-

image

with the important fact that I didn't cancel anything and the save came from my client cache - and nothing was lost.

Timesheet Update - Failed and Blocking Correlation

This next example shows a couple of things - the sleeping state and that the retry does work.  As long as you fix the underlying problem.  The queue is all data driven and if the data stays the same then it will behave exactly the same.  (One definition of insanity is doing the same thing over and over expecting a different outcome - same thing with the queue).  If I submit a timesheet with administrative time then when the update is processed it puts a calendar exception in to my calendar for the non-working time.  If as a resource I am checked out then this update can go into a sleeping state (Waiting to be processed (Sleeping)) - and it wakes up every 2 minutes and tries again.  If I happen to get checked in in the meantime then all is good and the process completes.  If not then eventually it will fail.  The error shown in the queue even gives you a reasonable clue to why it failed (if you know the secret language - CICO = Check-in check-out):-

Error summary/areas:
Array
CICOAlreadyCheckedOutToYou
Queue
GeneralQueueJobFailed
Error details:

<?xml version="1.0" encoding="utf-16"?>
<errinfo>
  <array name="Array" type="System.Guid">
    <item value="079d778a-2a14-455a-a52e-3141b57e75ea">
      <error id="10101" name="CICOAlreadyCheckedOutToYou" uid="ce366c36-421b-4c47-8fa0-d68f42ba63d6" />
    </item>
  </array>
  <general>
    <class name="Queue">
      <error id="26000" name="GeneralQueueJobFailed" uid="385171b0-3ee9-4087-b308-859cb62fea53" JobUID="702e81a6-4f0e-4faf-ab78-2ab81fe60972" ComputerName="SERVER2" GroupType="TimesheetUpdate" MessageType="UpdateTimesheetMessage" MessageId="1" Stage="" />
    </class>
  </general>
</errinfo>

To recover from this error you do not need to cancel - just fix the underlying problem, which in this case was that I had my account open in Manage Users on another IE session, and then select the job and click retry.

image

This time it all works fine - even the blocked jobs can continue and other related jobs get spawned to update the reporting DB.

image

So please, please, please - deleting the local cache and canceling queue jobs should be a last resort.  There is usually a better way.

Technorati Tags: Project Server 2007

Leave a Comment
  • Please add 2 and 3 and type the answer here:
  • Post
  • This is great in theory, but almost all of the blocking queue jobs I run into do not fall into the above two problems:

    How about a Project Publish failed and blocking at 95%? Client has been connected to the Project server with Proj Pro running for hours after the publish failed, so I know it's not a client connectivity or client disappearing problem.

    I would love to retry or not delete the job, but I really have no options, that I know of. I'd be happy to learn I'm wrong...

    (part of the Queue Job Error Details below for the most recent instance of this failure. I've seen this on a couple of different servers.)

    error id="23000" name="ProjectPublishFailure" uid="512ab003-062b-4800-b00a-2fc0c9d19ca7" projectuid="417dfaa4-e157-49ba-8bd3-3520fabbb5b5" messagetype="Microsoft.Office.Project.Server.BusinessLayer.QueueMsg.ConflictResolutionMessage" messageID="180" stage="" blocking="Block"

  • Great feedback James, and I agree that there certainly isn't a one size fits all.  

    The publish jobs are less of a risk to cancel as in fact there is no data to lose - and the client really plays no part in the publish apart from making the request (once the data is saved).  When you request a publish the queue job contains no real data (nothing gets "enqueued" - there is no real payload) but just a record requesting that the draft data be copied to the published database (in simple terms). Failure at 95% in this case probably relates to a data error and I would suggest turning up the ULS logging to Verbose for the queue and then search for SRA in the logs after the failure. I'm guessing an issue with Summary Resource Assignments.  In some cases errors like these can be resolved by cancelling (nothing to lose) then delete the project from the published store ONLY (Server Settings, Delete Enterprise Objects) and then re-publish. But again - no silver bullet, we build up a set of resolutions as we come across different failure modes.

    Brian.

  • I have two projects that were created in Project 2003 and managed through Project Server 2003.  I saved the files locally and converted them in Project 2007 Professional.  When I initially saved and published the projects, I was successful.  When users submit changes to tasks, I can approve them.  However, I can't publish the changes.  Through Project Server 2007 I just receive a message indicating that there were errors.  When I try to publish the changes through Project Professional, I receive a 23000 error.  The only solution I can seem to find is to completely delete the projects and recreate them.  I've done that three times with no success.  I would really appreciate any help with this problem.  Thanks.

  • Hi Kevin,

    It would help to know what the errors are.  If they keep coming back are you starting from the same template?  Does it happen with new projects from 2007?  One thing that can cause this issue is blank lines right under a summary task - but without detailed error messages from the ULS logs I can't tell if this is the problem for you.  Also some of these types of issues are resolved in SP1 and the post SP1 rollup hotfix - so best to get updated to these if you haven't already.  Both for the client and the server.

    Best regards,

    Brian.

  • Brian, Thanks very much for your assistance.  We're actually experiencing a couple of issues related to Project Server 2007.  But specific to the publishing problem, the following XML message that is generated when I try to publish changes to the project plans.  Thanks again for your help.

    ---

    <?xml version="1.0" encoding="utf-16"?>

    <errinfo>

                   <general>

                                   <class name="">

                                                   <error id="23000" name="ProjectPublishFailure" uid="f7c183d3-04f0-4077-be0f-fe11d13995a3" projectuid="a86562d5-e459-4cb1-bdda-ac69e476aece" messagetype="Microsoft.Office.Project.Server.BusinessLayer.QueueMsg.CheckWssSiteExistsMessage" messageID="8" stage="" blocking="DontBlock"/>

                                   </class>

                                   <class name="Queue">

                                                   <error id="26000" name="GeneralQueueJobFailed" uid="33b82d22-f628-4ddc-a50f-ab9004ced3f1" JobUID="9383f36f-37a4-4a16-bc4d-6d03ca3f51bc" ComputerName="SFGSHAREPOINT1" GroupType="ProjectPublish" MessageType="CheckWssSiteExistsMessage" MessageId="8" Stage=""/>

                                   </class>

                   </general>

    </errinfo>

  • Hi Kevin,

    You would need to look at the ULS logs in c:\program files\common files\microsoft shared\web server extensions\12\logs (default location) to get more information.  See my posting on the logs at http://blogs.msdn.com/brismith/archive/2006/12/14/finding-your-way-around-the-sharepoint-logs.aspx. The 23000 and 26000 errors are just generic "queue job failed" messages - but seem to be poiting to problems with checking the existence of the WSS sites.

    Best regards,

    Brian

  • Brian,

    Thanks for the additional information.  I will check out the logs and see what I can find.  Thanks again.

    Kevin

  • Thanks for the follow up Brian.  We installed both service packs and I manually recreated the project plans that came from Project Server 2003.  Those two events seemed to resolve both issues we were experiencing.  Thanks again for your help.

    Kevin

  • Brian,

    I have had many problems opening projects in Professional 2007 through Server 2007.  The problem is related to editing a project from the server through Professional 2007.  It doesn't seem to hand checkout rights back to the server.  What strikes me is that the only way to fix the issue is by clearing my local cache.  Any thoughts?

    Thanks.

    Kevin

  • Hi Kevin,

    If you have SP1 and the post SP1 rollup hotfix applied (Client and Server) and have started with a new cache (creating another account with the Start, Microsoft Office, Office Office Tools, Microsoft Office Project Server Accounts) will acheibe this - you can use the same details, just a different name for the account) then hopefully you should not be seeing this issue still.

    Best regards,

    Brian

  • These are very good information.  However this has gotten to a point that's very frustrating.  For example, I check out a project, made some changes and saved it.  I went to publish the project and it said on the bottom that the status of checking in with 0% and the expected wait time 5s. I waited 10 minutes and the status was never changed.  I login to the PWA as admin and canceled the project on the queue. Now the Job State is 'canceled' instead of 'waiting to be processed'. From the Force Check-in, I'm not able the check in the project.  I am using microsoft office project 2007.  Why does it take a long time for a project to process?

  • Dave and Brian,

    We installed SP1 and the hotfix.  It worked for about a week.  Now I'm experiencing the same problem again.  I open the project through the server, which launches Project 2007 Professional.  I make changes, save them and publish them.  I close the project and it prompts me to check it back in, which I do.  I then verify on the server that the project is checked and it is.  I then examine my local cache and it also lists the project as checked in.  But when I try to open the project again--either through Project Web Access or through Project Professional 2007--I receive a message stating that I can only open the file as Read-Only because it's checked out in another instance.  All I can do is clear my cache.  Hopefully they will address this with SP2.

  • Hi Dave,

    There can be a number of reasons why a process takes a while - and cancelling will not make it work any faster.  I am guessing in your case that there is an older "save" job already in the queue - so no new saves or check-ins will not work until this completes - or is cancelled.  Change the history settings to see if there are any old jobs for the same project.  Dealing with that problem should then make other things work OK.

    Best regards,

    Brian.

  • Hi Kevin,

    I am not aware of a regression coming on this one - so would love to get some reproducible steps to make this happen.  Certainly opening from Pro v PWA will use different cache directories - so can lead to this kind of behavior.  Did you start with a brand new cache after applying SP1 + rollup hotfix?  

    Best regards,

    Brian.

  • Brian,

    Thanks for your continued assistance with this.  Yes, after we installed SP1 and the hotfix, I created a completely new account on the Project server.  The fix worked for about a week.  I feel what's contributing to this issue is accessing the site through an internal wireless connection.  I can reproduce the error on demand, using either cache.  For now, I've just been removing the one file I'm having difficulty with from local cache before opening.  Thanks.

    Kevin

Page 1 of 3 (42 items) 123