Cascade Skyline - with Microsoft Logo and Project Support header - author Brian Smith

My Queue is Stuck! - How to manage your queue service in Project Server 2007

My Queue is Stuck! - How to manage your queue service in Project Server 2007

The new world of Project Server 2007 and the architectural changes are catching a few of our customers out - and I thought I'd share a few tips and tricks for keeping the queue flowing - and some tips for getting things moving again if they appear to have stopped. 

First I will point to a great TechNet article on the Queue and as you will all have read this then my explanations will make more sense :).

Under Server Settings in Project Web Access the Manage Queue option allows you to see what is happening in the project and timesheet queues - if you don't have admin access then the Personal Settings will give you a glimpse of your queue jobs.  The latter option may not however give you the complete picture and allow you to see what might be ahead or you.  It is like being stuck on the highway and not being able to see around the corner to where the flashing lights are... 

So lets start with some definitions:-

Waiting to be processed - means exactly what it says.  Once I get to the front of the queue then I am ready to go.  But there may be other active jobs ahead that will stop my job starting even if I am first in line.  The queue is clever enough that it will hold jobs back if their processing would interfere with other running jobs.  An example might be a publish job that will need to wait for a cube build to finish.

Processing - means that I made it to the front of the queue, was allocated a thread and am working away!  One thing I have noticed is that the % complete indicator doesn't always make you think that "processing" is happening - but generally it is.  Looking in the ULS logs, event logs or at general server activity (particularly the Microsoft.Office.Project.Server.Queuing.exe process should help if you have continued doubts that processing is moving along.

Skipped for optimization - is the queue's way of telling you that it is not going to do the same thing twice.  Some queue jobs have a payload (such as saving a project) and others are merely instructions (such as publish a project).  If several of the same instruction are in the queue, then only one needs to be actioned.  An example might be working on a project and publishing a few times during a period of time.  If the queue was busy all of these jobs might be sitting waiting for a while - and then rather than doing each in turn it just needs to do one.  It is just an instruction to publish the content of the saved project.  This would not happen with a queue job that had a payload as each of these contains real data that needs to be applied - rather than just an instruction to do something with data somewhere else.

Getting Queued - appears to be one of the more confusing messages.  I mentioned above that some jobs, such as save project from Project Professional, have a payload. This payload goes into the queue as a group of related messages, which then get processed once they reach the front of the queue.  Getting queued means that these messages are going into the queue.  It is possible that the Getting Queued message appears for some time because a very large project is coming in across a very slow link.  One other potential problem that can break things is if this flow in of messages does not complete.  Perhaps the Project Manager saving the project shuts down Project before it completes - or perhaps goes out of wireless range midway through the process.  Either way the Getting Queued could sit there for some time. To fix this up find the person who has this project in mid-save and get them to reconnect and complete the job.  As a last resort you can cancel the Getting Queued - but YOU WILL LOSE DATA!  Any changes the Project Manager made will not get saved.  To protect you from inadvertently canceling one of these jobs we add a check box under Advanced options labeled "Cancel jobs getting enqueued" which will need to be checked before these jobs can be canceled.

Failed and Not Blocking correlation - is a failure that is isolated and not stopping any other jobs from processing.  The term correlation is used to group related queue jobs together.  There should be an associated error message and entries in the log to help explain the problem.

Failed and Blocking correlation - means that something bad happened that is also blocking other things in the related group.  If a save fails then a publish could not continue would be one example.

Success - is the one message we like to see!  It can also be useful to sometimes show the Success messages (by default they are not shown in the Manage Queue display) as it is a way of seeing if the queue is working at all.  Adding the completion state of Success through the options on the manage queue page is how this is done.

Canceled - means what it says.  It could have been canceled by a user, but it is also possible for jobs to be canceled by the server.  One example would be a failure early on in a save from Project Professional.  A job would have been added to the queue for the save - but reconnection may lead to cancellation of this job and the addition of another save job - it really depends hoe far the save got before the problem.  I simulate bad things like this by pulling my network cable out just after hitting save - just to see what happens!

I will follow up with another posting on the queue with some further tips on troubleshooting -but my parting gift is a guide to what the dialogs at the bottom of Project Professional 2007 mean during a save.

    • Blue progress bar - saving to local cache
    • Synchronizing data to server... - The data is going from the local cache to the PSI and being passed into the queue (Getting Queued)
    • Save job xx% complete.  Expected Wait Time 20s - The job is either Waiting to be processed or more likely Processing.  Once you see this then it is safe to close Project Professional - your saves are safely in the queue!

Technorati Tags: Project Server 2007

Leave a Comment
  • Please add 5 and 5 and type the answer here:
  • Post
  • Hi Brian,

    We have a similar problem as KH however, our database files are on a local drive. The queue is stuck on one job "Reporting (Resource Sync)" with a job state of "Processing". I'm getting the same errors 7758, 7761 and 7754. Checked the ULS logs with no luck.

    Our SQL server is a VM and our Project Server is on a different VM.

    Is there a way to forcefully cancel this job from the database?

    Any help is greatly appreciated!

    Thanks,

    MAV

  • Una breve y concisa explicación sobre los estados de la cola y como enfrentar algunos problemas cotidianos

  • Hello,

    I am currently having the opposite problem. My queue is running away.  The server load is running around 80% CPU and memory usage.  When I try to select "Manage Queue" the request times out.  

    I dialed the queue setting back to one thread and set all the polling and retries to the maximum time to try to give resource back to the server.  

    Even so, this has been going on for two days now.  At first I thought I should let it run and it will eventually clear itself up.  But that does not seem to be the case.

    Any help is welcome!

    smckeown

  • Hello,

    Could you please help me with the error I get when I try to publish my project:

    <?xml version="1.0" encoding="utf-16"?> <errinfo>

      <general>

        <class name="Reporting message processor failed">

          <error id="24006" name="ReportingProjectChangeMessageFailed" uid="1c8dc596-cbd0-46e8-a4a8-313c8e8057d7" QueueMessageBody="Project UID='3e148e2d-f56c-4ff8-8fbb-9ad54a4c6c72'. PublishType='ProjectPublish'" Error="Violation of PRIMARY KEY constraint 'PK_MSP_EpmTask'. Cannot  insert duplicate key in object 'dbo.MSP_EpmTask'.&#xD;&#xA;The  statement has been terminated." />

          <error id="24006" name="ReportingProjectChangeMessageFailed" uid="6d6b1619-8726-476d-9a8f-8c374d23f966" QueueMessageBody="Project UID='3e148e2d-f56c-4ff8-8fbb-9ad54a4c6c72'. PublishType='ProjectPublish'" Error="The INSERT statement conflicted with the FOREIGN KEY constraint  &quot;FK_MSP_EpmAssignment_ProjectUID_TaskUID&quot;. The conflict  occurred in database &quot;ProjectServer_Reporting&quot;, table  &quot;dbo.MSP_EpmTask&quot;.&#xD;&#xA;The statement has been terminated." />

          <error id="24006" name="ReportingProjectChangeMessageFailed" uid="064cf886-db55-4e94-befa-6521a4908ac7" QueueMessageBody="Project UID='3e148e2d-f56c-4ff8-8fbb-9ad54a4c6c72'. PublishType='ProjectPublish'" Error="The INSERT statement conflicted with the FOREIGN KEY constraint  &quot;FK_MSP_EpmAssignment_ProjectUID_TaskUID&quot;. The conflict  occurred in database &quot;ProjectServer_Reporting&quot;, table  &quot;dbo.MSP_EpmTask&quot;.&#xD;&#xA;The statement has been terminated." />

          <error id="24006" name="ReportingProjectChangeMessageFailed" uid="07c2c2b0-6077-4a75-85b8-78b5d339b4a0" QueueMessageBody="Project UID='3e148e2d-f56c-4ff8-8fbb-9ad54a4c6c72'. PublishType='ProjectPublish'" Error="The INSERT statement conflicted with the FOREIGN KEY constraint  &quot;FK_MSP_EpmAssignment_ProjectUID_TaskUID&quot;. The conflict  occurred in database &quot;ProjectServer_Reporting&quot;, table  &quot;dbo.MSP_EpmTask&quot;.&#xD;&#xA;The statement has been terminated." />

          <error id="24006" name="ReportingProjectChangeMessageFailed" uid="d10e72bd-2c32-4250-a1a2-c3babca56c02" QueueMessageBody="Project UID='3e148e2d-f56c-4ff8-8fbb-9ad54a4c6c72'. PublishType='ProjectPublish'" Error="The INSERT statement conflicted with the FOREIGN KEY constraint  &quot;FK_MSP_EpmAssignment_ProjectUID_TaskUID&quot;. The conflict  occurred in database &quot;ProjectServer_Reporting&quot;, table  &quot;dbo.MSP_EpmTask&quot;.&#xD;&#xA;The statement has been terminated." />

          <error id="24006" name="ReportingProjectChangeMessageFailed" uid="f69cdfe6-c5cf-4cdc-8af6-f8018317d958" QueueMessageBody="Project UID='3e148e2d-f56c-4ff8-8fbb-9ad54a4c6c72'. PublishType='ProjectPublish'" Error="The INSERT statement conflicted with the FOREIGN KEY constraint  &quot;FK_MSP_EpmAssignment_ProjectUID_TaskUID&quot;. The conflict  occurred in database &quot;ProjectServer_Reporting&quot;, table  &quot;dbo.MSP_EpmTask&quot;.&#xD;&#xA;The statement has been terminated." />

        </class>

        <class name="Queue">

          <error id="26000" name="GeneralQueueJobFailed"  uid="f8f68b30-8124-41ae-9ab2-bc96e50a455e" JobUID="205f41d4-18ba-4a83-859d-0826ad566b44" ComputerName="PROJSRV" GroupType="ReportingProjectPublish" MessageType="ReportProjectPublishMessageEx" MessageId="1" Stage="" />

        </class>

      </general>

    </errinfo>

    What could be the cause?

    Thank you.

    Ruse

  • Have the problem with my queue being stuck, and was told by a project expert that the order in which the services are starting is incorrect.  He said there was a disconnect between project and sql and there were permission problems, Restarting the server and starting the services in the right order worked.  Well the system admin did an upgrade and rebooted the server now everything is stuck again.  The project expert did not tell me the order that the services had to start.  Do you know what order the services have to start in order to not have a queue problem?

    Thanks

    Karen

  • Hi Karen,

    I believe there was an issue that if SQL Server was restarted then the Project Services would get left hanging - but this was fixed some time ago in a service pack or CU.  If you are still suffering from something that looks like this then re-start the Project queue and event services. But it could be something else...

    Best regards,

    Brian.

  • Plz provide the resulation if phase same kind of problem

  • Hi Ashu,

    There isn't a single resolution - and it is not clear what problem you are facing.

    Best regards,

    Brian

  • We are experiencing the same issue with the queue being stuck. The status is currently "Waiting to be Processed" and it inumber 1 on the list. I have checked previous jobs back 12 months and nothing is still hanging there. I have re-started the services twice and rebooted the machine, still no go.

    I am trying to save a new project to the system for the first time.

    Also, which logs should I bee looking at

    Any ideas.

  • Hi Andrew,

    Sounds like the queue service isn't running.  Check using task manager - you should see two instances of the queue service and two of the event service.  If you see none then the service is probably stopped - if you see just one then for some reason it hasn't spawned the 2nd instance which processes the queue jobs.  Restarting the service would be the best first step - and if things are still bad then the logs created when the queue tried to restart should give a good idea what went wrong.

    Best regards,

    Brian.

  • We just restored 4 databases from prod to staging and then re-provisioned the site. This kicked off some 700 "Reporting (Resource Sync)" jobs and "User Synchronization" job. Looks like last one has been processed while others are still in "Waiting to be processed" mode. We have ~6000 resources and ~1500 projects in prod.

    We started getting strange errors in Event Viewer right away. Since then I re-provisioned the site again, restarted the server, restarted both services on the server, gave admin account dbowner rights on all 4 databases. Right now it keeps logging errors in Event Viewer on the server every minute and all jobs in the queue are at 0% completion. Nothing is being processed. Any idea on what could I be missing?

    Thanks,

    Luba

  • Addition to previous message- we are getting the following 3 types of errors repeatedly:

    Standard Information:PSI Entry Point:

    Project User: xxxx

    Correlation Id: 1cf8978f-bd5b-4b50-92c9-1c64904ccff6

    PWA Site URL: http://staging.domain.com/ProjectServer

    SSP Name: PRISM Stage SSP

    PSError: Success (0)

    Queue System Restarting due to an unexpected error. Queue type (Project, Timesheet queue etc): ProjectQ.  Error: System.Xml.XmlException: Root element is missing.

      at System.Xml.XmlTextReaderImpl.Throw(Exception e)

      at System.Xml.XmlTextReaderImpl.ParseDocumentContent()

      at System.Xml.XmlLoader.Load(XmlDocument doc, XmlReader reader, Boolean preserveWhitespace)

      at System.Xml.XmlDocument.Load(XmlReader reader)

      at System.Xml.XmlDocument.LoadXml(String xml)

      at Microsoft.Office.Project.Server.Errors.PSError..ctor(String xmlString)

      at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.CleanupCompletedThreads()

      at Microsoft.Office.Project.Server.BusinessLayer.Queue.Receiver.ThreadEntry()

    ===========================

    Standard Information:PSI Entry Point:

    Project User: xxxx

    Correlation Id: 1cf8978f-bd5b-4b50-92c9-1c64904ccff6

    PWA Site URL: http://staging.domain.com/ProjectServer

    SSP Name: PRISM Stage SSP

    PSError: Success (0)

    An unxpected exception occurred in the Project Server Queue. Queue type (Project Queue/Timesheet Queue): ProjectQ. Exception details: Root element is missing..

    ======================

    Standard Information:PSI Entry Point:

    Project User: xxxxx

    Correlation Id: 1cf8978f-bd5b-4b50-92c9-1c64904ccff6

    PWA Site URL: http://staging.domain.com/ProjectServer

    SSP Name: PRISM Stage SSP

    PSError: Success (0)

    An unxpected exception occurred in the Project Server Queue. Queue type (Project Queue/Timesheet Queue): ProjectQ. Exception details: The site with the id f36ddbb6-5927-4c2b-8c32-d484681f1575 could not be found..

    Luba

  • Not sure what happened here Luba.  The jobs kicked off after a provision may take a while - or could cause a deadlock.  Did you re-restore the databases after the first provision?  The bad XML error looks like something is not good in the queue tables.  Might need a support incident for one of my colleagues to help you out.

    Best regards,

    Brian.

  • Hi Brain,

     I am getting very strange behaviour on one of my Master Project. When-ever, I published the job it throws below exception. Also, another strange behaviour about this project is, it automatically stops generating a notification/alert message (particularly issue with this project only out of 200+ projects). Your expert adivse is needed to grap the problem and move towards the resolution stage :)

    Your ReportingProjectPublish job failed.  Its current state is FailedNotBlocking.  It was 0% complete.  It entered the queue at 02/13/2010 15:09:04.

    To get more information about the job failure, please go to Project Web Access.  Select Personal Settings from the left menu.  Then select My Queued Jobs.

    The errors returned from the queue are as follows:

    Error ID: 24006

    Error ID: 26000

    Detailed error below - send it to the administrator for more detailed troubleshooting.

    <?xml version="1.0" encoding="utf-16"?>

    <errinfo>

     <general>

       <class name="Reporting message processor failed">

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="30e983c1-4e90-493e-a2bb-605bd86db52c" QueueMessageBody="Project UID='2f29f726-9cd7-40d8-96c3-370e4ae05004'. PublishType='ProjectPublish'" Error="Exception of type 'System.OutOfMemoryException' was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="fe94bad8-2e64-418e-848d-0bd846ec0788" QueueMessageBody="Project UID='2f29f726-9cd7-40d8-96c3-370e4ae05004'. PublishType='ProjectPublish'" Error="Exception of type 'System.OutOfMemoryException' was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="b8b36074-c231-42e9-9102-0caa7a890d51" QueueMessageBody="Project UID='2f29f726-9cd7-40d8-96c3-370e4ae05004'. PublishType='ProjectPublish'" Error="Exception of type 'System.OutOfMemoryException' was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="08508e9e-4c34-4aeb-b998-8fddf6d16368" QueueMessageBody="Project UID='2f29f726-9cd7-40d8-96c3-370e4ae05004'. PublishType='ProjectPublish'" Error="Exception of type 'System.OutOfMemoryException' was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="f361ac92-35d6-4f80-afbd-72b34bdc03af" QueueMessageBody="Project UID='2f29f726-9cd7-40d8-96c3-370e4ae05004'. PublishType='ProjectPublish'" Error="Exception of type 'System.OutOfMemoryException' was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="ab2bdad0-420f-4c27-9505-ebe1b03d2748" QueueMessageBody="Project UID='2f29f726-9cd7-40d8-96c3-370e4ae05004'. PublishType='ProjectPublish'" Error="Exception of type 'System.OutOfMemoryException' was thrown." />

       </class>

       <class name="Queue">

         <error id="26000" name="GeneralQueueJobFailed" uid="c8b5407b-ce9c-4303-9a5d-c14594ca948c" JobUID="810177be-e177-4911-bec1-3c6e33c6f328" ComputerName="EPMAPP" GroupType="ReportingProjectPublish" MessageType="ReportProjectPublishMessageEx" MessageId="1" Stage="" />

       </class>

     </general>

    </errinfo>

    Regards,

    Hyder Zaidi

  • Hi Brian,

    I've been trying to resolve the subject error in my Project Server 2007 farm.  Nothing appears to be broken, but I get this warning message every couple of minutes.  The only reference I've been able to find on the Web has this error occurring in SharePoint where the solution was to use the command: stsadm -o sync -deleteolddatabases 1 to resolve it.  Unfortunately, this command does not exist in Project Server.

    Any suggestions?

    Thanks!!!

Page 5 of 7 (102 items) «34567