Cascade Skyline - with Microsoft Logo and Project Support header - author Brian Smith

My Queue is Stuck! - How to manage your queue service in Project Server 2007

My Queue is Stuck! - How to manage your queue service in Project Server 2007

The new world of Project Server 2007 and the architectural changes are catching a few of our customers out - and I thought I'd share a few tips and tricks for keeping the queue flowing - and some tips for getting things moving again if they appear to have stopped. 

First I will point to a great TechNet article on the Queue (*** 12/18/2014 update - try this link now http://technet.microsoft.com/en-us/library/cc197395(v=office.12).aspx ***) and as you will all have read this then my explanations will make more sense :).

Under Server Settings in Project Web Access the Manage Queue option allows you to see what is happening in the project and timesheet queues - if you don't have admin access then the Personal Settings will give you a glimpse of your queue jobs.  The latter option may not however give you the complete picture and allow you to see what might be ahead or you.  It is like being stuck on the highway and not being able to see around the corner to where the flashing lights are... 

So lets start with some definitions:-

Waiting to be processed - means exactly what it says.  Once I get to the front of the queue then I am ready to go.  But there may be other active jobs ahead that will stop my job starting even if I am first in line.  The queue is clever enough that it will hold jobs back if their processing would interfere with other running jobs.  An example might be a publish job that will need to wait for a cube build to finish.

Processing - means that I made it to the front of the queue, was allocated a thread and am working away!  One thing I have noticed is that the % complete indicator doesn't always make you think that "processing" is happening - but generally it is.  Looking in the ULS logs, event logs or at general server activity (particularly the Microsoft.Office.Project.Server.Queuing.exe process should help if you have continued doubts that processing is moving along.

Skipped for optimization - is the queue's way of telling you that it is not going to do the same thing twice.  Some queue jobs have a payload (such as saving a project) and others are merely instructions (such as publish a project).  If several of the same instruction are in the queue, then only one needs to be actioned.  An example might be working on a project and publishing a few times during a period of time.  If the queue was busy all of these jobs might be sitting waiting for a while - and then rather than doing each in turn it just needs to do one.  It is just an instruction to publish the content of the saved project.  This would not happen with a queue job that had a payload as each of these contains real data that needs to be applied - rather than just an instruction to do something with data somewhere else.

Getting Queued - appears to be one of the more confusing messages.  I mentioned above that some jobs, such as save project from Project Professional, have a payload. This payload goes into the queue as a group of related messages, which then get processed once they reach the front of the queue.  Getting queued means that these messages are going into the queue.  It is possible that the Getting Queued message appears for some time because a very large project is coming in across a very slow link.  One other potential problem that can break things is if this flow in of messages does not complete.  Perhaps the Project Manager saving the project shuts down Project before it completes - or perhaps goes out of wireless range midway through the process.  Either way the Getting Queued could sit there for some time. To fix this up find the person who has this project in mid-save and get them to reconnect and complete the job.  As a last resort you can cancel the Getting Queued - but YOU WILL LOSE DATA!  Any changes the Project Manager made will not get saved.  To protect you from inadvertently canceling one of these jobs we add a check box under Advanced options labeled "Cancel jobs getting enqueued" which will need to be checked before these jobs can be canceled.

Failed and Not Blocking correlation - is a failure that is isolated and not stopping any other jobs from processing.  The term correlation is used to group related queue jobs together.  There should be an associated error message and entries in the log to help explain the problem.

Failed and Blocking correlation - means that something bad happened that is also blocking other things in the related group.  If a save fails then a publish could not continue would be one example.

Success - is the one message we like to see!  It can also be useful to sometimes show the Success messages (by default they are not shown in the Manage Queue display) as it is a way of seeing if the queue is working at all.  Adding the completion state of Success through the options on the manage queue page is how this is done.

Canceled - means what it says.  It could have been canceled by a user, but it is also possible for jobs to be canceled by the server.  One example would be a failure early on in a save from Project Professional.  A job would have been added to the queue for the save - but reconnection may lead to cancellation of this job and the addition of another save job - it really depends hoe far the save got before the problem.  I simulate bad things like this by pulling my network cable out just after hitting save - just to see what happens!

I will follow up with another posting on the queue with some further tips on troubleshooting -but my parting gift is a guide to what the dialogs at the bottom of Project Professional 2007 mean during a save.

    • Blue progress bar- saving to local cache
    • Synchronizing data to server...- The data is going from the local cache to the PSI and being passed into the queue (Getting Queued)
    • Save job xx% complete.  Expected Wait Time 20s - The job is either Waiting to be processed or more likely Processing.  Once you see this then it is safe to close Project Professional - your saves are safely in the queue!

Technorati Tags: Project Server 2007

Leave a Comment
  • Please add 5 and 1 and type the answer here:
  • Post
  • I tried to build the cube,but my jobs are getting stuck in the queue.i even tried to force checkin the projects it gives me again the same state

    then i cancelled all the jobs in the queue and even restarted the service but still the queue is stuck.

  • Hi Devender,

    I would check the ULS logs.  If the cube job was waiting in the queue then something was stopping it.  First check the queue is running (see other postings on this) but if it is then you need to fix what is blocking this specific job.  Any amount of stopping and starting services isn'tr going to change the data that is driving the queue.

    Brian.

  • Hello!

    I have an annoyng problem with tasks publishing operation. I create a new task, I approve it, but the changes are not visible in Project Center. The only solution is to open the project with Project Proffesional and publish the project from there and in my scenario this is not possible.

    Every time I approve the task the job fails (Failed but not blocking correlation) with the following error message:  

    Error summary/areas:

    Reporting message processor failed

    ReportingProjectChangeMessageFailed

    ReportingProjectChangeMessageFailed

    ReportingProjectChangeMessageFailed

    ReportingProjectChangeMessageFailed

    ReportingProjectChangeMessageFailed

    ReportingProjectChangeMessageFailed

    Queue

    GeneralQueueJobFailed

    Error details:

    <?xml version="1.0" encoding="utf-16"?>

    <errinfo>

     <general>

       <class name="Reporting message processor failed">

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="9ce09def-e83c-4c57-a965-ddec33fa532f" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Violation of PRIMARY KEY constraint 'PK_MSP_EpmTask'. Cannot insert duplicate key in object 'dbo.MSP_EpmTask'.&#xD;&#xA;The statement has been terminated." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="78bac897-e890-438e-87e2-d6bd11692557" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="f46e6d07-1a03-4b17-8b4b-8a4e9252a933" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="7421949b-0ceb-4f7f-a7ea-87e118ca3b63" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="cacd3b25-fa97-4831-beaf-a8876ffeeb05" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />

         <error id="24006" name="ReportingProjectChangeMessageFailed" uid="fa15a345-96c4-4bb2-b4ed-06c7f9f11602" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />

       </class>

       <class name="Queue">

         <error id="26000" name="GeneralQueueJobFailed" uid="8ad9ba5e-54ec-4d67-8fea-7b770fc1694b" JobUID="eaba023e-60b2-47e3-ac38-cb6be8bb1349" ComputerName="PROJECT07" GroupType="ReportingProjectPublish" MessageType="ReportProjectPublishMessageEx" MessageId="1" Stage="" />

       </class>

     </general>

    </errinfo>

    Is there any way to solve this issue?

  • hi Brian,

    Thanks for your response ! The issue stands unresolved still. I tried to create a new SSP to build another PWA site, but failed. It throws "Access denied". Looks like some n/w policies are affecting the setup. However, the logs do not say anything.

    -anupom

  • I think you are on the right track with permissions.  Check that all the service accounts and the account you are using have correct access.  Do the logs really say nothing?  Perhaps this is a permission issue too?

  • Hi Brian, we have tried the range of measures described in the blog to get a project published, but cannot get it through. After checking right through the log as far back as we can it appears that the original publish was terminated at this level.

    What are the best steps for recuperating the file and retaining the data, and what is best practice if you cannot 'save' the file?

  • After a fresh install of Project Server 2007 and SP1, nothing in my queue ever got processed. I found that the queue service was not running. I tried to start it and it failed.

    "The Microsoft Office Project Server Queue Service service failed to start due to the following error:

    The service did not respond to the start or control request in a timely fashion."

    I tried various service accounts, from a domain account to Local System and nothing worked. Have you seen this before???

  • I think you are seeing this issue - it also needs a restart to become effective.

    It may be a time out issue getting a response from the Service Control manager.  One workaround my Colleague Eric Marinho da Silva found is as follows.  It involves making changes to the regsitry so only try this if you have backups of your registry and know what you are doing:-

    When we start to run the Queue or Event service we have the following event ids:

    7009

    Service control manager

    Timeout (30000 milliseconds) waiting for the     to connect.

    7000

    Service control manager

    The service did not respond to the start or control request in a timely fashion

    To fix the issue, please use the following steps:

    1. Run "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\bin\psconfig –cmd upgrade –inplace b2b –wait –force".

    2. In registry editor add/modify the following values on "HKLM\System\CurrentControlSet\Control":

    a) ServicesPipeTimeout   DWORD   60000  (decimal).

    b) WaitToKillServiceTimeout changed to “120000”.

    3. Reboot the server

    Please let me know if this works for you.

  • That worked. The service is running and my queue has been cleared. Thank you!

    FYI - I am running App and Web together on a virtual server with a separate SQL Server backend. The VM has 2GB RAM and is running on a physical server with Microsoft Virtual Server 2005 R2 SP1 on Intel VT. The SQL Server is an 8 core server with 32GB RAM and fibre channel SAN.

  • Bohdan

    Hope you are still looking at this message thread.

    I have discovered a fix for the sleeping thread issue where you see

    "[RDS] The RDS message will go to sleep because a RDB Refresh is in progress."

    in the SharePoint log.

    Basically, Project Server is stuck in a mode where it thinks a reporting database refresh is in progress and other jobs are getting blocked.  I'm not sure how you get into this state; possibly a hard server restart in the middle of an update.  You can go into the reporting database using SQL Server Management Studio and call the stored procedure MSP_RefreshModeCompleted.  Here's the SQL statements

    DECLARE @RC int

    DECLARE @AdminItemTypeUID uniqueidentifier

    DECLARE @RefreshStatus int

    SET @AdminItemTypeUID = 'c795e698-6248-4fa4-a788-6b605eedcfa6'

    SET @RefreshStatus = 0

    EXECUTE @RC = [ProjectServer_Reporting].[dbo].[MSP_RefreshModeCompleted]

      @AdminItemTypeUID

     ,@RefreshStatus

    Once this procedure is executed, values in the MSP_AdminStatus table will be updated.  The next time the job queue checks the status it will be able to complete the queued reporting jobs.

  • Hi Scott,

    Although I am usually against any direct database changes you are spot on with this posting - and the only thing I would add is for anyone using this script to be absolutly certain this is their scenario and that the reporting database refresh is not still running before trying this "fix".  

     Thanks for sharing!

    Best regards,

    Brian.

  • In reply to the posting of Mikey on the 1st of Feb. 2008:

    Hello,

    We are experiencing the same kind of problems you state with several or our

    projects aswell. We are still waiting for an implementation of SP1 by our IT

    department, so can't confirm this would fix it.

    Here is a part of the string we get on one of the failing project we intent

    to publish to the reporting database:

    <error id="24006" name="ReportingProjectChangeMessageFailed"

    uid="b0f26439-385b-4529-ac3f-e3c5edc1afde" QueueMessageBody="Project

    UID='bfd73c23-3d2b-4149-8b92-30d77f8f96bb'. PublishType='ProjectPublish'"

    Error="Cannot transfer project : 'bfd73c23-3d2b-4149-8b92-30d77f8f96bb'. A FK

    constraint was thrown." />

    Have you found out how to fix or resolve these kind of problems? This is a

    big problem as our projects have got Actual Work hours within the

    project-files, but in the published database these hours keep on growing and

    in the reporting database this proces has come to a standstill... so making

    reports with Data Analysis seems to be worthless now.

    Hope you know how to help us.

    Kind Regards,

    Wijgert IJlst

  • In addition to the above, I also have noticed that the concerning projects (which seem not to be published in reporting db again) are missing in the reports coming from the OLAP-Cube/Data Analysis.

    In Project Server (publish db) the current data from the concerning projects are available to (re)view.

    So now not only the data is failing to update into the reporting db, but also the older data is missing.

    What should I do to resolve this problem?

  • Hi Wijgert,

    It would be expected that you wouldn't see these in the cubes if they are not getting to the reporting db.  The error says FK problem which stands for foreign key.  I suspect a custom field or possibly even a resource that is referenced in the project failed to get to the reporting database.  Try opening and saving all lookup tables, then the same for custom fields (which will republish) and then use ProjTool from the SDK to re-publish - using the full publish option.  Check the queue frequently and address any errors that occur.  I can't say how to address as I don't know what the errors might be :).

    Best regards,

    Brian.

  • Hi Brian,

    So when you say "get them to reconnect and complete the job". Do you mean just have them open Project Pro and connect or do they have to connect and open the project schedule?

Page 3 of 7 (104 items) 12345»