The new world of Project Server 2007 and the architectural changes are catching a few of our customers out - and I thought I'd share a few tips and tricks for keeping the queue flowing - and some tips for getting things moving again if they appear to have stopped.
First I will point to a great TechNet article on the Queue (*** 12/18/2014 update - try this link now http://technet.microsoft.com/en-us/library/cc197395(v=office.12).aspx ***) and as you will all have read this then my explanations will make more sense :).
Under Server Settings in Project Web Access the Manage Queue option allows you to see what is happening in the project and timesheet queues - if you don't have admin access then the Personal Settings will give you a glimpse of your queue jobs. The latter option may not however give you the complete picture and allow you to see what might be ahead or you. It is like being stuck on the highway and not being able to see around the corner to where the flashing lights are...
So lets start with some definitions:-
Waiting to be processed - means exactly what it says. Once I get to the front of the queue then I am ready to go. But there may be other active jobs ahead that will stop my job starting even if I am first in line. The queue is clever enough that it will hold jobs back if their processing would interfere with other running jobs. An example might be a publish job that will need to wait for a cube build to finish.
Processing - means that I made it to the front of the queue, was allocated a thread and am working away! One thing I have noticed is that the % complete indicator doesn't always make you think that "processing" is happening - but generally it is. Looking in the ULS logs, event logs or at general server activity (particularly the Microsoft.Office.Project.Server.Queuing.exe process should help if you have continued doubts that processing is moving along.
Skipped for optimization - is the queue's way of telling you that it is not going to do the same thing twice. Some queue jobs have a payload (such as saving a project) and others are merely instructions (such as publish a project). If several of the same instruction are in the queue, then only one needs to be actioned. An example might be working on a project and publishing a few times during a period of time. If the queue was busy all of these jobs might be sitting waiting for a while - and then rather than doing each in turn it just needs to do one. It is just an instruction to publish the content of the saved project. This would not happen with a queue job that had a payload as each of these contains real data that needs to be applied - rather than just an instruction to do something with data somewhere else.
Getting Queued - appears to be one of the more confusing messages. I mentioned above that some jobs, such as save project from Project Professional, have a payload. This payload goes into the queue as a group of related messages, which then get processed once they reach the front of the queue. Getting queued means that these messages are going into the queue. It is possible that the Getting Queued message appears for some time because a very large project is coming in across a very slow link. One other potential problem that can break things is if this flow in of messages does not complete. Perhaps the Project Manager saving the project shuts down Project before it completes - or perhaps goes out of wireless range midway through the process. Either way the Getting Queued could sit there for some time. To fix this up find the person who has this project in mid-save and get them to reconnect and complete the job. As a last resort you can cancel the Getting Queued - but YOU WILL LOSE DATA! Any changes the Project Manager made will not get saved. To protect you from inadvertently canceling one of these jobs we add a check box under Advanced options labeled "Cancel jobs getting enqueued" which will need to be checked before these jobs can be canceled.
Failed and Not Blocking correlation - is a failure that is isolated and not stopping any other jobs from processing. The term correlation is used to group related queue jobs together. There should be an associated error message and entries in the log to help explain the problem.
Failed and Blocking correlation - means that something bad happened that is also blocking other things in the related group. If a save fails then a publish could not continue would be one example.
Success - is the one message we like to see! It can also be useful to sometimes show the Success messages (by default they are not shown in the Manage Queue display) as it is a way of seeing if the queue is working at all. Adding the completion state of Success through the options on the manage queue page is how this is done.
Canceled - means what it says. It could have been canceled by a user, but it is also possible for jobs to be canceled by the server. One example would be a failure early on in a save from Project Professional. A job would have been added to the queue for the save - but reconnection may lead to cancellation of this job and the addition of another save job - it really depends hoe far the save got before the problem. I simulate bad things like this by pulling my network cable out just after hitting save - just to see what happens!
I will follow up with another posting on the queue with some further tips on troubleshooting -but my parting gift is a guide to what the dialogs at the bottom of Project Professional 2007 mean during a save.
Technorati Tags: Project Server 2007
I tried to build the cube,but my jobs are getting stuck in the queue.i even tried to force checkin the projects it gives me again the same state
then i cancelled all the jobs in the queue and even restarted the service but still the queue is stuck.
I would check the ULS logs. If the cube job was waiting in the queue then something was stopping it. First check the queue is running (see other postings on this) but if it is then you need to fix what is blocking this specific job. Any amount of stopping and starting services isn'tr going to change the data that is driving the queue.
I have an annoyng problem with tasks publishing operation. I create a new task, I approve it, but the changes are not visible in Project Center. The only solution is to open the project with Project Proffesional and publish the project from there and in my scenario this is not possible.
Every time I approve the task the job fails (Failed but not blocking correlation) with the following error message:
Reporting message processor failed
<?xml version="1.0" encoding="utf-16"?>
<class name="Reporting message processor failed">
<error id="24006" name="ReportingProjectChangeMessageFailed" uid="9ce09def-e83c-4c57-a965-ddec33fa532f" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Violation of PRIMARY KEY constraint 'PK_MSP_EpmTask'. Cannot insert duplicate key in object 'dbo.MSP_EpmTask'.
The statement has been terminated." />
<error id="24006" name="ReportingProjectChangeMessageFailed" uid="78bac897-e890-438e-87e2-d6bd11692557" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />
<error id="24006" name="ReportingProjectChangeMessageFailed" uid="f46e6d07-1a03-4b17-8b4b-8a4e9252a933" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />
<error id="24006" name="ReportingProjectChangeMessageFailed" uid="7421949b-0ceb-4f7f-a7ea-87e118ca3b63" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />
<error id="24006" name="ReportingProjectChangeMessageFailed" uid="cacd3b25-fa97-4831-beaf-a8876ffeeb05" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />
<error id="24006" name="ReportingProjectChangeMessageFailed" uid="fa15a345-96c4-4bb2-b4ed-06c7f9f11602" QueueMessageBody="Project UID='1e5ae5f8-863c-4a7d-adc7-68300b338516'. PublishType='ProjectPublish'" Error="Cannot transfer project : '1e5ae5f8-863c-4a7d-adc7-68300b338516'. A FK constraint was thrown." />
<error id="26000" name="GeneralQueueJobFailed" uid="8ad9ba5e-54ec-4d67-8fea-7b770fc1694b" JobUID="eaba023e-60b2-47e3-ac38-cb6be8bb1349" ComputerName="PROJECT07" GroupType="ReportingProjectPublish" MessageType="ReportProjectPublishMessageEx" MessageId="1" Stage="" />
Is there any way to solve this issue?
Thanks for your response ! The issue stands unresolved still. I tried to create a new SSP to build another PWA site, but failed. It throws "Access denied". Looks like some n/w policies are affecting the setup. However, the logs do not say anything.
I think you are on the right track with permissions. Check that all the service accounts and the account you are using have correct access. Do the logs really say nothing? Perhaps this is a permission issue too?
Hi Brian, we have tried the range of measures described in the blog to get a project published, but cannot get it through. After checking right through the log as far back as we can it appears that the original publish was terminated at this level.
What are the best steps for recuperating the file and retaining the data, and what is best practice if you cannot 'save' the file?
After a fresh install of Project Server 2007 and SP1, nothing in my queue ever got processed. I found that the queue service was not running. I tried to start it and it failed.
"The Microsoft Office Project Server Queue Service service failed to start due to the following error:
The service did not respond to the start or control request in a timely fashion."
I tried various service accounts, from a domain account to Local System and nothing worked. Have you seen this before???
I think you are seeing this issue - it also needs a restart to become effective.
It may be a time out issue getting a response from the Service Control manager. One workaround my Colleague Eric Marinho da Silva found is as follows. It involves making changes to the regsitry so only try this if you have backups of your registry and know what you are doing:-
When we start to run the Queue or Event service we have the following event ids:
Service control manager
Timeout (30000 milliseconds) waiting for the to connect.
The service did not respond to the start or control request in a timely fashion
To fix the issue, please use the following steps:
1. Run "C:\Program Files\Common Files\Microsoft Shared\Web Server Extensions\12\bin\psconfig –cmd upgrade –inplace b2b –wait –force".
2. In registry editor add/modify the following values on "HKLM\System\CurrentControlSet\Control":
a) ServicesPipeTimeout DWORD 60000 (decimal).
b) WaitToKillServiceTimeout changed to “120000”.
3. Reboot the server
Please let me know if this works for you.
That worked. The service is running and my queue has been cleared. Thank you!
FYI - I am running App and Web together on a virtual server with a separate SQL Server backend. The VM has 2GB RAM and is running on a physical server with Microsoft Virtual Server 2005 R2 SP1 on Intel VT. The SQL Server is an 8 core server with 32GB RAM and fibre channel SAN.
Hope you are still looking at this message thread.
I have discovered a fix for the sleeping thread issue where you see
"[RDS] The RDS message will go to sleep because a RDB Refresh is in progress."
in the SharePoint log.
Basically, Project Server is stuck in a mode where it thinks a reporting database refresh is in progress and other jobs are getting blocked. I'm not sure how you get into this state; possibly a hard server restart in the middle of an update. You can go into the reporting database using SQL Server Management Studio and call the stored procedure MSP_RefreshModeCompleted. Here's the SQL statements
DECLARE @RC int
DECLARE @AdminItemTypeUID uniqueidentifier
DECLARE @RefreshStatus int
SET @AdminItemTypeUID = 'c795e698-6248-4fa4-a788-6b605eedcfa6'
SET @RefreshStatus = 0
EXECUTE @RC = [ProjectServer_Reporting].[dbo].[MSP_RefreshModeCompleted]
Once this procedure is executed, values in the MSP_AdminStatus table will be updated. The next time the job queue checks the status it will be able to complete the queued reporting jobs.
Although I am usually against any direct database changes you are spot on with this posting - and the only thing I would add is for anyone using this script to be absolutly certain this is their scenario and that the reporting database refresh is not still running before trying this "fix".
Thanks for sharing!
In reply to the posting of Mikey on the 1st of Feb. 2008:
We are experiencing the same kind of problems you state with several or our
projects aswell. We are still waiting for an implementation of SP1 by our IT
department, so can't confirm this would fix it.
Here is a part of the string we get on one of the failing project we intent
to publish to the reporting database:
<error id="24006" name="ReportingProjectChangeMessageFailed"
Error="Cannot transfer project : 'bfd73c23-3d2b-4149-8b92-30d77f8f96bb'. A FK
constraint was thrown." />
Have you found out how to fix or resolve these kind of problems? This is a
big problem as our projects have got Actual Work hours within the
project-files, but in the published database these hours keep on growing and
in the reporting database this proces has come to a standstill... so making
reports with Data Analysis seems to be worthless now.
Hope you know how to help us.
In addition to the above, I also have noticed that the concerning projects (which seem not to be published in reporting db again) are missing in the reports coming from the OLAP-Cube/Data Analysis.
In Project Server (publish db) the current data from the concerning projects are available to (re)view.
So now not only the data is failing to update into the reporting db, but also the older data is missing.
What should I do to resolve this problem?
It would be expected that you wouldn't see these in the cubes if they are not getting to the reporting db. The error says FK problem which stands for foreign key. I suspect a custom field or possibly even a resource that is referenced in the project failed to get to the reporting database. Try opening and saving all lookup tables, then the same for custom fields (which will republish) and then use ProjTool from the SDK to re-publish - using the full publish option. Check the queue frequently and address any errors that occur. I can't say how to address as I don't know what the errors might be :).
So when you say "get them to reconnect and complete the job". Do you mean just have them open Project Pro and connect or do they have to connect and open the project schedule?