Cascade Skyline - with Microsoft Logo and Project Support header - author Brian Smith

  • Brian Smith's Microsoft Project Support Blog

    Project Server 2010: Orphan baselines breaking the reporting publish

    • 15 Comments

    *** Update 7/31/2013 - This was fixed in the June 2012 Cumulative Update for Project 2010 – and was a client fix - http://support.microsoft.com/kb/2598351.  Customers can load the latest Cumulative Update for Project 2010 and it will include this fix.  The fix will stop the problem happening again but does not fix up any existing issues – so they may need to clean up the database using the guidance below.  If you need help then open a support incident with us – we do not charge for bug related incidents – so we can help you through this.  You should also use the feature on the server to control which patch level  of Project 2010 can connect to Project Server to ensure that this problem is not re-introduced by a user with an unpatched version of Project 2010. ***

     

     

    This problem has been around for a while and I know some customers were running into it very soon after the release, but we had been struggling to get a repro and understand exactly what was causing it.  We now understand the root cause and have a fix coming hopefully in the June 2012 Cumulative Update for Project Professional 2010 (no promises – but that is the current target) and there are some ways of working that can limit your chances of running into this – so decided we should share this to avoid continued inconvenience until we get the fix out there.

    First lets take a look at the symptoms.  The most usual indication of the problem, as the title suggests, is orphan baseline values leading to the error when publishing – a Failed But Not Blocking Correlation problem on a Reporting (Project Publish) job that will show several of the following errors if you click through for the error details:

    ReportingProjectChangeMessageFailed (24006) - The INSERT statement conflicted with the FOREIGN KEY constraint "FK_MSP_EpmTaskBaseline_ProjectUID_TaskUID". The conflict occurred in database "ProjectServer_Reporting", table "dbo.MSP_EpmTask". The statement has been terminated..

    GeneralQueueJobFailed (26000) - ReportingProjectPublish.ReportProjectPublishMessageEx

    These failures are for the reporting job – so will mean that reports based on the reporting database, and any fresh OLAP cube builds could be missing data.

    Sometimes there may also be a crash on saving, either with a fairly generic MSSOAP 16 Send Incomplete error from Project Professional 2010 (though a subsequent save will work fine), or from PWA a queue error -

    GeneralQueueException (9131) A Project Operation failed due to a Queue Exception. Sub Job ID is: . Exception details are: System.NullReferenceException: …at Microsoft.Office.Project.DataEdit.Assignments.AssignmentCalendarUpdateHelper.ConvertActualContourToElapsed(,,,

    There may then be issues with users accessing timesheets – The view failed to load.  Press OK to reload this view… (and OK will not help).

    image

    The error that will be found in the ULS logs will refer to a Calendar whose UID cannot be found…

    Exception occurred in method Microsoft.Office.Project.Server.BusinessLayer.Statusing.StatusingGetMyWorkForGridJson System.InvalidOperationException: CacheProjectBaseCalendars could not find project calendar for project. CalUid=0c13de33-2a07-4310-b091-c77990d9dd6a   

    The root of all these issues is that when you use any of the Save & Send options (XML, CSV, Excel etc.) that we are incorrectly changing some of the GUIDs associated with entities such as the tasks and calendars.  Now this isn’t affecting the main tasks and assignment GUIDs as these bad values are not persisted back to the database – but we do however create a new baseline for these non-existent new task GUIDs, and can also save a bad calendar GUID – which leads to the Timesheet problem.

    First the best way to avoid this issue, and then on to the detection and clean up at the database level.

    If you do need to use Save & Send then the best practice until we release the fix for this is to first save the plan to the server, and publish if you need to.  Then do whatever you need to with Save & Send, and then immediately after this – close and check in the plan – but do not re-save to the server.  Discard changes if it asks – but of course you will have needed to save BEFORE you did the Save & Send (just making sure you are paying attention) to avoid losing any changes you really needed.  As the bad stuff will also get persisted to the local cache, this is one of those rare occasions when you will find me suggesting that the project is removed from the local cache – after ensuring that the save and check-in completed successfully.

    WARNING – the following steps are direct queries against the Project Server databases – please be sure you are working against the right databases when using these – and have a database backup should any problems occur.

    The detection of this condition is pretty straightforward, as we are just looking for baselines that exist for a task that does not exist, so the following query executed against the Draft database will do this (Change the name to match your specific DBs – the default ProjectServer_ names are used below:

    -- Detect for orphan baseline task records that can cause reporting publish job failures.

    USE ProjectServer_Draft -- specify the appropriate draft database

    select PROJ_NAME, MTB.PROJ_UID,TASK_UID,TB_BASE_NUM from MSP_TASK_BASELINES MTB
    inner join MSP_PROJECTS MP on MTB.proj_uid=MP.proj_uid
    where TASK_UID not in (select TASK_UID from MSP_TASKS)

    This will return rows if the condition exists – and identify which projects – as before clean-up you will probably want to get them removed from the PM’s local cache as otherwise they could be re-introduced.

    The next scripts do the cleaning up in the DB, and they are simply deleting baseline records where the tasks are non-existent.

    -- Script to run on the draft DB
    USE ProjectServer_Draft -- specify the appropriate draft database

    delete from MSP_TASK_BASELINES where TASK_UID not in (select TASK_UID from MSP_TASKS)

    -- Script to run on the published DB
    USE ProjectServer_Published -- specify the appropriate published database

    delete from MSP_TASK_BASELINES where TASK_UID not in (select TASK_UID from MSP_TASKS)

    I hope this helps to understand the nature of the issue and ways to avoid it until the fix comes along.  Our apologies for the inconvenience I know this has caused many of our customers – and hopefully for those who have needed to re-run the clean-up scripts regularly this may give a way to reduce the pain.

    If you need any assistance with these steps then feel free to open a support incident – and when I say free I mean free – this is a bug and we do not charge for incidents that are due to bugs (or we will refund – which amounts to the same thing).

    The ULS log entry associated with the initial Queue errors above (for the benefit of the search engines):

    05/01/2012 11:57:55.67    Microsoft.Office.Project.Server (0x1D74)    0x335C    Project Server    Reporting    atwj    Critical    Standard Information:PSI Entry Point:   Project User: REDMOND\brismith  Correlation Id: e1f4e953-7dea-448a-a528-709075c698bf  PWA Site URL: http://brismith8100/PWA  SSP Name: Project Server Service Application  PSError: ReportingProjectChangeMessageFailed (24006) RDS: The request to synchronize change(s) to project Project UID='216733b0-e194-469a-afc3-9235da4ce4c1'. PublishType='ProjectPublish' failed.  Message: 'ReportingProjectChangeMessageFailed'. Message Body: The INSERT statement conflicted with the FOREIGN KEY constraint "FK_MSP_EpmTaskBaseline_ProjectUID_TaskUID". The conflict occurred in database "ProjectServer_Reporting", table "dbo.MSP_EpmTask".  The statement has been terminated. Error:(null)    e1f4e953-7dea-448a-a528-709075c698bf

    and for the Timesheet error:

    05/01/2012 12:13:29.65    w3wp.exe (0x2444)    0x23D8    Project Server    Task Statusing and Updates    btw9    High    CacheProjectBaseCalendars: could not locate data for calendar 0c13de33-2a07-4310-b091-c77990d9dd6a for project 216733b0-e194-469a-afc3-9235da4ce4c1    e5dd4eaf-551a-469b-a3e0-1f60e2f3d1af

    05/01/2012 12:13:29.85    w3wp.exe (0x2444)    0x23D8    Project Server    General    0000    Exception    Exception occurred in method Microsoft.Office.Project.Server.BusinessLayer.Statusing.StatusingGetMyWorkForGridJson System.InvalidOperationException: CacheProjectBaseCalendars could not find project calendar for project. CalUid=0c13de33-2a07-4310-b091-c77990d9dd6a     at Microsoft.Office.Project.Server.BusinessLayer.TimePhasedDataAccess.CacheProjectBaseCalendars()     at Microsoft.Office.Project.Server.BusinessLayer.TimePhasedDataAccess..ctor(StatusingPageLoadDataSet dataset)     at Microsoft.Office.Project.Server.BusinessLayer.Statusing.ReadStatusTimephasedDataForResource(IList`1 gridChanges, Guid[] vAssnUids, IDictionary`2 assn2proj, StatusingTimephasedPeriod[] tpdPeriods, DateTime tpStart, DateTime tpEnd)     at Microsoft.Office.Project.Server.BusinessLayer.Statusing.<>c__DisplayClass57.<CreateTimephasedDataColumnFiller>b__56(IEnumerable`1 Keys)     at Microsoft.SharePoint.JSGrid.GridSerializer.BuildOutput()     at Microsoft.SharePoint.JSGrid.GridSerializer.ToJson(Serializer s)     at Microsoft.SharePoint.JsonUtilities.Serializer.SerializeToJson(Object o)     at Microsoft.Office.Project.Server.BusinessLayer.Statusing.GetMyWorkForGridJson(JsGridSerializerArguments gridSerializerArgs, String gridChangesJson, String projectAssignmentsMap, Guid viewUid, String timephasedStart, String timephasedEnd, Byte pane, Int32 durationType, Int32 workType, Int32 dateFormat, Boolean clearPersistedProperties, Nullable`1 rowFilterType)     at Microsoft.Office.Project.Server.Wcf.Implementation.PWAImpl.StatusingGetMyWorkForGridJson(JsGridSerializerArguments gridSerializerArgs, String gridChangesJson, String projectAssignmentsMap, Guid viewUid, String timephasedStart, String timephasedEnd, Byte pane, Int32 durationType, Int32 workType, Int32 dateFormat, Boolean clearPersistedProperties, Nullable`1 rowFilterType)    e5dd4eaf-551a-469b-a3e0-1f60e2f3d1af

  • Brian Smith's Microsoft Project Support Blog

    Project Server 2007: Microsoft Office December 2008 Cumulative Update and KBs Released!

    • 50 Comments

    ***Update*** If you are running MOSS with Project you should NOT load the December CU for MOSS as it has been withdrawn because of a search issue that is resolved in the February CU.  Thanks Shazeb for bringing this to my attention.  For full details of the reasons see http://blogs.msdn.com/joerg_sinemus/archive/2009/03/26/moss-december-cu-960011-is-not-available-anymore.aspx and for details fot the Feb CU see http://blogs.msdn.com/brismith/archive/2009/03/13/project-server-2007-cumulative-update-for-february-2009.aspx 

    ***Update*** Panic over - I'd misinterpreted an issue we had with a more recent internal build.  Everything public is OK and always has been.  Just with the weather conditions around Redmond I was having difficuty getting comfirmation.  Betterr safe than sorry - hence the warning.

    ***Update***  We are just investigating a potential anomoly with the 960011 and 960010 patches.  For now I suggest just using the individual ones, 959644, 959637 and 960313.  I will update again once I know more. 

    Starting with the December 2008 Cumulative Update for Office Server 2007 (KB960011), the Project Server updates can be obtained packaged together with the Office Server ones – as well as being available individually.  Also included in the Office Server CU are InfoPath Forms Server, and of course the core Microsoft Office SharePoint Server fixes including Excel Services.  This encompasses both the global and local patches.  Windows SharePoint Server Cumulative Update for December 2008 (KB960010) is still kept separate, but does include both local and global patches.

    Both these Office and WSS CUs now include their respective Infrastructure Updates – so no need to install separately.  To be up to date all that is needed is the latest service pack and the latest CUs for WSS and Office Server.  The WSS CU can be requested here, and the Office Server one (including Project Server) here.  The client hotfix for Project Professional 2007 can be requested here

    These are the KB articles detailing the hotfixes:

    Windows SharePoint Server 3.0 - KB959644

    Office SharePoint Server 2007 – KB959637

    Project Server 2007 – KB960313

    Project Professional 2007 – KB959643

    and by following the Project Server link you can request the individual Project Server package – rather than getting is as part of the Office Server one (but why would you?).  For guidance on deployment of cumulative updates we have this article - http://technet.microsoft.com/en-us/library/dd239177.aspx 

    And finally - just in time for the holidays the Project Professional 2007 CU does contain the fix for the “check-in pending” issue!

     *** Update *** See also http://blogs.msdn.com/brismith/archive/2009/02/11/project-server-2007-check-in-still-pending-even-after-december-cu.aspx

    Project 2007 still displays a plan as checked out after the Program Manager checks in the plan. Additionally, the plan is listed as checked in on the server.

    I know a lot of you have been waiting for this one.  Thanks for your patience.

  • Brian Smith's Microsoft Project Support Blog

    Patience please - async processes at work

    • 23 Comments

    One thing that I have touched on before is the asynchronous behavior of the some of the processes in Project Server and Windows SharePoint Services 3.0.  This is still catching some people out.  For instance, if you provision a new PWA site and then decide you want to change something - so you delete it - it looks like it has gone right away.  However, the site collection that was created as part of the process (usually /PWA) takes a while to be removed.  This means you may see an error if you try to re-create the PWA site immediately.  Another point of confusion is the "Waiting to be processed" message in the queue service of PWA.  This can sit there and look like it is doing nothing, when it is actually waiting for some other background task to complete.  Canceling and retry may make no difference - it will go when it is ready.  Of course there are times when something has gone wrong and it will never process - such as the queue service not running.  Perhaps a password has changed - or a connectivity or permission issue is stopping the account running the queue service from being able to get to the jobs (the queue jobs are held in the Project Server database tables).  Normally a look in the ULS logs or event logs will give clues to both these types of failures and also sometimes the reasons why a job may just be sleeping.

     For some internal details of the queue service take a look at the TechNet article at http://technet2.microsoft.com/Office/en-us/library/0845d622-95ab-4c20-b419-0dbd5aab33a51033.mspx?mfr=true

    The most common time for patience if if you have saved and closed a large project to the server and you are working on a low bandwidth or high latency connection.  This scenario was virtually impossible with 2003 but with 2007 it can be done.  However the save and check-in of the project may still take a while from the point you click the button on the client - and you may well find that the project is still checked out if you try to open it up too soon.  And remember - force check-in will not speed anything up - this just puts another job on the queue!

    Technorati Tags:

     

  • Brian Smith's Microsoft Project Support Blog

    Backup and Restore - what are the options?

    • 48 Comments

    A while back I posted about the backup and restore process and recommended using the SharePoint utility in Central Administration.  That is still my preference and I recently used this to rebuild my server farm including 3 servers, 2 shared service providers and a dozen PWA instances covering about 10 different languages.  I must admit it did take several attempts and my one big take-away is - try, try, try again until it all works in one go.  The process isn't very forgiving if one piece fails, so best to remedy the problem and run the whole restore again.  The problems I ran into were services that needed starting (Office search in my case, and the Project application) or databases I'd forgotten to delete.  Top tip number two is that if you have a complex restore to do and you are changing machine names, URLs, paths to db files, service accounts etc. it can be very tedious to edit these in the UI each time - particularly for all 12 PWA sites!  So a short cut is first to take a backup of the spbackup.xml file, and then edit the original and change the various things that need changing in the XML file.  It should be easy to work out what needs editing - especially if you have already been through the restore process a couple of times.

    So this restore gets everything back how it was, on either the same server or a different configuration. 

    What if you only have database backups?  What can you do and what do you lose?  This is a reasonably common scenario - particularly for people with a 2003 background who are used to working with just the WSS content db and the ProjectServer db.  But things have changed - and the main challenge is that the PWA site is now held within the content database.  This causes some issues as if you want to move the databases to another server you will need to re-provision this PWA site - and you can't do this against the same content db if it already exists.  And if you delete it then everything under /PWA is gone!  Which may well include any workspaces for your projects as this is the default location for them.  So you have a few options, and we will assume here that you are restoring and re-attaching your main content database to the default web site, and restoring your 4 project server databases.

    1. Re-provision your /PWA site to a different location - such as /PWA2, and all your old content will still be under /PWA and you will still be able to get to it.  This is a good way to work going forward as you now have two independent locations for your PWA site and your other WSS content.
    2. Don't attach the original content db and provision your PWA site to /PWA, then attach your content db to another web application (different port).  You can then re-link your projects to point to the different location for the content.
    3. An extension of option 2 - you could use stsadm export and import functions to copy the individual workspaces from their new location back under /PWA - and all is back how it was - well almost.

    So what do you still lose by going this route?  Fidelity.  If you have customized your PWA site through site settings you will lose these changes using any method other than the SharePoint backup/restore.  As an example the following site was replicated using option 1 and the highlighted changes (and the colour from the theme) are lost in the restored site pictured below.  You do however keep any changes made through Server Settings such as menu edits (My Timecard in place of My Timesheets in the screen shots).

    Original PWA Site

    Restored PWA Site

    I hope this helps understand what is stored where - and the challenges of moving data between servers, either for migration or disaster recovery.

    Technorati Tags: Project Server 2007

  • Brian Smith's Microsoft Project Support Blog

    Project Server 2007: WSS Sync issues – Error: Access Denied

    • 17 Comments

    After the notorious check-in pending, and the slow cube building issues, the next hot topic has to be problems around WSS Synchronization.  This was noted in one of the recent TechNet updates (I’ve also just corrected the link in my previous posting – sorry) for Active Directory synchronization, but AD Sync is just one of the scenarios leading to this issue.  In case you are not familiar with this one, basically there are two parts to the story:

    • Sync of permissions for sites (including /PWA) can be really slow if you have lots of users
    • If more than one sync is in progress then SQL deadlocks are likely to occur

    For our larger customers the first item is bad enough, but generally if the sync is slow there is a greater risk of another sync clashing.  These synchronizations can be a result of changes in group security settings, syncs to AD or just re-sync’ing the site permissions for the workspaces.  They are also more likely if your category settings mean that virtually all users can access all projects – more permissions to sync!

    The typical symptoms for users is that they can be left out in the cold with an Error: Access Denied – with the option to sign in as a different user.  Behind the scenes, in the queue and the ULS logs you will see such things as:

    Manage Queue page

    "User Synchronization for Project Web Access App Root Site and Project WSS Workspace" fail with the Job State error of "Failed But Not Blocking Correlation"

    ULS Logs

    10/06/2008 17:11:37.16  Microsoft.Office.Project.Server (0x07D8) 0x0CF0 Windows SharePoint Services    Database                       6f8g Unexpected Unexpected query execution failure, error code 1205. Additional error information from SQL Server is included below. "Transaction (Process ID 98) was deadlocked on lock resources with another process and has been chosen as the deadlock victim. Rerun the transaction." Query text (if available): "{?=call proc_SecAddPrincipalToRole(?,?,?,?,?,?)}"

    The updated part of the TechNet article is worth repeating here, as it gives some suggestions for reducing the occurrence of the problem, and links to some tools which you may find useful, so here it is:

    Caution Caution:

    Under certain circumstances, synchronizing Project Server users and workspaces with Active Directory can cause a “deadlock” situation in which all users are locked out of a PWA site or the respective workspaces. This causes user synchronization jobs to fail and site permissions to synchronize partially or not at all. Users may not be able to log on to PWA or their workspaces.

    A deadlock can occur if the user synchronization process is taking too long to complete. This is due to the synchronization job iterating through many users and workspaces, for example, when large membership changes are being made. A synchronization job remaining in the queue a long time increases the possibility of other jobs starting inadvertently, which can also cause a deadlock.

    To reduce the chance of a deadlock, you can do the following:

    • Before making large group membership changes, verify that there are no jobs named “User Synchronization for Project Web Access App Root Site and Project WSS Workspaces” currently processing or waiting to be processed in the queue.

    • Run the Project Server Workspace Sync tool on the CodePlex site (http://go.microsoft.com/fwlink/?LinkId=147394). The tool controls what is to be synchronized when the job starts — PWA and workspaces, workspaces only, PWA only, or no synchronization for either PWA or workspaces — and allows the administrator to perform the user synchronization during non-working or off-peak hours when server overhead is lower.

      Note that the Project Server Workspace Sync tool does not speed up the synchronization process beyond normal. However, being able to synchronize users when server overhead is lower reduces the possibility of synchronization failures.

    Unfortunately there isn’t a fix coming for this one any time soon – it is harder to do than you might think, with so many moving parts.  I hope this at least helps you to work around the issues and sorry for the undoubted inconvenience this one has been causing.

     

    Technorati Tags: ,

Page 5 of 91 (454 items) «34567»