Cascade Skyline - with Microsoft Logo and Project Support header - author Brian Smith

  • Brian Smith's Microsoft Project Support Blog

    Slow OLAP Cube Builds and Large TempDB - Revisited

    • 12 Comments

    After quite a bit of work on this one it appears that SP1 (and a slightly earlier hotfix) were almost coincidental in this problem rather than being the cause - and it can occur even without SP1 if you add enough dimensions to your cubes.  The best resolution is a combination of the first 2 or all 3 of the following items:

    1.  Keep you statistics up to date!  In minor cases updating the statistics on your reporting database can help - but in all cases it is required to get the full benefit.  You should set up a maintenance plan or initial for testing you could use sp_updatestats on your reporting database.  See Christophe's recent blog for some guidance here - database maintenance is one of the keys to better queue performance.

    2.  if you see a "No Join Predicate" warning in the execution plan of the large query causing the problem then apply SQL Server 2005 SP2 Cumulative Update 6 http://support.microsoft.com/kb/946608 .  The SQL bug is resolved by Hotfix 942444 which is in that roll up.  Please also read http://support.microsoft.com/kb/942444 carefully, particularly the details of setting Trace Flags 4101 and 4121 - which are required to be set to active this hotfix.  See the remarks section at http://msdn2.microsoft.com/en-us/library/ms188396.aspx for more details of trace flag setting.  Use the command line option -T4101 and -T4121, and set these in the SQL Server Configuration Manager for the instance of SQL Server 2005 used for your reporting database.

    *** Update - SQL Server 2008 R2 - CU7 + setting the flag for -T4136 will get you back to good build times if you have upgraded and gone slow again - thanks to Flo ***

    ***Update 2 - I think this should be -T4199 for 2008 R2 to do the exact same - -T4136 is for parameter sniffing - perhaps that results in the good execution plan too ***

    3.  If this still isn't giving you the speed of cube build you require then it might be time to consider upgrading your server hardware.  In our tests with very large cube builds you can improve the build times with more RAM - and also x64 hardware (which also allows you to support even more RAM).

    If you are not sure about step 2 - then see my previous posting which will help you identify the bad query - and then paste this into a SQL Query window against you reporting database and click to see the estimated query plan.  In the execution plan - which will be quite large - you are looking for something like this:

    image

    And if you hover over the warning triangle you should see a warning of "No Join Predicate".  Also note the expense (74% of the cost of the query) attributed to the Table Spool.  This is where tempdb is getting heavily used.

    As an example of potential improvements after following these steps (without any hardware upgrades) we have seen cube builds of 20+ hours complete in 45 minutes, and a 1 hour 15 minute build come down to less than 1 minute.  Your mileage may vary :).

    You may still see quite heavy tempdb usage and this is to be expected with such a large query - and should not be a concern.  If Your server doesn't have the space for the tempdb SQL is trying to use after following these steps then time to invest in more disk space!

  • Brian Smith's Microsoft Project Support Blog

    Lies and out-dated statistics

    • 0 Comments

    The saying goes that you can prove anything with statistics - and I am sure the same figures are being used by the McCain, Obama and Clinton camps to tell vastly different stories.  But enough about politics - what has this to do with Project Server 2007?  SQL Server uses statistics too - and if they are not up to date it might make the wrong decisions!

    As an example if you were trying to find me in a database and there were indexes on my department, surname and date of birth, and you knew may name, birth month (but not the actual day or year) and my department what is the best way to find me?  With a name like Smith, you probably wouldn't want to start there.  So perhaps you would find all the people in Project Support, then look for any with surname Smith, then trim down to the ones born in the right month and finally see if any (or hopefully the only one) is called Brian.  If you were looking for Chris Boyd, or Christophe Fiessinger then you might start your filtering on their names first - as there aren't as many Boyds or Fiessingers as there are Smiths. What you are doing is building an execution plan, and usually the most efficient way is to trim down the number of results at the earliest point in the plan.  SQL Server does this by keeping statistical data about the contents of data tables and indexes, and builds plans according to this data.  If the data is not accurate it can lead to execution plans that are not optimal.

    Maintenance plans in SQL Server give you the ability to schedule jobs to reorganize your indexes to ensure they are efficiently stored both for reading and writing - and also update your statistics to ensure SQL will make the best possible decisions when deciding a query plan.  For SharePoint databases you should avoid using the default rebuild index option of the maintenance plan wizard as it does not re-apply existing index options - but you can script your own tasks to do this.

    There are some great resources online for maintenance of your databases - I'd suggest the recent Whitepaper titled Database Maintenance for Microsoft® SharePoint® Products and Technologies and also for general background reading the SQL Books Online, Database Engine Administrator InfoCenter is an excellent resource. 

    I will be following up in the next few weeks with blogs specifically about two areas where a good set of up to date statistics can improve the performance in Project Server 2007.

    • Building Data Analysis cubes - outdated statistics are one of the causes of recent problems seen with slow cube building and heavy tempdb usage.
    • Project Center Views - the error that can appear after around 30 seconds if you have many projects, and also large project that you drill into.
  • Brian Smith's Microsoft Project Support Blog

    OLAP and Data Analysis - timeouts, plan guides and tempdb

    • 3 Comments

    UPDATE: Be sure to see the more recent post - http://blogs.msdn.com/brismith/archive/2008/03/24/slow-olap-cube-builds-and-large-tempdb-revisited.aspx  

    Some of this has been in previous postings or comments on postings - but thought it was time for a round-up.  As a quick intro this blog will look at more information on tempdb growth some of you may have experienced since Project Server 2007 SP1, as well as timeouts that you might have hit due to the same problem.  I'll also talk more about SQL Server 2005 plan guides used to overcome this problem - and finally errors you may see after your reporting database gets rebuilt - or refreshed as we call it in the queue.

    TEMPDB Growth

    In SP1, and in a hotfix just prior to SP1 we introduced some dimensions to the cube that we had missed in the initial release.  These related to some of the flags for generic and team resources and a few other things.  For customers that were also using many other dimensions and had large datasets these changes gave longer queue build times and also increased the use of tempdb as the SQL execution plan made use of table spools which create temporary tables in the tempdb database.  In many cases the workaround described in my previous blog  gives some relief from this problem.  You may still need a reasonable amount of tempdb so don't bother shrinking tempdb below the level it would normally use - it will just slow things down as it has to grow again next time.

    In some cases the increased cube build time may lead to the timeout issue dealt with below - and also applying the workaround can be challenging.  To help understand the workaround a bit more on to plan guides...

    Plan Guides

    SQL Server Books Online have always been a great resource since the early days of SQL Server.  This is still true and the plan guide documentation can be found at Understanding Plan Guides.  Basically whenever SQL runs a query it will look for any plan guides (in the system table sys.plan_guides - so select * from sys.plan_guides will show what is set) and if it has query_text in that table that matches the query it will follow the hints in that table.  The query needs to be an exact match so any change in date or dimensions (or even order of dimensions added to a cube) may "break" the plan guide and the workaround would need to be re-applied.  As the default name of the plan (from the workaround) is already in existence either a new name needs to be used or the plan guide needs to be dropped from the database.  An example of the command to drop a plan guide called guide_forceorder would be:

    EXEC sp_control_plan_guide N'DROP', N'guide_forceorder';

    You can also enable and disable the plan guides using the sp_control_plan system stored procedure.

    To see if a plan guide is being used you can run the profiler trace and add the Showplan XML event type (located under the Performance node).  Then in the trace you should find the Showplan XML event for the affected query and it will start something like:

    <ShowPlanXML xmlns="http://schemas.microsoft.com/sqlserver/2004/07/showplan" Version="1.0" Build="9.00.3054.00"><BatchSequence><Batch><Statements><StmtSimple PlanGuideDB="pwa_Reporting" PlanGuideName="guide_forceorder"><QueryPlan CachedPlanSize="1463" CompileTime="336" CompileCPU="336" CompileMemory="9432"><RelOp NodeId="0" PhysicalOp="Compute Scalar" LogicalOp="Compute Scalar" EstimateRows="1" EstimateIO="0" EstimateCPU="1e-007" AvgRowSize="1625" EstimatedTotalSubtreeCost="778.134" Parallel="0" EstimateRebinds="0" EstimateRewinds="0"><OutputList><ColumnReference Column="Expr1007"/><ColumnReference….

    If you don't find this then perhaps the query in the plan doesn't match what Analysis Services is using to pull the data.

    Timeouts

    One of my tips for support engineers is that if something fails at very specific times then be suspicious.  So when a cube fails at just over an hour suspect that the default External Command Timeout is still set in Analysis Services.  The default is 3600 and is in seconds - so is equal to 1 hour.  Right click the AS server in Management Studio, then set Show Advanced (All) Properties and check the External Command Timeout.  Increase as appropriate - 36000 would be 10 hours.  For the SP1 issue you may find that this is the first error you hit - then once you get past this the tempdb problem may give you issues.

    The error for the timeout issue is:

    Failed to build the OLAP cubes. Error: Analysis Services session failed with the
    following error: Failed to process the Analysis Services database <cube name> on
    the <Analysis Server name> server. Error: Internal error: The operation terminated
    unsuccessfully


    Internal error: The operation terminated unsuccessfully. Internal error: The
    operation terminated unsuccessfully. OLE DB error: OLE DB or ODBC error:
    Unspecified error. Errors in the OLAP storage engine: An error occurred while
    processing the 'Assignment Timephased' partition of the 'Assignment Timephased'
    measure group for the 'Assignment Timephased' cube from the <cube name>
    database.

    The initial part of the error may be different depending on the exact point the timeout stops things.  The first part will also be localized if you are running a language pack.

    Refreshing the reporting database loses the Assignment dimensions

    The error for this problem is:

    Failed to build the OLAP cubes. Error: Invalid call to GetJoinForCustomFieldDimension. Cannot join "Task Non Timephased" fact with a custom field of entity Assignment

    And is caused when you have added dimensions to your assignment cube and then have done something that will have caused the reporting database to refresh.  Restoring from certain administrative backups will do this.  The fix is easy - just remove the dimensions from the assignment cube, save the configuration - then add them back again and save again.  To be sure this is your issue take a look at the view MSP_EpmAssignment_OlapView in the reporting database.   If the last column is AssignmentBaseline10BudgetMaterialWork then this means no dimensions are added.  If the repository is expecting some dimensions you will get the error.  A screenshot of the view with added dimensions will look something like:

    clip_image002

    I hope this helps to understand more about the cube build process and some of the problems you can run into.

  • Brian Smith's Microsoft Project Support Blog

    COMException on x64 platforms when automating the Project client via the Primary Interop Assembly (PIA)

    • 8 Comments

    ***UPDATE*** Hotfix now available - http://blogs.msdn.com/brismith/archive/2008/11/18/now-fixed-comexception-on-x64-platforms-when-automating-office-clients-via-the-primary-interop-assembly-pia.aspx  

    You may see this error when using the CodePlex Test Data Population sample for creating project data using the WinProj tab, or just using your own code to automate winproj.exe (the Microsoft Office Project Professional 2007 client application).  It is only a problem with the object model interaction and not an issue with PSI calls.  It is the Tasks.Add() method which is the trigger for the problem, and it will work just fine on x86, but fails on x64.

    Currently the x64 platforms do not support more than 1024 methods on an object (which comes down to around 1017 once the COM standard methods are deducted) and the Tasks object has a lot of methods.

    The error is:

    Error HRESULT E_FAIL has been returned from a call to a COM component. System.Collections.ListDictionaryInternal.

    One work around we have found is re-writing to avoid using the Tasks method.  So the following code:

    protected void Page_Load(object sender, EventArgs e)
    {
    string filename = "c:\\test.mpp";
    ApplicationClass a = new ApplicationClass();
    a.FileNew(Type.Missing, Type.Missing, Type.Missing, Type.Missing);
    Microsoft.Office.Interop.MSProject.Project p = a.ActiveProject;
     p.Tasks.Add("test", Type.Missing);
    p.SaveAs(filename, PjFileFormat.pjMPP, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, "MSProject.mpp.9", Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing);
    a.Quit(PjSaveType.pjDoNotSave);
    lblfilename.Text = filename;
    }

    could be re-written as:

    protected void Page_Load(object sender, EventArgs e)
    {
    MSProject.Application objAppProject;
    MSProject.Project objProject;
    string filename = "c:\\test.mpp";
    objAppProject = new Microsoft.Office.Interop.MSProject.Application();
    objAppProject.FileNew(Type.Missing, Type.Missing, Type.Missing, Type.Missing);
    objAppProject.EditGoTo(1, Type.Missing);
    objAppProject.SetTaskField("Name", "Test", true, Type.Missing, 1, Type.Missing);
    objProject = objAppProject.ActiveProject;
    objProject.SaveAs(filename, MSProject.PjFileFormat.pjMPP, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing, "MSProject.mpp.9", Type.Missing, Type.Missing, Type.Missing, Type.Missing, Type.Missing);
    objAppProject.Quit(MSProject.PjSaveType.pjDoNotSave);
    lblfilename.Text = filename;
    }

    and would work in both x86 and x64 environments.

    This is currently being worked on by the Windows teams and we are anticipating a fix.

  • Brian Smith's Microsoft Project Support Blog

    Loading projects from server or mpp files can be slow with some security products

    • 0 Comments

    One thing we have seen recently in support is several cases where security products can slow down the loading of projects into Project Professional 2007 both from mpp files or from Project Server 2007 connections.  An example of this is McAfee Host Intrusion Prevention, and we have seen this make projects load up to ten times slower.  We are working with McAfee to see if there is a workaround for this and I would certainly not suggest disabling any security products on your servers.  It might however be worth testing in a secure environment if you think your speed might be affected by something like this - if only to eliminate it as a "suspect".

    I'm hoping we may just need to exclude something and things will speed up - if anyone has experience of HIP and can give me pointers I'm happy to talk.

Page 78 of 96 (476 items) «7677787980»