June, 2010

  • System Center Platform Team

    Service Manager 2010 SDK Documentation Released!

    • 1 Comments

    The Service Manager 2010 Software Development Kit (SDK) documentation is now released on MSDN.  There are some good code samples and most importantly all the reference documentation for the API layer.

    MSDN also now supports community contributions.  Please feel free to add your own comments, examples, code samples, etc!

    http://msdn.microsoft.com/en-us/library/gg469708.aspx

  • System Center Platform Team

    Troubleshooting the Data Warehouse: An overview

    • 0 Comments

    I know that troubleshooting the data warehouse can be a bit intimidating. Sometimes even figuring out where to begin to look can be difficult. In this series of blog posts, I am going to help you figure out how to troubleshoot the data warehouse. I’ll use this post as an index with links to all of the subsequent topics.

     

    Where to begin?

     

    Depending on what kinds of issues you see, your start point might be different. Here I am going to talk about the general approach. 

    If you observe a failure or something not too right, generally, there are a few places we can start our debugging.

     

    1.      Start with the Operations Manager event log on the Data Warehouse server

     

    This is the absolutely most important place to begin! Almost all of the errors from the Data Warehouse are output to this event log. Events in this log will have two different sources: Deployment and Data Warehouse.

    Events with a source of Deployment are usually from management pack deployment which includes report deployment or building out the data warehouse (like creating outriggers, dimensions, fact tables, etc).  If you see an error in the event log, it usually has instructions on how to recover from it. For example, it may indicate you need to restart the services. In the Data Warehouse Management Server, there are three services just like on the Service Manager Management Group server:

    ·         System Center Data Access Service,

    ·         System Center Management,

    ·         System Center management Configuration

    It’s usually best to recycle all of them at the same time.

    Once your data warehouse is deployed, events are more likely to have a source of Data Warehouse. These events are written by jobs within the normal course of operations like the ETL jobs, the MPSync job and the DWMaintenance job. However, these all still write their events to the Operations Manager event log.

     

    2. Data Warehouse Workspace in the Service Manager console

     

    When you click on the Data Warehouse Jobs in the Data Warehouse workspace, you should see the ETL jobs and MPSync Job status. If your deployment was successful and your data warehouse is correctly registered to at least one Service Manager management group, you’ll see at least 5 jobs. Every job should either be in a status of “Running” or “Not Started”. 

     

    If you see a job is in Failed status, you can click on this job and then click on Modules from the Tasks pane to find out which job module has failed. However to find out why it failed, the best place to get information from is the Operations Manager event log on the Data Warehouse server. You can also get more details from Powershell cmdlets which I will talk about in the next step. 

     

    In the Data Warehouse workspace, you can also click on the Management Packs link in the left pane. Here’s where you can see all the management packs in the Data Warehouse and the status of their deployment. When you import a management pack to Service Manager, the MPSync job will synchronize it to the Data Warehouse (hence the name…MPSync means “management pack synchronization”). When you get the list of management packs in the data warehouse, you can find out if your management pack has been deployed successfully or not. 

    If your management pack has defined data warehouse specific elements such as outriggers, dimensions, fact tables or reports, this Management Pack must be successfully deployed before the new tables and reports will be ready to use.

     

    3. Powershell

     

    The Powershell cmdlets provide much more detailed information about the Data Warehouse jobs than the console does.  This will likely change in a future release, but for now I really recommend you learn how to use the following useful cmdlets:

    ·         Get-SCDWMgmtGroup

    This command tells us the sources which are registered with DW currently. We expect to see at least two different DataSourceName.

     

    ·         Get-SCDWJob

    This command tells us all DW jobs status in the current batch.

    In the above screenshot, you can check whether the jobs are enabled or not (see the highlight), which jobs are running and when they started. 

    When MPSync Job or DWMaintenance job start, they both disable all of the ETL jobs so you will see the IsEnabled column set to false for each of the ETL jobs. This means that even if the ETL job status shows it is running, it actually isn’t. When MPSync Job or DWMaintenance job complete the ETL jobs will automatically get enabled and resume processing.

    Jobs are normally in the “Not Started” status and if it is you can assume the previous batch has completed. If you’d like, you can use the following command to view the latest few batches of a specific job.

    Get-SCDWJob –JobName <Specific job name>  -NumberOfBatches <number>

    In the above screenshot, you can see the latest MPSyncJob completed, when it started and when it ended. If you’d like you can calculate how long it ran, and what the next batch id and status is.  The job batch id is always incremental.

    ·         Get-SCDWJobModule

    This command provides detailed information about the specific modules within the job.  This is very useful when you see job failed and want to find out what caused the failure.

    List of common issues and how to troubleshoot them

    This list is not exhaustive, but it does cover most of the common issues customers have run into. This list will get added to over time so feel free to check back if you run into a new issue:

    ·         Reports aren’t deployed after registering the data warehouse

    ·         Job(s) fail after importing a custom management pack

    ·         Data warehouse isn’t getting new data or jobs seem to run forever

    ·         Custom data warehouse extensions don’t appear in the data warehouse

    ·         Management packs are stuck in “pending association” after registering the data warehouse

    ·         ETL jobs fail due to login credentials issues

     

  • System Center Platform Team

    Troubleshooting the Data Warehouse: Custom data warehouse extensions don’t appear in the data warehouse

    • 0 Comments

    This is the latest in “Troubleshooting the Data Warehouse” series. The goal of this series is to not only help you troubleshoot specific issues you may be experiencing but also teach you more about the data warehouse. For a complete list of posts in the series, check out Troubleshooting the Data Warehouse: An overview.

     

    The symptom

     

    After importing your management pack which defines some dimensions and/or fact tables to Service Manager, MPSync job has run several times but you still don’t see your dimension and/or fact tables in the DataMart.

     

    The troubleshooting steps

     

    To troubleshoot this issue, investigate the following things in order.

    1. Ensure your management pack is sealed

     

    MPSync Job can only bring sealed management pack from Service Manager to Data Warehouse. If you haven’t sealed your management pack, the simple solution is to seal it and then import it to Service Manager.

    2. Ensure your management pack has synced to Data Warehouse

     

    ·         Open the Console

    ·         Click on the Data Warehouse wunderbar

    ·         Click on Management Packs then find your management pack in the list of management packs. To do this, use the search feature (type your management pack name in the search box).

    If you don’t see your management pack,

    ·         It might have failed to import to Data Warehouse Management Server. Go to the Data Warehouse Management Server, open the Operations Manager event log and then filter the events with Event Sources as OpsMgr SDK Service.

    ·         The MPSync job may not have run yet. It runs on a recurring schedule, out of the box it runs every hour. You can modify the schedule with Powershell. In order to speed up the management pack synchronization, after import your management pack, you can manually resume MPSync job either from UI or powershell.

     

    3. Check the deployment status of your management pack

     

    If the management pack is sealed and has synced to Data Warehouse

    ·         Open the Console

    ·         Click on the Data Warehouse wunderbar

    ·         Click on Management Packs then find your management pack in the list of management packs. To do this, use the search feature (type your management pack name in the search box).

    ·         Check the deployment status (last column) of your management pack.

     

    4. If the deployment status is Failed

     

    Go to the Data Warehouse Management Server, open the Operations Manager event log and then filter the events with Event Sources as Deployment.

    If there is error message, usually the error message will indicate what went wrong. If after correct the management pack, the error still occurs. In order not to affect the overall system, you can uninstall this management pack from Service manager Console, Administration > Management Packs. After MPSync job run, the management pack will be uninstalled from Data Warehouse Management Server.

     

  • System Center Platform Team

    Troubleshooting the Data Warehouse: Data Warehouse isn’t getting new data or jobs seem to run forever

    • 1 Comments

    This is the latest in “Troubleshooting the Data Warehouse” series. The goal of this series is to not only help you troubleshoot specific issues you may be experiencing but also teach you more about the data warehouse. For a complete list of posts in the series, check out Troubleshooting the Data Warehouse: An overview.

     

    The symptom

     

    I don’t see data or new data in my reports or ETL Jobs look like running forever and never in Not Started status.

     

    The troubleshooting steps

     

    To troubleshoot the above issue, please verify things in the following orders

    1. Use the Get-SCDWJob PowerShell cmdlet to determine if all ETL jobs are enabled

    Start PowerShell and type Get-SCDWJob

    ·         If the ETL jobs are disabled and either MPSyncJob or DWMaintenance job is running, you’ll need to wait a little bit to get the job status again (because these two background jobs disable the ETL jobs).  However, if these two jobs are Not Started and the ETL jobs are disabled, you can use the Enable-SCDWJob cmdlet to enable each of them. For example

    Enable-SCDWJob -JobName Transform.Common

     

    ·         If ETL jobs, MPSync job and DWMaintenance job are all enabled and running, but their individual batch id has not changed for a long time or use Get-SCDWJobModule for specific job, we don’t see any module actually is running, check the event log and see if there is any error message. Sometimes, the error message might be many days old, we need to dig deep in the old day events.

     

    ·         Check if three services (System Center Data Access Service, System Center Management, System Center management Configuration) in Data warehouse Management server are actually running.

    In Data Warehouse Management Server, click Start > Run .., enter Services.msc, in the Services window, verify if the following services are running or not

    o   System Center Data Access Service

    o   System Center Management

    o   System Center management Configuration

    If any of them are not running, restart all three services. In addition, if all services are actually running, events from Event Source Data Warehouse and OpsMgr SDK Service are outputted to Operations Manager event log. We can use this information as another source to verify if all services are running or not. If you don’t see events from Event Source Data Warehouse and OpsMgr SDK Service for a while, restart all three services regardless.

     

  • System Center Platform Team

    Troubleshooting the Data Warehouse: ETL failures due to login issues

    • 0 Comments

    This is the latest in “Troubleshooting the Data Warehouse” series. The goal of this series is to not only help you troubleshoot specific issues you may be experiencing but also teach you more about the data warehouse. For a complete list of posts in the series, check out Troubleshooting the Data Warehouse: An overview.

     

    The symptom

     

    Some or all ETL jobs have failed. The Operations Manager event log in the Data Warehouse Management Server indicates the ETL job failure is related to a login user failure.

     

    The troubleshooting steps

     

    To troubleshoot this issue, investigate the following things in order.

    1. Check if the password for each run as accounts has changed or expired.

     

    If it is, you need to update it. To do this:

    ·         Click on Data Warehouse > Security > Run As Account, click on the related run as account, then click the Properties from the Tasks pane, update the Password field in the window and then click OK

    ·         If this run as account is Operational System Account, you also need to update the services that are running under this account.

    o   In Data Warehouse Management Server, click Start > Run .., type Services.msc

    o   In the Services window, update the passwords for the services that run under this account, for example,  System Center Data Access Service and System Center management Configuration

    o   Restart the services

     

    Note that MP Sync job and Extract job for Service Manager can use a different run as account other than Operational System Account. This run as account is created while Service Manager is registered to Data Warehouse. See the following Data Warehouse Registration Wizard, the New … button

     

    It is easy to update the password if it is expired. However it is complicated to update the system if you change the run as account. We don’t recommend that you change run as accounts.

     

    2. If the job failure is not related to password

     

    You should make sure the run as account for the failed job can be used to connect to the target database. For example, ensure the Extract job run as account can be used to connect to the Service Manager database. If not, make sure the SQL service hosting the database is running.

     

     

     

     

  • System Center Platform Team

    Troubleshooting the Data Warehouse: Management packs are stuck in “pending association”

    • 3 Comments

     

    This is the latest in “Troubleshooting the Data Warehouse” series. The goal of this series is to not only help you troubleshoot specific issues you may be experiencing but also teach you more about the data warehouse. For a complete list of posts in the series, check out Troubleshooting the Data Warehouse: An overview.

     

     

    The symptom

     

    After registering Service Manager to the Data Warehouse and several (4+) hours have passed some management packs are still stuck in “pending association” state. You can check this by opening the Console, clicking Data Warehouse > Data Warehouse Jobs > MPSync Job, and then click Details from the Tasks Pane

     

    The troubleshooting steps

     

    To troubleshoot this issue, investigate the following things in order.

    1. Review each batch ID for the “stuck” MP via the MPSyncJob dialog box

    In the MP Sync Job dialog, click on the Management Pack column name to sort the list according to Management Pack name.  Find those management packs with Pending Association status, check if in the later batch, the Pending Association management pack turns to Associated.

    For example in the following figure I’ve highlighted:

    ·         In batch 136, Management Pack Microsoft.SystemCenter.ConfigurationManager is in Pending Association

    ·         In batch 207, Management Pack Microsoft.SystemCenter.ConfigurationManager is in Associated

    This means the management pack is associated properly in batch 207 even though it ran into an error in batch 136. Because it recovered in batch 207, the management pack is correctly associated and the sync completed successfully. So even if a management packs is Pending Association in one batch, if in a later batch it is successfully Associated, then the management pack is correctly and completely deployed and synchronized.

    If in the MP Sync Job dialog, the Pending Association status for a management pack repeats for every batch, you’ll need to dig a bit deeper to figure out what causes this management pack to fail to associate.

     

    2. Look for deployment failures in management packs which this management pack depends on

    Click on the Data Warehouse > Management Packs and then click the Deployment Status column header in the main pane.  If you see any management pack whose deployment status is Failed and some are Not Started this is usually due to a management pack dependency. Because management packs can depend on others, any failure can have a domino effect and cause other management packs not able to deploy. Any impacted management pack will have a status of Not Started (it’s not Failed, but because another did Fail it hasn’t yet started).

     

    3. Find the deployment failures in the event log

     Go to the Data Warehouse Management Server, open the Operations Manager Event log, filter the event log to the events whose Event Source is Deployment and Event Level is Warning or Error. 

    If there is error message similar to the below highlighted text you’ll need to unregister the Data Warehouse from Service Manager, reinstall the Data Warehouse and then re-register Service Manager to Data warehouse.

                  Deployment Execution Infrastructure has retried the maximum number of times and is giving up on this execution step. 

           MP Element ID: DerivedManagementPack.SystemDerivedMp.ServiceManager.ActivityManagement.Library.Datawarehouse

           MP name: ServiceManager.ActivityManagement.Library.Datawarehouse

           MP version: 7.0.5826.0

           Operation: Install

                    Error message: Cannot find resource with ID TransformActivityStatusResource,

     

  • System Center Platform Team

    Troubleshooting the Data Warehouse: Job(s) fail after importing a custom management pack

    • 1 Comments

    This is the latest in “Troubleshooting the Data Warehouse” series. The goal of this series is to not only help you troubleshoot specific issues you may be experiencing but also teach you more about the data warehouse. For a complete list of posts in the series, check out Troubleshooting the Data Warehouse: An overview.

     

    The symptom

     

    One or more data warehouse jobs start failing after importing a custom management pack to Service Manager and synchronizing it to the Data Warehouse

    The troubleshooting steps

     

    To troubleshoot the above issue, please verify things in the following orders

    1. Check the event log to ensure the root cause is the custom management pack

    ·         In Data Warehouse Management Server, open Operations Manager event log.

    ·         Find the event which is related to this job failure.

    ·         Determine the failure is related to the custom management pack you just imported.

    2. Remove the custom management pack and recover from the failure

    If the failure is related to the custom management pack, you should remove it and let the rest of the data warehouse operate as usual. You can fix the MP and reimport it later.

    ·         Uninstall the custom management pack via the Service Manager console

    ·         Run MP Sync job

    ·         Verify that this custom management pack is not in the Data Warehouse > Management Packs

    ·         After MP Sync job is completed, resume the failed job either from UI or cmdlet

    ·         You should be good to go

    3. Fix and reimport the custom management pack

    ·         First you must remove the custom management pack and recover from the failure (step 2 above)

    ·         Fix the custom management pack

    ·         Import the custom management pack with fix into Service Manager, run MP Sync job to sync it to Data Warehouse

     

     

  • System Center Platform Team

    Troubleshooting the Data Warehouse: Reports aren’t deployed after registering the data warehouse

    • 3 Comments

    This is the latest in “Troubleshooting the Data Warehouse” series. The goal of this series is to not only help you troubleshoot specific issues you may be experiencing but also teach you more about the data warehouse. For a complete list of posts in the series, check out Troubleshooting the Data Warehouse: An overview.

     

    The symptom

     

    When you open the console, a dialog box appears indicating that the Reporting Service is unavailable. Or if the Report wunderbar appears, there are no reports in the Reporting workspace and if you look on the Reporting Services server, no reports have been deployed.

    All other aspects of the data warehouse deployment seem to have gone smoothly. For example, when you click on the Data Warehouse wunderbar then Data Warehouse Jobs, you see 2 extract jobs, a transform job, a load job and an MPSync job.

     

    The troubleshooting steps

     

    To troubleshoot the above issue, please verify things in the following orders

     

    1. Check the deployment status of your management packs

    In the Service Manager Console

    ·         Click on the Data Warehouse wunderbar

    ·         Click on Management Packs and in the search Filter field type report. This will filter to the report related management packs

    ·         Check the deployment status (last column) of the following MPs…none should have Failed

    o   ServiceManager.ActivityManagement.Report.Library

    o   ServiceManager.ProblemManagement.Report.Library

    o   ServiceManager.IncidentManagement.Report.Library

    o   ServiceManager.ConfigurationManagement.Report.Library

    o   ServiceManager.ChangeManagement.Report.Library

     

    2. Check the event log for error messages about the assembly Microsoft.EnterpriseManagement.Reporting.Code.dll

    If any of the above 5 management packs failed deployment

    ·         Go to the Data Warehouse Management Server

    ·         Open the Operations Manager Event log

    ·         Filter the events with Event Sources as Deployment and Event Level as Error

    ·         If there are error messages in the event log which indicate “cannot load Assembly Microsoft.EnterpriseManagement.Reporting.Code.dll”

    o   Your SSRS installation may be on a different server than the Data Warehouse Management Server. If so, please follow the deployment guide and copy this assembly to the SSRS server then add access permission to it.

    For detail instruction please follow the deployment guide.

    o   If your SSRS installation is on the same server as the Data Warehouse Management Server, restart the SSRS service.

    ·         To restart SSRS

    o   Log on to the server where SSRS is installed

    o   Click Start > All Programs > Microsoft SQL Server 2008 > Configuration Tools > Reporting Services Configuration Manager

    o   In the pop up window, click Connect, then click Stop, after that click Start.

     

    3. Redeploy the failed report management packs

     

    In the Service Manager Console:

    ·         Click on the Data Warehouse wunderbar

    ·         Click on  Management Packs and in the search filter type report

    ·         For each of the MPs listed in step 1, click Restart Deployment in the right hand Task Pane. Note that if the deployment status of a management pack is completed, click on this management pack, the Restart Deployment is grey out in the Task Pane

    Once the deployment status of the report mps turned from Failed to Completed, open the Console again, reports should display in the Reporting workspace. You may need to restart the console to see the reports as the console caches the list of reports.

Page 1 of 1 (8 items)