Microsoft SQL Server Development Customer Advisory Team : SQL Server Best Practices

Ultimate guide to upgrading to SQL Server 2008

denny.lee — Mon, 08 Dec 2008 03:57:00 GMT

For the ultimate guide to upgrading to SQL Server 2008, please refer to the SQL Server 2008 Upgrade Technical Reference Guide.

A successful upgrade to SQL Server 2008 should be smooth and trouble-free. To achieve that smooth transition, you must devote plan sufficiently for the upgrade, and match the complexity of your database application. Otherwise, you risk costly and stressful errors and upgrade problems. Like all IT projects, planning for every contingency and then testing your plan gives you confidence that you will succeed. But if you ignore the planning process, you increase the chances of running into difficulties that can derail and delay your upgrade. This document covers the essential phases and steps involved in upgrading existing SQL Server 2000 and 2005 instances to SQL Server 2008 by using best practices. These include preparation tasks, upgrade tasks, and post-upgrade tasks.

Chapter 1 gives an overview of the technical issues and decisions that are involved in an upgrade to SQL Server 2008, as well as recommendations for planning and deploying an upgrade.
Chapter 2 addresses issues related to upgrading to SQL Server 2008 Management Tools.
Chapters 3 through 8 focus on upgrade issues for SQL Server relational databases.
Chapter 9 addresses upgrading to SQL Server 2008 Express.
Chapters 10 through 14 focus on upgrading to SQL Server 2008 Business Intelligence components: Analysis Services, Data Mining, Integration Services, and Reporting Services.
Chapter 15 addresses the implications of upgrading to SQL Server 2008 for other Microsoft applications and platforms.
Appendix 1 contains a table of allowed SQL Server 2008 version and edition upgrade paths.
Appendix 2 contains an upgrade planning checklist.

Best Practices for Integrated Full Text Search (iFTS) in SQL 2008

kevincox — Thu, 06 Nov 2008 04:43:00 GMT

This blog is a result of lessons learned while working with the new Integrated Full Text Search in SQL Server 2008. It is not intended to be an in-depth tutorial on how to implement it. Instead, it will concentrate on best practices. The basics of how the new full text search feature works is found in Books On Line.

If you would like a good introductory tutorial, start with SQL 2008 Books Online, then read this whitepaper: http://msdn.microsoft.com/en-us/library/cc721269(SQL.100).aspx. It is one of the best whitepapers I’ve seen recently on a product feature.

Also for more introductory lessons here is an excellent link: http://sqlblogcasts.com/blogs/simons/archive/2008/02/19/SQL-Server-2008---iFTS-Introduction.aspx

Simon Sabin also has a nice series of posts on the new iFTS features: http://www.sqlskills.com/blogs/simon/2008/02/20/SQLServer2008IFTSNewFeatures.aspx

The new iFTS feature is not rocket science, but you should expect to spend some time learning about the new vocabulary, like Fragments, Master Merge, etc. You will also spend some time learning the new DMVs (like sys.dm_fts_index_keywords and sys.dm_fts_index_keywords_by_document) .

Performance

Here is what we noticed about the performance of iFTS in a production environment.

The full text searches and queries are very fast if you can keep the full text indexes up to date. There are two choices – automatic or manual index updates. Most applications should be able to run with automatic tracking turned on. The higher volume transaction systems may start to experience some blocking on inserts and updates at times. The blocks are short and do not last more than a few milliseconds most of the time. However, if your transaction volume is high enough where the blocking starts to impact the user experience, then turn off automatic updates and run manual merges. A merge will take the word fragments and merge them with the master fragment.

To get the best performance from your full text index, do the following new iFTS Best Practices:

1. Put the full text index in its own filegroup. This will avoid fragmenting the main data file(s).

2. Use Varchar(max) instead of image or text for the fields that you want to use as the base for your full text index.

3. Turn off auto updates if you see blocking on inserts and updates and do manual merges.

4. Avoid running Master Merges and a rebuild/reorg on another index on the same table at the same time. Make sure a Master Merge job does not overlap with a job that rebuilds a non clustered index on the same table.

Another interesting thing we learned is that if you have to rebuild the entire full text index for any reason, the full text queries will still run. You will notice that they may be very fast at first. This is because a full rebuild will drop the index and all the fragments, so the queries will be running against an empty index at first. They will continue to run as the index is built. The good thing is that you should rarely, if ever, have to do a full rebuild of the index. Instead, any major work should be done with ALTER INDEX START FULL POPULATION. This not a perfect solution since it does UPDATE statements to the index, which does its normal exclusive lock. It does the update in chunks which means that the locks are only briefly held. However, you will experience more blocking than normal for the full text queries while the update is running.

iFilters

Different document types need iFilters installed. Most common document types have available iFilters. The one exception at the time of this writing is the 64 bit PDF filter. There is one available but it has bugs that cause aborts. Hopefully Adobe can deliver the fixes soon.

Limitations

As good as the performance is on the new iFTS feature, there are still some limitations that will cause concern for some projects. First, the table must have a single column primary key. Second, the SWITCH command for partitions does not work. Third, there are no good scale out solutions with iFTS. Multiple indexes cannot appear as one and iFTS does not work with distributed partitioned views. We are working on improving these limitations for the next version of SQL Server. If you need a higher volume solution, ask your local Microsoft contact about FAST.

One other item you will be interested in is that the full text index size in SQL 2008 is approximately 50% bigger than the full text indexes in SQL 2005. This is generally not a problem, just a fact that you should know.

Handling multiple languages in a single document is a hard problem. Which word breaker do you use to shred the original document, and which language are you going to specify for the query? For example, if you have a document with Korean and English and you use the Korean word breaker to process the document, then if you search the document for English words it will only find the exact words and not any other forms of the words (like ing and s).

Memory Settings

The new full text daemon (FDHost) runs as a windows program and is not part of the SQL Server process, which means that they will compete for memory. This is normally not a concern because SQL Server generally plays well with other processes running on the same server. However, we noticed that if you have very large documents, like greater than 100MB, or even hundreds of 2MB documents being parsed at the same time, then you may want to make some configuration changes.

There are two options in this case:

1. Sp_fulltext_Service can be used to set the block size. Try increasing the block size to 512k.

2. Use sp_configure to set the maximum memory for SQL Server down a bit. How much down will depend on how many documents you intend to process simultaneously.

Other notes about FDHost

· The Katmai FDhost runs outside of SQL server memory space. There are a lot of similarities between Yukon MSFTEFD and Katmai FDHost processes.

· The Filter daemon manager (runs within SQL process) manages the FDHost process and it will launch one FDHost process for single thread and one for multi-thread FDHost processing. These two do not necessarily start at the same time.

o Single or multithreaded fdhost depends entirely of the iFilter used: if the document filter supports multithreading, then we load it into a multithreaded ready fdhost, otherwise not.

o Named Pipes must be enabled on the SQL server box because Filter daemon manager uses named pipes to communicate with FDHost.

· FDHost processes are not shared across multiple Katmai iFTS instances on the same server.

· During the full population of the FT indexes, the FDHost process may need a lot of memory. It’s a good practice to:

o Schedule the full FT index population off business hours when the SQL server is not being used by other processes or queries.

o You may need to manually reduce the SQL Server max memory to give more memory to FDHost as noted above. However, if SQL Server is not being heavily used, FDHost will get the memory left on the box.

· To improve the performance between FDHost manager and FDHost, users can change some configuration settings (such as ISM – internal shared memory, batch size, etc.) to improve the consumption process of the chunks of data sent to the FDHost.

Miscellaneous notes:

· The New Database dialog box in Management Studio has an option grayed out. Just below the name and owner there is a grayed out check box. In the released version of SQL Server 2008 the full text options are on by default. This was left in place in case any customers had references to it in scripts.

· iFTS support for Danish, Polish and Turkish is not on by default. These word breakers are delivered with SQL Server. Since they are written by 3^rd party companies it was decided to leave them off by default.

Upgrade

If you are upgrading from SQL 2005, then you have a decision to make on whether to just upgrade the full text index or start over and rebuild it. The best practice recommendation is that if there is a new word breaker available for your language, then you should completely rebuild the full text index after upgrading, if you want to use the new word breakers and stemmers. http://msdn.microsoft.com/en-us/library/cc721269(SQL.100).aspx has a list of which word breakers are exactly the same and which are new.

Kevin Cox and Xiaoyu Li

SQL PASS 2008 Just may be the best PASS Conference yet

marksou — Fri, 26 Sep 2008 02:30:00 GMT

Coming up this November 18-21^st is the SQL PASS Conference in Seattle, WA (right in Microsoft’s backyard). See http://summit2008.sqlpass.org/ for all the details. This SQL PASS will be very special. The SQL Server engineering team is sending up to 300 developers to the conference. You won’t have to walk very far in order to run into one. To meet the engineers 1:1 and discuss your SQL Server applications should be a big justification for attending.

The SQLCAT – SQL Server Customer Advisory Team will be there in full force for the entire four days. You won’t miss us as we will all have the same colored shirts with our SQLCAT logo. We will have a SQLCAT Track with up to 15 sessions focusing mostly on interesting learning’s from early deployments of SQL Server 2008. You can find details of the sessions on our website: SQLCAT Track at PASS. We want to hear from you. We want to learn about your SQL Server deployments, your successes and your challenges. You are more than welcome to walk up to a SQLCAT employee at anytime and say hello. There may even be a surprise or two if you do.

I look forward to seeing you there!

New Best Practices Articles Published - Running Microsoft SQL Server 2008 Analysis Services on Windows Server 2008 vs. Windows Server 2003 and Memory Preallocation: Lessons Learned

carl.rabeler — Wed, 16 Jul 2008 21:49:00 GMT

Due to the improvements in the Windows Server 2008 memory manager related to the change in the algorithm for scanning pages in memory, SQL Server 2008 Analysis Services performed equally well during both partition and dimension processing with or without memory preallocation when running on Windows Server 2008. However, SQL Server 2008 Analysis Services performed substantially better during both partition and dimension processing with the use of the memory preallocation configuration setting when running on Windows Server 2003. Specifically, the processing performance of Analysis Services 2008 without memory preallocation on Windows Server 2008 was virtually identical to the processing performance of Analysis Services 2008 on Windows Server 2003 with memory preallocation. With the hardware that we used in our tests and with the Analysis Services objects that we processed, we observed a performance benefit of approximately 100% during partition processing and a performance benefit of between 30-40% during dimension processing.

If you use memory preallocation with SQL Server 2008 (or SQL Server 2005), use a value that is low enough to ensure that sufficient memory remains for other processes on the computer (avoiding paging) and high enough for Analysis Services (use the peak value for the Process: Private Bytes counter for the msmdsrv instance to establish this value).

Note: While we did not specifically test SQL Server 2005 Analysis Services for this technical note, the results discussed here apply to Analysis Services 2005 as well as to Analysis Services 2008 because the underlying code base for the portion of Analysis Services 2008 that we tested has not changed.

http://sqlcat.com/technicalnotes/archive/2008/07/16/running-microsoft-sql-server-2008-analysis-services-on-windows-server-2008-vs-windows-server-2003-and-memory-preallocation-lessons-learned.aspx

New Best Practices Articles Published - Scaling Up Reporting Services 2008 vs. Reporting Services 2005: Lessons Learned

carl.rabeler — Wed, 16 Jul 2008 21:45:00 GMT

Reporting Services 2008 was able to respond to 3–4 times the total number of users and their requests on the same hardware without HTTP 503 Service Is Unavailable errors compared with Reporting Services 2005, regardless of the type of renderer. In stark contrast, Reporting Services 2005 generated excessive HTTP 503 Service Is Unavailable errors as the number of users and their requests increased, regardless of the report renderer.

Our tests clearly show that the new memory management architecture of the report server enables Reporting Services 2008 to scale very well, particularly on the new four-processor, quad-core processors. With our test workload, Reporting Services 2008 consistently outperformed SQL Server 2005 with the PDF and XLS renderers on the four-processor, quad-core hardware platform (16 cores) both in terms of response time and in terms of total throughput. Furthermore, with these renderers on this hardware platform, Reporting Services dramatically outperformed other hardware platforms regardless of Reporting Services version, responding to 3–5 times the number of requests than when running on either of the other hardware platforms. As a result, we recommend that you scale up to four-processor, quad-core servers for performance and scale out to a two-node deployment for high availability. Thereafter, as demand for more capacity occurs, add more four-processor, quad-core servers.

Finally, with all renderers and with all hardware platforms using our test workload, the performance bottlenecks were the processor on the front-end server and the disk subsystem on the data source with Reporting Services 2008, whereas the Reporting Services front-end Web service was the performance bottleneck with Reporting Services 2005.

http://sqlcat.com/technicalnotes/archive/2008/07/09/scaling-up-reporting-services-2008-vs-reporting-services-2005-lessons-learned.aspx

SSIS 2008 - world record ETL performance

tkejser — Fri, 29 Feb 2008 11:13:00 GMT

Using SQL Server 2008 and SSIS 2008 the SQL Server Performance team working with Unisys and SQL CAT have shown SSIS scaling to world record performance! We loaded 1.18 TB in under 30 minutes.

You can find more information here: http://blogs.msdn.com/sqlperf/archive/2008/02/27/etl-world-record.aspx Also watch for a paper from the performance team about how this was achieved and lessons learned in this exercise.

In addition, the SQL CAT team have been doing work on fast data loads using bulk inserts. Watch sqlcat.com for TechNotes and best practices learned from this exercise.

Introducing SQLCAT.com

denny.lee — Sat, 02 Feb 2008 20:05:00 GMT

The SQL Customer Advisory Team has put the final touches on the site http://sqlcat.com/ which is our new hub for all things that are part of the SQLCAT! Our new site is a portal to all SQLCAT resources including:

Top 10 Lists: Summary list of Best Practices and Recommendations
Technical Notes: Deep level technical short papers
Technical Spotlights: Technical end-to-end customer case studies
Searchable and Tagger friendly; post comments and provide feedback!
Search all of our SQLCAT Best Practices Whitepapers
Easily find our SQLCAT Blogs and other materials

The site is regularly updated with deep technical information on enterprise SQL implementations.

New Partition Management Tool Available on Codeplex

stuarto — Mon, 28 Jan 2008 23:02:00 GMT

As anyone working with partitioned tables knows by now, the process of creating stand-alone ‘staging’ tables to support switching data into or out of a partition can be a management challenge, especially if the design of the partition table or its indices change over time. A staging table for new data to be switched-in must include check constraints that match the partition boundaries of the target, and include all columns, constraints and indexes appearing in the partition table. Often, the code to create the staging tables resides in a set of manually created management scripts that must remain carefully synchronized to any changes made in the partition table design. When columns are added, column characteristics change (such as nullability or length) or index definitions are tweaked, the management script creating the staging tables must also be updated. If management scripts fall out of synch, often we discover it only while in the middle of a critical month-end partition switch process, raising the risk of unplanned extension or delay of the maintenance window.

I’ve developed a simple, shared source Partition Management Tool for SQL Server 2005 to make the process of creating the staging tables for partition management nearly foolproof. At runtime, the definition of all columns, indexes, and constraints for the partition table are read using the SMO object model, and a perfect clone table matching a specific partition is created.

The tool also provides an option for creating a staging table without indexes, and then for building indexes later after the table is populated – for applications that prefer to bulk load into a new partition without indexes present. It also includes a quick, transparent method to very quickly and simply clear the contents of a single partition within a partitioned table – a task that you may want to perform when replacing the complete contents of one partition.

Also rather than requiring you to have knowledge of the specific partition number that you are referencing, you may indicate the appropriate partition by passing a value of the partitioning column that falls within the desired partition range when invoking the tool.

The Partition Management Tool runs from the command line, so unfortunately it cannot be used from within SQL Server stored procedures or batches. That’s because SMO-based CLR routines cannot run in-proc in SQL Server 2005. But the tool can be invoked from within SSIS (as an Execute Process task), so that runtime-dependent package variables can be integrated and the script can become part of an automated sliding window scenario orchestrated by SSIS. When invoking from SSIS, the command line arguments of the tool can be populated by expressions within SSIS.

You can download the tool and associated C# source code from SQL Server 2005 Partition Management Tool .

Stuart Ozer

FoxPro to SQL Server migration experience

kevincox — Fri, 25 Jan 2008 21:52:00 GMT

At first I was going to write a detailed white paper about the conversion effort. But someone has already done it. There is a very good guide at:

http://nationalcom.com/home/download/Conversion-VFP-SQLServer.pdf

Since FoxPro is nearing its end of life, the company I was working with needed to convert the system to newer technology. They chose Microsoft SQL Server and Microsoft .NET. There was over 1 terabyte of data being processed in FoxPro.

The project was fairly easy as far as the data conversion goes. There were a few decisions to make and one thing we learned about the difference in the NUMERIC data type between FoxPro and SQL Server.

The end result is pretty exciting. Some of the processing jobs that took over 1 day in FoxPro now take 25 minutes in SQL Server. The improvement is due to several facts: 1) The FoxPro code was doing row-at-a-time processing and the Transact-SQL code was written to do set processing; 2) FoxPro has a file limit of 2GB and with over 1TB of total data there were almost 1000 files to open and read through.

The most difficult part of the project was taking years worth of FoxPro code from different developers. The code had started with DOS FoxPro and had been converted to Windows FoxPro and upgraded by different people with different coding styles over the years. And as usual, the documentation was the code. J

Here are the notes from the data conversion effort. I’m only covering the data types that we had in our schema, which was not a very wide range. If you have more data types they are most likely covered in the guides noted above.

1. If you have MEMO data type in FoxPro you should use VARCHAR(MAX) in SQL Server. We could have used TEXT but we know that is being phased out in some future version in favor of VARCHAR(MAX). And VARCHAR(MAX) has better performance.

2. There are many DATE types in FoxPro. The decision was pretty easy to convert them all to DATETIME in SQL Server 2005. In SQL Server 2008 there are more DATE types so you will have to study which FoxPro types map to the new SQL Server 2008 types. DATETIME data was exported in character format using YYYYMMDDHHMMSS.

3. In FoxPro NUMERIC(5,2) allows a range of -99999 to 99999. In SQL Server using the same precision the range is -999.99 to 999.99. This was the only data type in our schema that needed to be altered at the data type level. So in SQL Server we had to use NUMERIC (7,2) to get the same range.

4. FoxPro LOGICAL fields were either exported as T/F or 0/1. They either went into a BIT data type in SQL Server or a TINYINT. The TINYINT was chosen if we wanted to do any calculations on the field in the future; i.e. SUM, AVG, etc.

Not knowing much about SQL Server, the database guys made a pretty good choice to dump the data to flat files in character mode, then used a simple Transact-SQL script to loop through the files and import them using the BULK INSERT command. A quote from one of the lead developers on the project: “We knew about Integration Services and could have used that but with our lack of experience with this tool we decided to use T-SQL scripts instead”. The scripts and data conversion took one person about 3 weeks to write and tune. This same person converted most of the other server side processing scripts to Transact-SQL in about 2 months.

Since FoxPro has a limit of 2GB file size and there was over 1TB of data in total, there were almost 1,000 files to convert. Everything was exported from FoxPro as character using a tab delimiter. It was determined that this was a safe delimiter after having to go through the memo fields to make sure there were no tab characters embedded.

Just as a side note, the FoxPro screens were all rewritten and reengineered using ASP.NET and C#. I will let someone else write about that experience if they wish.

Thanks to Spyros Christodoulou and Petros Hadjigeorgis from Nielsen EMEA for their assistance in writing this blog and for making the project successful.

kevin cox

SQL Server 2005 Best Practices Analyzer (January 2008) -- Now Available!

Paul Mestemaker — Sat, 19 Jan 2008 00:36:00 GMT

It's a new year and, with it, we come bearing gifts! We have a pretty significant update to SQL Server 2005 Best Practices Analyzer. It contains many new and updated rules for Analysis Services, a few important rules for the Relational Engine, a couple bug fixes for the UI and command line tools. And all of these rules have rich documentation telling you what need to know. ......(read more)