TSQLNet Weblog - Cihangir Biyikoglu

Upgrading to SQL Server 2005 - How DEFAULT_SCHEMA helped improve performance for a number of apps.

cihangir — Fri, 30 Mar 2007 17:22:00 GMT

Here is my blog post at the ISV Team blog. DEFAULT_SCHEMA setting helped in a number of cases, especially with legacy code. It could help you if have upgraded from an earlier version of SQL Server and you are not parameterizing your trivial queries. http://blogs.msdn.com/mssqlisv/archive/2007/03/23/upgrading-to-sql-server-2005-and-default-schema-setting.aspx

-Cihan

PASS 06: Hidden Gems in SQL Server 2005 - Slides available at the PASS website

cihangir — Mon, 04 Dec 2006 20:05:00 GMT

Greetings, my pass talk is now available at sqlpass.org for download. The session was on some of the great hidden parts of sql server that served us well in real life imlpementations. There are quite a few I could have mentioned but I picked the top 10 that I think is the most impactful. Here is the list;

10 Tablediff.exe : Table1 == Table2?
9 Trigger for Logon events (SP2)
8 Boosting performance with persisted-computed-columns
7 Can you fix bad code automagically? DEFAULT_SCHEMA setting in sys.database_principals
6 Can you fix more bad code automagically? Forced Parameterization
5 Same data, less space? (SP2) VarDecimal storage format
4 Autoindex: What would you like to index today?: sys.dm_db_index_usage_stats & sys.dm_db_missing_index_*
3 “What is the hottest query?” answered in seconds: sys.dm_exec_query_stats
2 Scalable Shared Databases - Scaleout for eradonly reporting servers.
1 Softer side of Numa - Soft vs Hard Numa - the magic of '8'.

Enjoy

MARS, Transactions and SQL Error 3997, 3988 or 3983

cihangir — Tue, 21 Nov 2006 00:24:00 GMT

Mars and Transactions

MARS (multiple active result-sets) does allow multiple commands to be active on the same connection, however there are some limitations on allowing multiple transactions to be active. I received a number of questions around this lately. I'll try to address the common denominator scenario in these questions and how to work around some of the error messages you may receive if MARS does not like what you are trying to do with your transactions.

Scenario:

The scenario in question involves implementing a flow that reads a result-set from SQL Server in a loop and goes back into the database to update some records. The app executes the update(s) in a transaction.

Here is the simplified ado.net code: Note that I am leaving out the cleanup and some of the initialization for compactness and table t1 is defined as "id int primary key, c2 xml not null".

...

SqlConnection mycconnection1 = new SqlConnection(@"...;MultipleActiveResultSets=true");

SqlCommand mycommand1 = new SqlCommand("select * from t1", mycconnection1);

SqlCommand mycommand2 = new SqlCommand("update t1 set c2='<test>123</test>' where id=@id", mycconnection1);

mycommand2.Parameters.Add("@id", SqlDbType.Int, 4);

mycconnection1.Open();

SqlDataReader mydr = mycommand1.ExecuteReader();// SELECT * FROM T1

while (mydr.Read()) //WHILE READING RESULTS

{

(new SqlCommand("begin tran", mycconnection1)).ExecuteNonQuery(); //BEGIN TRAN

int id = mydr.GetInt32(0);

mycommand2.Parameters["@id"].Value = id;

mycommand2.ExecuteNonQuery(); //UPDATE T1 SET C2=... WHERE ID=@ID

(new SqlCommand("commit tran", mycconnection1)).ExecuteNonQuery(); //COMMIT TRAN

}

...

The code will error at runtime. The error message you get back will depend on the method you used to manage your transactions but it is likely going to be one of these;

SQL Error 3997 "A transaction that was started in a MARS batch is still active at the end of the batch. The transaction is rolled back."

SQL Error 3988 "New transaction is not allowed because there are other threads running in the session."

SQL Error 3983 "The operation failed because the session is not single threaded."

So… What now?

When thinking about transactions it is critical to understand that almost all statements in SQL Server run in a transaction without you having to implicitly or explicitly start one. Most obvious example is : 'delete from t1' will delete all records or none. There is nothing in between when SQL Server executes this statement. In the case above however, starting a new transaction on connection that already has an active atomic command executing is not allowed. Depending on the situation and the behavior you want you may be able to do one of the following;

#1 You can simply encapsulate all commands in a single transaction. Obviously in this case, you get one big transaction and that means you hold on to all your resources until the end of the entire transaction. That could mean more blocking in your application.

SqlConnection mycconnection1 = new SqlConnection(@"Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=tempdb;Data Source=patates\yukon;MultipleActiveResultSets=true");

SqlCommand mycommand1 = new SqlCommand("select * from t1", mycconnection1);

SqlCommand mycommand2 = new SqlCommand("update t1 set c2='<test>123</test>' where id=@id", mycconnection1);

mycommand2.Parameters.Add("@id", SqlDbType.Int, 4);

SqlTransaction mytransaction1;

mycconnection1.Open();

mytransaction1 = mycconnection1.BeginTransaction();

mycommand1.Transaction = mytransaction1;

mycommand2.Transaction = mytransaction1;

SqlDataReader mydr = mycommand1.ExecuteReader();

while (mydr.Read())

{

int id = mydr.GetInt32(0);

mycommand2.Parameters["@id"].Value = id;

mycommand2.ExecuteNonQuery();

}

...

#2 alternatively, you can submit the inner command you want transacted as a single batch or pack the commands in a stored procedure with the transaction statements with the transaction statements in the batch or the procedure. SQL Server can allow this special case and the batch or the stored procedure can work in its own transaction. So the code looks like this;

//BATCH

SqlConnection mycconnection1 = new SqlConnection(@"Integrated Security=SSPI;Persist Security Info=False;Initial Catalog=tempdb;Data Source=patates\yukon;MultipleActiveResultSets=true");

SqlCommand mycommand1 = new SqlCommand("select * from t1", mycconnection1);

SqlCommand mycommand2 = new SqlCommand("begin tran;update t1 set c2='<test>123</test>' where id=@id;commit tran", mycconnection1);

mycommand2.Parameters.Add("@id", SqlDbType.Int, 4);

mycconnection1.Open();

SqlDataReader mydr = mycommand1.ExecuteReader();

while (mydr.Read())

{

int id = mydr.GetInt32(0);

mycommand2.Parameters["@id"].Value = id;

mycommand2.ExecuteNonQuery();

}

...

This is stating the obvious but the code above works because we have a well defined boundary for the transactions.

If you have comments on MARS and transactions and other data retrieval ‘flows’ that you are having difficulties with, please send those my way through the comments section below.

That's it for now. Thanks

Financial Services Developers Conference Presentation - Data Integration

cbiyikoglu — Wed, 26 Apr 2006 13:24:00 GMT

FSG Dev. Conf. 06 is over. Thanks to the folks who attended. Here is the presentation and the Demos as promised for the SQL Server Data Integration - Turning Data Into Intelligence.

"OPENROWSET BULK" and "Fuzzy Lookup" were the champions of the presentation. I'd certainly like to find out about your experience with both of these components.

Many Thanks

SQL ISV Advisory Team Blog

cbiyikoglu — Tue, 28 Mar 2006 20:55:00 GMT

ISV Advisory team is starting to blog as well. Good one to bookmark!

http://blogs.msdn.com/mssqlisv/Default.aspx

Forced and Simple Parameterization in SQL Server 2005

cbiyikoglu — Fri, 04 Nov 2005 03:18:00 GMT

As the optimizer gets more and more sophisticated, cost of optimization naturally increases. When that is the case; simple and forced parameterization comes to the rescue (much like you neighborhood plumber). These features simply allow the reuse of execution plans from similar queries thus minimizing the parse & compile time. The benefits vary from workload to workload but applications with simpler-but-chatty query workloads that spend relatively large percentage of the execution time in compilation (parse and compile time outputed from STATISTICS TIME output) will benefit greatly from these features in SQL Server 2005.

Many of you are probably familiar with auto or simple parameterization so I’ll jump to the new forced option right away. This is a new database level option in SQL Server 2005. It simply lets you parameterize a much wider range of queries. The following queries for example would not get parameterize in SQL Server 2000 or 2005 so they would by default be parsed and complied twice.

"select top 1 c1 from t1 where c2=0" & "select top 1 c1 from t1 where c2=1"

However force parameterization can help turn this query into a parameterized form that gets reused on the second execution.

"(@0 int)select top 1 c1 from t1 where c2 = @0"

Forced Parameterization comes with a warning however: sometimes you want similar queries to recompile and use distinct execution plans. With forced parameterization queries may end up sharing plans inappropriately. I highly recommend you digest the information on the book online page and test before you decide to enable this in your environment.

What I’d like to do for the rest of the text is to talk about what you will oserve in SQL Server 2005 execution plan cache with parameterization options, as it may seem a little unusual once you start digging in.

First, if you look into your good old syscacheobjects or the new sys.dm_exec_cached_plans views, you will still see the individual statements with the literals in the query text in your execution plan cache (both in simple and forced parameterization). However if you observe closely you will see that we do not have full blown execution plans for these objects with literals (sys.dm_exec_query_plan) and naturally we do not keep query statistics (dm_exec_query_stats) because they never really get executed. Their purpose in life is mainly to serve as pointers back to the actual parameterized plan. In fact the sys.dm_exec_cached_plans.refcounts of the parameterized plan reflects the count of these empty objects in the cache. The presence of these objects does not mean simple or forced parameterization is not doing its job. As long as the parameterized version is in the cache, you are in good shape. You could use something like the following query to see the query text and the plans of the cached queries:

select c.text,a.refcounts,a.usecounts,d.query_plan from sys.dm_exec_cached_plans a outer apply sys.dm_exec_plan_attributes(a.plan_handle) b outer apply sys.dm_exec_sql_text(cast(b.value as varbinary)) c outer apply sys.dm_exec_query_plan(a.plan_handle) d where b.attribute='sql_handle' and a.cacheobjtype = 'Compiled Plan' order by a.usecounts desc

I also have seen a few people trip over this: when looking at compilations, performance monitor (perfmon) counters can be deceptive. You may see a none-zero SQL Compilations/Sec value even after enabling forced parameterization. Don’t panic… We still report compilation with simple/forced-parameterized queries. If SQL Server 2005 is not reporting Failed or Unsafe Auto-Params/Sec, then we are doing all we can to reuse execution plans.

Happy Testing…

Cihangir Biyikoglu

SQL Server 2005 is out the door

cbiyikoglu — Sun, 30 Oct 2005 19:12:00 GMT

Well, many lines of code and 5 years later, SQL Server 2005 is out the door. I have been in the lab benchmarking with close partners lately and I am certainly impressed with the results. With all the added sophistication and new features, we are still providing many perf benefits to apps upgrading from SQL 2000. I thought I point out one cool presentation to people who are doing the same type of work these days: This presentation highlights all the tranparant features that you get for free when you upgrade to 2005.

http://msevents.microsoft.com/cui/WebCastEventDetails.aspx?EventID=1032275601 &EventCategory=5 &culture=en-US &CountryCode=US

Phenomenal Result from Dell/Intel and x64 SQL Server 2005

cbiyikoglu — Tue, 27 Sep 2005 21:50:00 GMT

I have not had too much time to blog lately but here is a great result from SQL Server 2005 x64 on Dell Poweredge 2800 with Dual-Code Intel Xeon.

http://www.tpc.org/tpcc/results/tpcc_result_detail.asp?id=105092601

The $0.99 Price/Performance number is a great results that proves how far we came on pushing the cost of technology down. For a little over $38K, being able to drive close to 31K users on the standard edition of SQL Server 2005 and Windows 2003 is just phenomenal. I think dual-core should take off (if it has not already) from here.

Enjoy.

SQL Server in the Financial Services Industry Webcast

cbiyikoglu — Thu, 30 Jun 2005 23:30:00 GMT

I shared some of our experiences with financial services industry implementations in a 90 min webcast. I touched on 5 distinct scenarios where SQL Server 2005 may help financial services partners and customers.
- Cost: full lifecycle of a solution - Development, deployment and maintenance
- Regulations and Compliance issues - How does SQL Server 2005 help you comply with Sarbanes-Oxley, AMI, BASEL II etc.
- Data and Information integration - Access to many data formats and data system from within SQL Server.
- Business Continuity and Reliability - Low cost High Availability
- Scalability and Performance

The recorded version of the live webcast is available here:
http://msevents.microsoft.com/CUI/WebCastEventDetails.aspx?EventID=1032275631&EventCategory=5&culture=en-US&CountryCode=US

Cihangir Biyikoglu - SQL Server Program Manager

Great Samples on Service Broker

cbiyikoglu — Wed, 25 May 2005 00:55:00 GMT

I recently came across a few good samples:

Async Task Management: http://www.microsoft.com/technet/technetmag/issues/2005/05/ServiceBroker/default.aspx

Very Basic Trading System: http://blogs.msdn.com/rushidesai/archive/2005/05/19/420212.aspx

Enjoy!

Algorithmic Trading

cbiyikoglu — Thu, 12 May 2005 01:29:00 GMT

I recently started reading about Algorithmic Trading. There is a huge potential here for computer science. According to this presentation by Carl Carrie - JPMorgan & Scott Atwell - Manager of FIX Trading and Connectivity: (http://www.jandj.com/presentations/wednesday/AlgoTradingFIX.pdf), algorithmic trading will make up %21 of the buy-side trading. That is huge. Certain futurists even believe traders will be endangered species within our lifetime. I am a little skeptic on that one but I have been proven wrong once or twice before :).

There certainly is a great need for high performance computing, large-volume and real-time data processing in the financial vertical and SQL is ready for that type of game with CLR integration. I have seen very creative uses of SQL and SQLCLR already but if you are tackling these types of tough problems, contact me... I am certainly interesting in hearing about it.

By the way; all Financial Services DevCon material is available at http://www.financialdevelopers.com/fsdevcon05.aspx

Slides and Demos from Financial Services Developer Conference in NY

cbiyikoglu — Mon, 25 Apr 2005 19:38:00 GMT

I promised to make the presentation and demos available here and here they are:

Presentation: http://www.tsqlnet.com/downloads/fsgdevcon/fsgdevcon_presentation.zip

Data Integration Demo: http://www.tsqlnet.com/downloads/fsgdevcon/fsgdevcon_demo.zip

I have only demo #1 shared here. Demo is on data integration and was run on SQL Server 2005 April CTP. I demonstrate a few uses of OPENROWSET BULK and relational, custom, semi or unstructured files using format files, UDTs, TVFs, XML data types and SINGLE_BLOB options. Very similar to the example in my earlier blog entry: http://blogs.msdn.com/cbiyikoglu/archive/2005/03/17/398321.aspx

Demo #2 was on the great profiler and perfmon integration we achieved: if you'd like to see it for yourself, just record perfmon and profiler together against sql server while you have some decent load. Then open the profiler data you save and use "file>import performance data" to bring in correlated perfmon data. Just play with selecting a range from perfmon or clicking commands in profiler etc.

Let me know if you have trouble getting the data integration demo to work or if you have any questions. As I said I am very interested in hearing about the testing you guys are doing on SQL Server 2005 and would love to hear about any backward compatibility issues and any numbers comparing your SQL Server 2000 and 2005 numbers on the same HW.

FSG Developer Conference in New York

cbiyikoglu — Sun, 10 Apr 2005 22:23:00 GMT

Next week, I am heading out to the financial services developer conference to deliver a SQL Server 2005 presentation. The toughest part of delivering a talk on SQL Server 2005 is to find a way to fit it all into a 90 minute slot. You can talk about individual features for 24 hours nonstop but I am planning on taking a different approach this time around: instead of talking about individual features, I'll cover the scenarios we enabled in SQL Server 2005… Feature bags that can help solve a number of classic problems that resonate with the financial developer. Here is a quick peak at the list:

SQL Server Central Data Store: Enhanced bridging capabilities for bringing heterogeneous systems and data together.
Extensibility & Rich Development Platform: Great developer productivity features and language synergy: SQLCLR, XML and TSQL.
Privacy, Auditing and Security: Hardened and reduced product surface and new encryption capabilities.
Improved Scalability & Performance: Scale from pocket devices to high end servers. I'll also talk about query processing enhancements that can help your app when you upgrade your backend to SQL Server 2005.

This talk is talking place on the 21st of April at 12:30 in the breakout room at the Grand Hyatt. I'll see you there.

Some good blogs I read today.

cbiyikoglu — Wed, 23 Mar 2005 21:43:00 GMT

I don't like pointing to other blogs too much but I want to make sure noone misses these:

SQLOS Caching by Slava

SQL Testing from Ed Triou and Andre Milbradt

Enjoy!

Data Integration - Part II - UDTs and OPENROWSET to parse custom file formats

cbiyikoglu — Thu, 17 Mar 2005 23:22:00 GMT

SQLCLR is the Swiss Army knife with 1000s of foldable parts. You cannot beat the ability to plug right in and reuse good set of framework functions from the DBMS. If your DBA is one of those that say; “no way. I am not letting C# into my database”, this is what I’d tell him… I am sure one day someone insane enough will to try to write the entire framework using sql but until then we need SQLCLR. =)

Enough said: Here is a quick example of how to use UDTs as a way to parse custom file formats. I am not sure if this is the killer UDT demo but it certainly is a good example of SQL Server 2005 as a data integration platform.

Let’s say you have a folder full of files that have the following custom format and want to bring this data into your database. You have the ticker, the date and how many transactions are included in the file are all divided by ";". Right next to that part, you have and array of volume+price data delimited by “|”. This format and the data are totally arbitrary but you get the idea.

File1.txt - MSFT;10/10/04;3;10|5.001;20|6.05;30|7.33
File2.txt - ORCL;10/11/04;5;1|1.01;20|5.05;300|4.04;4000|3.03;50000|2.02
File3.txt - IBM;10/10/04;2;100|0.5;10|0.025

You create a user defined type - UDT that can parse this out.

CustomTradeDataStruct: This is the main struct for the whole file.

TradeArrayStruct: This is the struct that represents the variable length array for volume and price data.

[Serializable]

public struct TradeArrayStruct

{

public int volume;

public decimal price;

public TradeArrayStruct(string s){…}

public override string ToString(){…}

void IBinarySerialize.Read(System.IO.BinaryReader r){…}

void IBinarySerialize.Write(System.IO.BinaryWriter w){…}

};

[Serializable]

[SqlUserDefinedType(Format.UserDefined, MaxByteSize = 8000)]

public struct CustomTradeDataStruct : INullable, IBinarySerialize

{

string ticker;

string startdate;

int daterange;

TradeArrayStruct[] trades;

public CustomTradeDataStruct(string s){…}

public override string ToString(){…}

public bool IsNull {…}

…

public static CustomTradeDataStruct Null {…}

public static CustomTradeDataStruct Parse(SqlString s) {…}

void IBinarySerialize.Read(System.IO.BinaryReader r){…}

void IBinarySerialize.Write(System.IO.BinaryWriter w){…}

}

Ok here is the stuff we have been building up to:

SELECT cast(t1.c1 as CustomTradeDataStruct).ToString()

FROM OPENROWSET(BULK 'C:\File1.txt', SINGLE_NCLOB) AS t1(c1)

Now you have the data in SQL Server, parsed through a UDT. There is a great set of things you can do to improve the access to this data like creating a bunch of properties to allow access to individual fields in the file or creating a table valued function to manipulate the volume+price array like a table etc.

Cool or not so cool? Let me know what you guys think.