SQL Server - The One Stop Shop

Accessing REST based Web services using SQL CLR

mrkuldeep — Wed, 18 Jun 2008 15:57:00 GMT

In one of scenario I came across with requirements to consume REST based web services in SQL Server, I have done research and found that this can be very well done using SQL CLR features, before I start with how to use SQL CLR to consume REST web services, it is imperative to explain you the REST web services concept.

What is REST web service?

REST can be described as an architecture style. The acronym REST stands for ‘Representational State Transfer’, this means that every unique URI is representation of some object and you can get the content of this object using HTTP GET as well as you can use POST, PUT and DELETE to play with content of this object.

REST - An Architecture style, not a standard

This is really good to know that REST is not a standard as you will not see any W3C standards putting any REST specifications. The reason is that REST is simply an architecture style, you can’t pack a toolkit of this; instead you can use this style during writing your web services. Though REST is an architecture style but it use certain standards like:

· HTTP

· text/xml, text/html, etc (MIME types)

· URL

You can get in detail information on REST web services by referring to below links

http://www.xfront.com/REST-Web-Services.html

http://bitworking.org/news/How_to_create_a_REST_Protocol

http://www.petefreitag.com/item/431.cfm

PS: REST based web services in .NET can be written using WCF .NET version 3.5

How to Call REST Web Service using SQL CLR

REST based web services are most of the time in form of URI, to consume these web services one has to write SQL CLR stored procedures. I have taken an example to make standard PUT and POST calls to REST based web services, this code can be used to practically make call to any REST web services that have PUT or POST payload. Below code snippet describes .NET code written to make CLR stored procedures

using System;

using System.Data;

using System.Data.SqlClient;

using System.Data.SqlTypes;

using Microsoft.SqlServer.Server;

using System.Net;

using System.IO;

public partial class StoredProcedures

{

[Microsoft.SqlServer.Server.SqlProcedure]

public static void SampleWSPut(SqlString weburl, out SqlString returnval)

{

string url = Convert.ToString(weburl);

string feedData=string.Empty;

try

{

HttpWebRequest request = null;

HttpWebResponse response = null;

Stream stream = null;

StreamReader streamReader = null;

request = (HttpWebRequest)WebRequest.Create(url);

request.Method = "PUT"; // you have to change to

//PUT/POST/GET/DELETE based on your scenerio…

request.ContentLength = 0;

response = (HttpWebResponse)request.GetResponse();

stream = response.GetResponseStream();

streamReader = new StreamReader(stream);

feedData = streamReader.ReadToEnd();

response.Close();

stream.Dispose();

streamReader.Dispose();

}

catch (Exception ex)

{

SqlContext.Pipe.Send(ex.Message.ToString());

}

returnval = feedData;

}

[Microsoft.SqlServer.Server.SqlProcedure]

public static void SampleWSPost(SqlString weburl, out SqlString returnval)

{

string url = Convert.ToString(weburl);

string feedData = string.Empty;

try

{

HttpWebRequest request = null;

HttpWebResponse response = null;

Stream stream = null;

StreamReader streamReader = null;

request = (HttpWebRequest)WebRequest.Create(url);

request.Method = "POST";

response = (HttpWebResponse)request.GetResponse();

stream = response.GetResponseStream();

streamReader = new StreamReader(stream);

feedData = streamReader.ReadToEnd();

response.Close();

stream.Dispose();

streamReader.Dispose();

}

catch (Exception ex)

{

SqlContext.Pipe.Send(ex.Message.ToString());

}

returnval = feedData;

}

};

The above code depicts that we have used HttpWebRequest class object to create an URI based request, and later we have defined the type of method to make this URI calls, that is ‘PUT’ and ‘POST’. You can change this method based on your payload request.

When the above code will be compiled in assembly and later registered as an assembly in SQL Server, we’ll create two CLR stored procedures to bind assembly stored procedure methods with them.

Generate Assembly and Register in SQL Server

The code here guide you to create assembly of class and later register this assembly in SQL Server

¾ Generate Assembly

// Creates StoredProcedures.dll

csc /t:library StoredProcedures.cs

¾ Register Assembly in SQL Server

USE [SampleDB]

CREATE ASSEMBLY SPAssembly

FROM 'C:\SampleCLRSP\StoredProcedures.dll'

WITH PERMISSION_SET = UNSAFE;

GO

Create CLR Stored Procedure

I have put the T-SQL scripts here to create your CLR stored procedure to make call to any REST web service for PUT and POST payload

USE [SampleDB]

/****** Object: StoredProcedure [dbo].[SampleWSPut] Script Date: 06/18/2008 06:02:32 ******/

CREATE PROCEDURE [dbo].[SampleWSPut]

@weburl [nvarchar](4000),

@returnval [nvarchar](2000) OUTPUT

WITH EXECUTE AS CALLER

EXTERNAL NAME [SPAssembly].[StoredProcedures].[SampleWSPut]

/****** Object: StoredProcedure [dbo].[SampleWSPost] Script Date: 06/18/2008 06:02:19 ******/

CREATE PROCEDURE [dbo].[SampleWSPost]

@weburl [nvarchar](4000),

@returnval [nvarchar](2000) OUTPUT

WITH EXECUTE AS CALLER

EXTERNAL NAME [SPAssembly].[StoredProcedures].[SampleWSPost]

Calling CLR stored procedures

Below example shows that how can you make call to REST based web services using CLR stored procedures. CLR stored procedures written by us make call to REST web service and collect the response send by them, response could be a simple HTTP response message, for example 100, 200, 401 etc. or it could be full XML or MIME message returned to your application.

Declare @Response NVARCHAR(2000)

EXECUTE SampleWSPOST 'http://sampledev03/wcfweb/calendarWS.svc/?appid=MMM&AuctionCode=ABC&months=4',@Response OUT

SELECT @Response

Declare @Response NVARCHAR(2000)

EXECUTE SampleWSPUT 'http://sampledev03/wcfweb/calendarWS.svc/?appid=MMM&AuctionCode=ABC&months=4',@Response OUT

SELECT @Response

Hope this article will help you to create CLR stored procedures to make REST based web service calls.

Thanks,

Kuldeep

Deprecated types are not supported as OUTPUT parameters

mrkuldeep — Tue, 25 Mar 2008 18:13:00 GMT

During working on one simple stored procedure I came to one interesting scenario. If we are developing a stored procedure in SQL Server 2005 that take some input or output parameters and if those parameters are of deprecated data types like Text, NText and Image then you are going to get an error when you will pass parameters to this stored procedure using .NET codes (using System.Data.SqlClient) . The error message will look like what you can see in below screen shot.

The only way to fix this error is to use new SqlDbType data types (VarChar, NVarChar and VarBinary) while passing parameters to stored procedure through .NET app.

Attached is a small sample to test this out.

Testing Steps

1. Create stored procedure in your database (see attached stored_proc.sql)

2. Open the attached solution ‘ConsoleApplication1.sln

3. Open Program.cs

4. Closely look at codes.

5. Change the data source and InitialCatlog to your server name and database name.

6. For @out_image parameter the SqlDbType is set to ‘Image’

7. Now run the codes and see what error you get.

8. Now change the SqlDbType of @outimage to ‘VarBinary’

9. Execute the codes and see now everything is fine.

Thanks,

Kuldeep Chauhan (MSFT)

Play with SSIS Flat File Source Error Flow Output

mrkuldeep — Tue, 01 Jan 2008 05:17:00 GMT

In general as we know that SSIS tasks error flow redirection of data through error pipe sends erroneous data in form of rows with multiple columns. For an example, if we use a lookup task in our data flow, any failing lookups can be redirected to a table or file, the data that comes through error flow pipe is always in form of rows with individual column values, an example could be EmpID, Ename, Designation, MGRID. However, there is a slight difference in this behavior if we talk about flat file sources. When flat file source finds a erroneous row, it is indeed transferred to error pipe but this row comes as one column, this means all [EmpID, Ename, Designation and MGRID] will be visible as a single column as opposed to 4 columns. So if you have a requirement where you need to pick individual column value for future error data analysis purpose, you can’t straight forward do this.

There are two ways to pick up individual columns from flat file source error flow

1. By using String functions over erroneous row (i.e. FindString and SubString).

2. By using a Script task to pick our individual columns from row.

Taking about option 1, yes you can find a column in a row using FindString and SubString but if it comes to taking multiple columns out from an error row, it’s definitely a tedious job. I indeed advise to use Script task in that case. See in below screenshot that how only ‘Flat File Source Error Output Column’ is displayed as opposed to multiple columns like EmpID, Ename, Designation and MGRID

As you see in screenshot we have to play with ‘Flat File Source Error Output Column’, however this column has DT_Text as default datatype, you need to first convert datatype of this column to DT_STR to use any string functions, in the code sample provided in this blog, you will find how conversion task is used in conjunction with derived column task to derive required columns.

As said in Option 2, we can use Script task for taking out individual columns out from a row. This approach is pretty handy, as it gives you power of .NET functions to play with your data. The point to note here is that data from ‘Flat File Source Error Output Column’ is passed to script task in the form of BLOB data, which means before using any string functions to pick your columns, it is essential to convert error output column to text data and then do the next step.

I have attached a sample demo to show how you can take individual columns out using any of two approaches.

Sample Code: 'FlatFileErrorFlow.zip'

Code Guidelines:

1. Download ‘FlatFileErrorFlow.zip’ file, unzip it in a folder.

2. Run ‘CreatTableScripts.sql’ in any of your test database, if you don’t have any test database, just create one.

3. Open the solution file under \FlatFileErrorFlowDemo\FlatFileErrorFlow\FlatFileErrorFlow.sln

4. Open package FlatFileErrorFlow_StringFunction.dtsx and FlatFileErrorFlow_Script.dtsx and change the ‘Test’ OLEDB connection manager to point to your database used in step2, also change the ‘InputFileConnection’ flat file connection manger to point to the folder where ‘Inputfile.txt’ is present. This text file is present in your \FlatFileErrorFlowDemo folder.

5. Before running the package, analyze the table schema and Inputfile.txt, we are passing 4 column values in each row i.e. EmpID, EmpName, Designation and MGRID but in Inputfile.txt there are two rows with more than 4 columns i.e. they are erroneous rows and should be re-directed to a table called SuspectData.

6. We have used data viewers to help you see that error row by default come as a single column and further that has been broken into relevant columns to pass desired column data to underlying tables.

Hope this will help….!

Kuldeepc

MSFT

Now import and export your audio/video data in SQL Server 2005

mrkuldeep — Mon, 15 Oct 2007 03:50:00 GMT

As we all know that we can store different format files in SQL Server in the form of data rows but most of the time it is hard for us to find a working demo on how to store audio/video file in SQL Server 2005 database.

I have created a demo .NET app that we’ll use during our demo to import and export files from SQL Server 2005. In fact we’ll use .NET FileStream & Byte arrays in our sample app to see how import and export of file happens in SQL Server 2005.

For storing files in form of data in SQL Server 2005, we’ll use VarBinary datatype columns.

Here is the code and code guidelines that you should follow to run this demo

Code - Refer to SQL_File_Import_Export.Zip

Code Guidelines –

1. First, create a table MyBlobTable in your database,
Create Table MyBlobTable (id int identity(1,1), blob_description varchar(100),
blob_data varbinary (max))

2. Open SQL2005BLOB.sln solution file in .NET IDE.

2.Now change connection string in sConnectionString variable,
change it to point to your local SQL Server/DB.

2. In ImportBLOB() sub routine, change FS (File Stream) object path to point
to your local media file.

3. In RetrieveBLOB() sub routine, change MediaFile object path to your
desired output path, this will be the path where your media file will be
generated from your varbinary column data.

4. As you know from step 1 that every entry will update identity column
value, therefore you need to update your query statement with id column value
accordingly (
Dim da As New SqlDataAdapter _
("Select blob_data From MyBLOBTable where id = 4", cn)
)

-Kuldeep Chauhan

Upcoming SQL Server 2005 webcasts

mrkuldeep — Sat, 13 Oct 2007 01:18:00 GMT

Upcoming webcasts on SQL Server 2005. Don't forget to attend them....!

Webcast Schedule - October 2007

Attend a series of LIVE Webcasts this October by Microsoft experts on Infrastructure Architecture - Case Study Review, SQL Server 2005 - Basics, SQL Server 2005: Database Maintenance, Monitoring SQL Server 2005 Performance and get on to the edge of the steep-learning curve.

Register Now!

Date	Topic
October 15, 2007 (4:00 pm - 5:30 pm)	Infrastructure Architecture - Case Study Review Infrastructure Architecture - Case Study Review. Speaker: Ramnish Singh
October 17, 2007 (2:00 pm - 3:00 pm)	SQL Server 2005 - Basics This session we will take a look at the various SQL Server 2005 Editions available in the market. We will then take a dive into the various sub-components introduced with SQL Server 2005. The session will build the foundation for getting started with SQL Server 2005. Speaker: Vinod Kumar
October 18, 2007 (4:00 pm - 5:30 pm)	SQL Server 2005: Database Maintenance Although many of the concepts of database maintenance in a SQL Server 2005 Environment remain the same as with previous SQL releases, there are several new features. Database snapshots provide a way to revert to a point of consistency, while partial restores allows users to access unaffected files and pages. You can now continue the backup and restore operations after a failure and configure fast recovery during crash recovering and mirroring failover. It is now possible to verify the integrity of data pages during backup and restore operations. Come, hear about all of these and see some of these features in action with me! Speaker: L Srividya
October 19, 2007 (4:00 pm - 5:30 pm)	Monitoring SQL Server 2005 Performance A number of tools are available to inspect and record activity in a SQL Server installation. Some are built into SQL Server and some are built into the Operating System. Some of these tools include the System monitor, Data Tuning Advisor, Activity Monitor, etc. This session takes you through a few of these tools for troubleshooting and optimizing your SQL Server performance. Speaker: L Srividya

Register Now!

Please Note:

Each Webcast / Live Meeting Session has a Maximum Capacity of 200 participants. You are advised to join in the Live Meeting Session 15 minutes prior to the scheduled time to ensure your participation
You need to register for each Webcast that you intend to attend. Registering for one Webcast does not register you for all webcasts in the series.
This meeting will broadcast internet audio directly to your computer. Please ensure that you have Windows Media Player 9 or higher installed.
First-time users: Click here to install the Windows-based Live Meeting Console before your meeting.
You need to register for each Webcast that intend attend. Registering one does not all webcasts in the series.

©2007 Microsoft Corporation. All rights reserved.
If you prefer not to receive future promotional e-mails of this type from Microsoft, please send an email at erase@microsoft.com. We will promptly update your preferences; however, you may still receive previously initiated promotional communications from Microsoft.

Kuldeep Chauhan (MSFT)

SQL Server 2008 (Formerly code name Katmai) June CTP available now

mrkuldeep — Tue, 05 Jun 2007 08:39:00 GMT

The most awaited CTP of SQL Server 2008 (Formerly code name Katmai) has been released on 06/04/2007.

You can download it from http://connect.microsoft.com/sqlserver. You can also watch Ted Kummert talking about SQL Server 2008 here

Some of the key improvements in this CTP are:

§ Table Value Parameters

§ Star Join Query Optimizations

§ Change Data Capture (CDC)

§ MERGE SQL Statement

§ Declarative Management Framework (DMF)

§ VSTA Support for the Script Task and Component

§ AS Dimension Design: Improved Supportability and Integration of Best Practices

Kuldeep Chauhan

MSFT

SQL Server Integration Services (SSIS) 10 Quick Best Practices

mrkuldeep — Mon, 21 May 2007 16:44:00 GMT

Here are the 10 SSIS best practices that would be good to follow during any SSIS package development

§ The most desired feature in SSIS packages development is re-usability. In other ways, we can call them as standard packages that can be re-used during different ETL component development. In SSIS, this can be easily achieved using template features. SSIS template packages are the re-usable packages that one can use in any SSIS project at any number of times. To know more about how to configure this, please see http://support.microsoft.com/kb/908018

§ Avoid using dot (.) naming convention for your package names. Dot (.) naming convention sometime confuses with the SQL Server object naming convention and hence should be avoided. Good approach would be to use underscore (_) instead of using dot. Also make sure that package names should not exceed 100 characters. During package deployment in SQL Server type mode, it is noticed that any character over 100 are automatically removed from package name. This might result your SSIS package failure during runtime, especially when you are using ‘Execute Package Tasks’ in your package.

§ The flow of data from upstream to downstream in a package is a memory intensive task, at most of the steps and component level we have to carefully check and make sure that any unnecessary columns are not passed to downstream. This helps in avoiding extra execution time overhead of package and in turn improves overall performance of package execution.

§ While configuring any OLEDB connection manager as a source, avoid using ‘Table or view’ as data access mode, this is similar to ‘SELECT * FROM <TABLE_NAME>, and as most of us know, SELECT * is our enemy, it takes all the columns in account including those which are not even required. Always try to use ‘SQL command’ data access mode and only include required column names in your SELECT T-SQL statement. In this way you can block passing unnecessary columns to downstream.

§ In your Data Flow Tasks, use Flat File connection manager very carefully, creating Flat File connection manager with default setting will use data type string [DT_STR] as a default for all the column values. This always might not be a right option because you might have some numeric, integer or Boolean columns in your source, passing them as a string to downstream would take unnecessary memory space and may cause some error at the later stages of package execution.

§ Sorting of data is a time consuming operation, in SSIS you can sort data coming from upstream using ‘Sort’ transformation, however this is a memory intensive task and sometime result in degrade in overall package execution performance. As a best practice, at most of the places where we know that data is coming from SQL Server database tables, it’s better to perform the sorting operation at the database level where sorting can be performed within the query. This is in fact good because SQL Server database sorting is much refined and happens at SQL Server level. This in turn sometime results overall performance improvement in package execution.

§ During SSIS packages development, most of the time one has to share his package with other team members or one has to deploy same package to any other dev, UAT or production systems. One thing that a developer has to make sure is to use correct package protection level. If someone goes with the default package protection level ‘EncryptSenstiveWithUserKey’ then same package might not execute as expected in other environments because package was encrypted with user’s personal key. To make package execution smooth across environment, one has to first understand the package protection level property behaviour, please see http://msdn2.microsoft.com/en-us/library/microsoft.sqlserver.dts.runtime.dtsprotectionlevel.aspx . In general, to avoid most of the package deployment error from one system to another system, set package protection level to ‘DontSaveSenstive’.

§ It’s a best practice to take use of Sequence containers in SSIS packages to group different components at ‘Control Flow’ level. This offers a rich set of facilities

o Provides a scope for variables that a group of related tasks and containers can use

o Provides facility to manage properties of multiple tasks by setting property at Sequence container level

o Provide facility to set transaction isolation level at Sequence container level.

For more information on Sequence containers, please see http://msdn2.microsoft.com/en-us/library/ms139855.aspx.

§ If you are designing an ETL solution for a small, medium or large enterprise business need, it’s always good to have a feature of restarting failed packages from the point of failure. SSIS have an out of the box feature called ‘Checkpoint’ to support restart of failed packages from the point of failure. However, you have to configure the checkpoint feature at the package level. For more information, please see http://msdn2.microsoft.com/en-us/library/ms140226.aspx.

§ Execute SQL Task is our best friend in SSIS; we can use this to run a single or multiple SQL statement at a time. The beauty of this component is that it can return results in different ways e.g. single row, full result set and XML. You can create different type of connection using this component like OLEDB, ODBC, ADO, ADO.NET and SQL Mobile type etc. I prefer to use this component most of the time with my FOR Each Loop container to define iteration loop on the basis of result returned by Execute SQL Task. For more information, please see http://msdn2.microsoft.com/en-us/library/ms141003.aspx & http://www.sqlis.com/58.aspx.

Kuldeep Chauhan (Kuldeepc)

MSFT