Microsoft Codename "Data Transfer" Lab

Thanks for using the Data Transfer Lab

shoshe — Thu, 07 Jun 2012 05:26:00 GMT

When we first envisioned the SQL Azure Labs site, we imagined a way to give our customers and partners access to prototype ideas that we've been kicking around. These services aren't production quality and we never intended to keep them around. Our objective was simply to start a discussion with you to determine what you find valuable – not through theory or discussion, but through concrete services that you could test and experience.

Today we announced, after seven months, that we have retired Microsoft Codename “Data Transfer" Lab. I want to thank all of you who used the lab and provided feedback through the SQL Azure Labs Support Forum on MSDN, through pop-up surveys in the tool and through your comments emailed to the team. Here's what we learned:

50% needed one-time transfers

25% needed scheduled transfers

40% were transferring data to custom application

30% were transferring data to use SQL Azure Reporting

59% wanted to import XML

And, here are some of the top suggestions we received from you:

Need to allow exporting SQL Azure data into CSV/Excel files
Need to enable upload of zipped BCP csv files
Ignore empty rows in files
Allow uploading all files in a local directory that match a pattern

We would like to thank every one of you who participated for your valuable feedback!

Shoshanna Budzianowski and the “Data Transfer” Labs team

Email: datatransfer@microsoft.com

Twitter: @SQLAzureLabs #sqlazurelabs #datatransfer

Blog: http://blogs.msdn.com/b/data_transfer

Support added for Cloud Numerics format

AdamHurwitz — Mon, 12 Mar 2012 19:53:00 GMT

Microsoft Codename "Cloud Numerics" is a SQL Azure Lab that lets you model and analyze data at scale. Now when you want to upload a file with Microsoft Codename "Data Transfer" to Windows Azure Blob storage for use in Cloud Numerics, you can choose to have the file converted to the Numerics Binary Format. This only applies to CSV and Excel files that contain numerical data ready for analysis with Cloud Numerics.

When uploading to blob you will be presented with a choice of output format.

When you select Numerics Binary Format, you will receive additional options regarding the file that you are uploading.

You can run a model in Windows Azure after the data transfer of the Numerics Binary Format file completes. Here is a C# example of a model that computes Eigen values using Cloud Numerics:

public static void Main(string[] args)

{

// Step 1: Initialize the Microsoft.Numerics runtime to create

// and operate on Microsoft Numerics distribute arrays

// DO NOT REMOVE THIS LINE

NumericsRuntime.Initialize();

//Setup the Azure values used in the Data Transfer

string account = "tbd"; // Azure Account

string key = "tbd"; // Azure Storage Key

string container = "tbd"; // Azure Container

string file = "tbd"; // Name of Numerics Binary Format File

//Load the Numerics Binary File into a distributed array

var sr = new SequenceReader(account, key, containerName, fileName, 0);

var berlinAdapterResult = Loader.LoadData<double>(sr);

long[] shape1 = berlinAdapterResult.Shape.ToArray();

int ndims1 = berlinAdapterResult.NumberOfDimensions;

Console.WriteLine("Container: {0}, File name: {1}", container, file);

for (int i = 0; i < ndims1; i++)

{

Console.WriteLine("Dimension {0} has length = {1}", i, shape1[i]);

}

//Calculate the Eigen values

var result = Decompositions.EigenValues(berlinAdapterResult);

Console.WriteLine("Eigen values :\n {0}", result.ToString());

// Shutdown the Microsoft.Numerics runtime
NumericsRuntime.Shutdown();

Console.WriteLine("Numeric Binary Format Successfully Read.");

Console.WriteLine("Hit enter to continue ... ");

Console.ReadLine();

}

To learn more about deploying this model, please visit SQL Azure Labs Microsoft Codename “Cloud Numerics”.

New Service Update Released for Microsoft Codename "Data Transfer"

AdamHurwitz — Mon, 06 Feb 2012 22:32:58 GMT

I’m happy to announce that we have just released another update for the Microsoft Codename “Data Transfer” lab.

We listened to you. We received feedback from a lot of users on our feature voting site and through email. We took the top voted feature and made it our focus for this release.

Update and Replace data in SQL Azure

Now when you upload a file to SQL Azure, you will have the choice of importing the data into a new or existing table.

If you choose an existing table, then you will have the choice to Update or Replace the data in the table. Update is an upsert (update or insert depending on whether the primary key is found). Replace does a truncate and insert.

And then we’ve created a really easy-to-use UI for choosing the column mappings between your file and the table schema. With a click you can drag-and-drop the columns, either up and down to change position or to the Unmapped list.

Big Data

We have also done a lot of work on the internals of the service so that it can accept a very large volume of data. This is not something that you can see now because it takes a few of our short releases to surface the functionality. But stay tuned!

Enjoy – and please let us know how it goes and what you need.

Microsoft Codename "Data Transfer" Lab Refresh

shoshe — Sat, 17 Dec 2011 00:22:00 GMT

We committed to continually improving Microsoft Codename “Data Transfer” lab. We launched the lab less than a month ago, and we just completed our first major refresh to the service. This refresh addresses a few issues we found and a few issues you found.

We’re more forgiving on blob names, but remember container names still have their own limitations. See: Naming and Referencing Containers, Blobs, and Metadata
You can now set CSV delimiters and qualifiers
Field type guessing does a better job. We analyze as much of the file possible in a few short seconds, rather than just deferring to the top 100 rows. If you don’t like our guess, you can still override it in the Table Settings page. You can get to that page by selecting Edit table Defaults on the File selection page:

You now have the option to treat empty string as NULL
And, most importantly, we’re continuing to improve our handling of large files. We’ve made significant improvements upload reliability for files up to 200 MB in size

Thanks for all the feedback you’ve submitted through surveys and feature voting. Keep the feedback coming, and get your vote in. Please feel free to contact us directly at datatransfer@microsoft.com, take the survey, or add your suggestions and issues to the SQL Azure Labs Support Forum. We look forward to hearing from you.

Happy Transferring,

Shoshanna Budzianowski and the “Data Transfer” Labs team

Email: datatransfer@microsoft.com

Twitter: @SQLAzureLabs #sqlazurelabs #datatransfer

Blog: http://blogs.msdn.com/b/data_transfer

How to upload your Excel and CSV data to the Cloud

Liam Cavanagh - MSFT — Thu, 01 Dec 2011 00:16:00 GMT

The following post was originally written on my blog at: http://blog.liamcavanagh.com/2011/11/how-to-upload-your-excel-and-csv-data-to-the-cloud/

As a SQL Azure Program Manager at Microsoft, one of the most common questions we get from customers is how to get their data into the cloud. Just like SQL Server, SQL Azure has import tools like BCP that allow you to build scripts to load your data. I have never been completely happy with these solutions because for many people, they just want a quick and easy way to load their data and don’t want to mess around with complex command line tools or deal with firewall issues associated with connecting on-premises systems to the cloud. That was a primary reason why our group chose to create a SQL Azure Lab called: Microsoft Codename “Data Transfer”.

The purpose of this data transfer service is really simple. To give you an easy way to load your Excel or CSV (Comma Separated) files into SQL Azure. All you need to do is tell us where you want us to load your data to, point us at the file and we will do the rest.

Watch as WMV

A big piece of the work that we do for you is in what we call “Data Type Guessing”. This is where we look at your file and makes some guesses as to the best data-type to use when creating the SQL Azure table. Much of this is based on the years of experience we have gained from the SQL Server Integration Services (SSIS) technology. Although these guesses can be over-ridden by more advanced users, it is our belief that most people will just want to get it up there so that they can continue with their work.

The other capability Microsoft Codename “Data Transfer” provides is the ability to upload files to Windows Azure Blob store. Once again, this is pretty simple. You choose the Windows Azure blob storage account you want us to upload the file to, choose the file and you are done. From here, we upload the file and provide you with a private link that allows you to view, share or use that file.

I hope you will give this Data Transfer service a try and let me know what you think. Do you like the idea? Are there other sources, destinations or new features you would like us to add? I can be contacted by email at: Liam <dot> Cavanagh AT Microsoft <dot> com. Liam

Liam

Announcing Microsoft Codename “Data Transfer” Lab

shoshe — Wed, 30 Nov 2011 17:02:37 GMT

Today we are announcing the release of the Microsoft Codename “Data Transfer” Lab. Microsoft Codename "Data Transfer" is an easy-to-use Web application to import your data into SQL Azure or Blob storage. Whether you’re:

preparing data for cloud analytics
uploading files to power your Azure service or Web application
propagating your data to SQL Azure for publication through Windows Azure Marketplace
or simply testing the Azure platform

Importing your data into Azure is the first step. With Microsoft Codename “Data Transfer” Lab you can now transfer your data into Azure using any standard modern browser. The data transfer service transports files up to 200 MB in size through an HTTP SSL connection. You’ll need to supply your Windows Live ID credentials and Windows Azure account information to the service. If you don’t have an account, click here to sign up for a free trial.

One of the goals we have for this lab is to take the complexity out of typical operations, such as loading structured data in CSV and Excel files into SQL Azure. In the interest of optimizing your time and resources, we’re providing simple parsing, type discovery and conversion over your files. You can choose to keep the defaults we provide, or override them. The transfer service first analyzes the file and presents you with our best guess. You’ll be able to change the table name, column names, and data types before saving the file to SQL Azure.

We’ll be evolving this lab over the next few months. We plan to support larger file sizes, multi-file transfer, simple conversions for more types of files into SQL Azure, and reliable transfer that handles dropped connections. If you have feedback or suggestions, please feel free to contact us directly at datatransfer@microsoft.com, take the survey, or add your suggestions and issues to the SQL Azure Labs Support Forum. We look forward to hearing from you.

Happy Transferring,

Shoshanna Budzianowski and the “Data Transfer” Labs team

Email: datatransfer@microsoft.com

Twitter: @SQLAzureLabs #sqlazurelabs #datatransfer

Blog: http://blogs.msdn.com/b/data_transfer