Carpe Datum

Application Lifecycle Management Overview for Windows Azure

2012-02-07T15:58:39Z

Developing in Windows Azure is at once not that much different from what you’re familiar with in on-premises systems, and different in significant ways. Because of these differences, developers often ask about the specific process to develop and deploy a Windows Azure application - more formally called an Application Lifecycle Management, or ALM.

There are specific resources you can use to learn more about various parts of ALM - I’ve referenced those at the end of this post. But ALM has multiple definitions, from the governance of code injection, domain upgrade, testing, process flow and more. Many developers are interested in the finer-grained information, like how do I develop and deploy an application? What tools do I need, and how do I get the code running somewhere that I can test?

I’ll cover the very high-level process here, and refer you to specifics at the end of each section, so that you can take it all in at one viewing, and then bookmark for more detail when you need more information. I won’t be covering processes like Continuous Integration or Agile and other methodologies in this post - I’ll blog those later.

Initial Development

You start with writing code. You have three ways to do this. You can use Visual Studio (even the Express Edition Works), Eclipse, or by leveraging the REST API format. You can do this in a standalone (non-connected) environment like your laptop.

Using Visual Studio is one of the simplest methods to create an Azure application, allowing you to combine the Azure components you want to leverage (Storage, Compute, SQL Azure, the Service Bus, etc.) along with the on-premises code you have now or are creating. Once you’ve installed and patched Visual Studio, just download and install the Windows Azure Software Development Kit (SDK) and you’ll have not only all the API’s you need to talk to Azure, but a fully functioning local environment to run and test your code before you deploy it. You’ll also get a robust set of samples. You can download what you need for all of that (free) here: http://www.windowsazure.com/en-us/develop/downloads/ . There’s a step-by-step process here: http://msdn.microsoft.com/en-us/magazine/ee336122.aspx

You can also use Eclipse to develop for Windows Azure. You won’t get the full runtime environment in just that kit alone, but you can use this successfully on a Linux system. I have several folks using this method. The downloads and documentation for that is here: http://www.windowsazure4e.org/

You can use REST API’s to hit Azure Assets and control them. Not my preferred method, but possible. There are REST API’s for various sections of Azure. You can find the main reference for that here: http://msdn.microsoft.com/en-us/library/windowsazure/ff800682.aspx

Note: We recently demonstrated using a Cloud-based Integrated Development Environment (IDE) for Node.js deployment to Windows Azure. More on that here: http://www.readwriteweb.com/cloud/2012/01/cloud9-ide-to-enable-nodejs-ap.php

Deploying to a Test Instance

After you write the code, you’ll need to test it somewhere. The Azure Emulator on your development laptop is for a single user on that laptop, and it also has some subtle differences from the production fabric as you might imagine. Normally you’ll set up a small subscription to run and test the application, just like you would have a set of test servers. Each subscription has its own management keys and certificates, so this assists in keeping the testing environment separate for billing and control.

More on that general information here: http://msdn.microsoft.com/en-us/library/ff803362.aspx

Deploying to Production

Once you have developed the code and tested it, you need to move it to a location where users can access it. In reality, there is no physical difference in the type of machines, fabric or any other component in “Production” Windows Azure accounts and the “Test” accounts, but you’ll most often pick smaller systems to deploy on in testing, and you’ll probably keep the URL in the plain format.

In the Production Windows Azure account, the team normally limits the access to the account for deployment to a separate set of developers. This ensures code flow and control. A DNS name is normally mapped to the longer, Microsoft-generated URL so that your users access the application or data the way you want them to.

More on setting up an account here: http://techinch.com/2010/06/14/setup-your-windows-azure-account/

Managing Code Change

With the application deployed, there are two broad tasks you need to consider. One is managing changes through the application, and the other involves management, monitoring and performance tuning for an application.

To make a code change, the standard ALM process is followed, just as above. You can use command-line tools to automate the process as you would with an on-premises system. A vide on that shows you how: http://www.microsoftpdc.com/2009/SVC25. Normally this is used with an “In-Place” upgrade into Production Account, since your testing is completed in a separate account. More on that process here: http://msdn.microsoft.com/en-us/library/windowsazure/ee517255.aspx

One difference is the “VIP Swap” process you can use for the final push to Production. In essence, this allows you to have two copies of the application running on the Production account, with a quick way to cut over and back when you’re ready. The process for that is detailed here: http://msdn.microsoft.com/en-us/library/windowsazure/ee517253.aspx

For monitoring, you have several options. You should enable the Windows Azure Diagnostics in your code - more on that here: http://archive.msdn.microsoft.com/WADiagnostics.

You can observe uptime and other information on the Windows Azure Service Dashboard, where you can also consume the uptime as an RSS feed: http://www.windowsazure.com/en-us/support/service-dashboard/

From there, you can also use System Center to monitor not only Windows Azure deployments but internal applications as well. The Management Pack and documentation for that is here: http://www.microsoft.com/download/en/details.aspx?id=11324.

There are also 3rd-party tools to manage Windows Azure. More on that here: http://www.bing.com/search?q=monitor+Windows+Azure&form=OSDSRC

Other References:

There is a lot more detail in this official reference: https://www.windowsazure.com/en-us/develop/net/fundamentals/deploying-applications/

Bryan Group explains the ramifications of the Secure Development Lifecycle (SDL) with lots of collateral you can review: http://blogs.msdn.com/b/bryang/archive/2011/04/26/applying-the-sdl-to-windows-azure.aspx

Book Review (Book 8) - The Elements of Persuasion

2012-01-30T15:57:23Z

This is a continuation of the books I challenged myself to read to help my career - one a month, for year. You can read my first book review here. The book I chose for January 2012 was: The Elements of Persuasion by Richard Maxwell and Robert Dickman.

Why I chose this Book:

As I mentioned in a previous review, I think good storytelling is an essential part of any career. Communication is basic in not only our professional but personal lives, and everyone I’ve met responds well to stories, from children to executive audiences. Not only that, learning to tell a story helps you formulate concepts about the topic, which is yet another way of learning.

I heard about this book from a couple of folks, and it landed within search of “storytelling” and “business”. Whenever I just search for “Storytelling” I either end up with lists of stories (which is fine) or lists of children’s books on storytelling (which is also fine) but neither of these are quite what I’m looking for.

A quick search on Amazon and I located the book, and then a quick check of my various e-library offerings and I downloaded it to my laptop for reading.

What I learned:

This is a “selling” book, but not like you might think. It’s not a book of a quick sale like at a car-lot or a “quick-sale” environment. It’s more along the lines at the executive level and longer-term sales - those involve stories as well.

Sadly, this is another “business book” - the kind I normally don’t like much. There are typical case-study layouts with lots of examples, but in my mind not enough didactic information to actually help you develop a good story-telling mantra.

Even so, I learned some interesting things about the process these authors use. Some of the case studies are interesting, and I did pull out that a story should work towards a single, defining sentence. This isn’t unique to this book, but it is a reinforcement of what I’ve learned elsewhere. Although nothing to do with storytelling, I did like the reference to Lockheed’s “14 Rules”, which I hadn’t read before. They also break down the storytelling process into five elements, which is actually covered better (in my mind) in a book called “20 Master Plots” , which may actually be the storytelling book I’ve been searching for.

Or perhaps I should just write the one I’m looking for.

At any rate, not sure I would recommend this book to others - perhaps as a check-out, but not a purchase, at least if this is for the same reason I looked it up.

Raw Notes:

As I read, I take notes - it’s called “reading with a pencil”. These are the notes I made to myself, in no particular order and with no context other than the book itself:

Stories are interesting to us all.
Describes five elements in a story, but in fact this is for only one type of story. Other books describes more story types.
Very standard business book, but there are good tips in some of the chapters.
Explained how to connect with the audience, good points Spends a lot of time referring to other books The book of five rings Work towards a single, memorable sentence.
Changes partway through into stories about stories. This is better.
A mix of storytelling and sales, although this was touted for sales, feels much more like selling than storytelling, advertisements.
Interesting story about memory championships, where contestants memorize cards. They use unusual stories.
Look up Lockheed and the 14 rules

Team Foundation Server (TFS) in the Cloud - My Experience So Far

2012-01-24T13:45:13Z

I recently joined a software development project that involves not only myself and other internal Microsoft employees, but a partner and a customer as well. We are building a hybrid solution that uses assets on premises as well as Windows Azure for processing. When we put the team together we picked a methodology (Agile) for the project (we use multiple methodologies at Microsoft - whatever the project needs) and then we started talking about Source Control.

We’re all comfortable with various tools for check-in-check-out, branching, and so on. We have all used GIT, SVN, and TFS. Some of us have even used Source Safe in past, but that’s another post. Each company has a full set of Source Control systems in place. But using each other’s systems requires logins, firewalls and the like - so we decided to use the TFS Service Preview to run the entire project from “the cloud”. Here are my experiences with that.

The process was really simple. In fact, we talked about using the cloud TFS in the first SCRUM, and the team was working from the Work Items list that afternoon. The original account login provides a web interface to allow people to join the team. Each of us happened to have a Live.Com address, so we just invited those addresses to join and they got a link, like this:

projectname.tfspreview.com

I’m using Visual Studio, and it’s a requirement for TFS preview to have SP1 installed, and this patch: KB2581206

From there, I opened Visual Studio and navigated from the main menu to Team and then Connect to Team Foundation Server. I’m given this menu:

Selecting port 443 and HTTPS (for security) and then ensuring the lower link has the “tfs” appended as the location, I opened the project.

(This VSTS screenshot is of a project I did in my University of Washington class I teach - I never show client code or names in a blog post)

From there it’s a normal set of operations. Right now the preview doesn’t have some things I’d really like, such as an automated build or some of the testing tools, but you can read this blog entry to learn more about the entire sign-up process, and what the team has planned.

Each day I log in to the project, and I’m given this new sign-in option:

I click the option, and I open the environment, hit My Work Items query, and get to work. All in all, a seamless - although basic - experience. The speed at which we could set up and work on a project was really sweet. It’s remarkable how un-remarkable this is - I just do my work each day, everything is running and backed up in the cloud. I think that’s the point.

Bug-Out Bags and Cloud Architecture Considerations

2012-01-20T18:00:58Z

I served in the U.S. Military for a while, and as part of my training we had to maintain a “Bug-Out Bag”, which was a large duffle-bag full of certain items that we could live on/fight with in an emergency. I’ve carried the spirit of that idea forward with me into civilian life, in Florida and especially here in the Pacific Northwest.

In Florida we dealt with the threat of hurricanes - I went through four of those in one year that hit my area. You’re without power, it floods quickly, and it gets wicked hot. You roof might be gone, whatever. Here in the Pacific Northwest, I live near one of the largest volcano's in the world, we have flooding, and recently we were hit with an ice-storm. Now I’ve lived all over the world, from Alaska to North Dakota and even near the Kamchatka Peninsula in Russia, and I can handle the snow. But ice - that’s a toughie no matter where you live. We had so much that it split my little pine tree in front of the house in half.

We lost power - although I think the folks at Puget Sound Energy did an amazing job at getting us back up in less than 24 hours, but we weren’t worried anyway. That bug-out bag mentality carried forward to a “second pantry” we keep in the garage.

We have a large plastic box (that will fit in the back of the Subaru) with dried goods like pasta, and canned goods and even a little cook stove. We have 25 gallons of clean water in Jerry-Cans. We have batteries, candles and matches. And we have flashlights around every door. We use supplies from the “pantry” to fill our house pantry, and then refill the emergency one from the grocery store. That way everything is fresh, rotated, and we can “bug-out” here at home or on the road.

So what does this have to do with Distributed Computing Architectures?

It’s the thought process. In both the military and civilian life, I’ve done a few things:

Sat down and thought carefully about exactly what I need. Did I include a can-opener? A small shovel to dig out of whatever I got stuck in? Then I weed out what I *really* don’t need.
Put those things into a small, manageable container.
Tried them - even when (especially when) I didn’t have an emergency
Tweaked the process to see what I could do better.

Have you done this when you moved an app to the “cloud”? Each of these has a computing parallel - do you know what you would do if you couldn’t access the Distributed Computing Environment?

I’ve found these thoughts are actually a great place to start - keeps the process simplified from the start, and gives you a sense of assurance when you’re asked if you can recover from an emergency.

Cloud Computing In Action: How I work with Live Mesh, SkyDrive, and Office Live Workspace

2012-01-16T15:27:44Z

Recently I had an tweetversation with a couple of friends on some confusion around two of our products: SkyDrive and Live Mesh. Like most of our software, there’s no single way to do things. That can be a strength or it can cause that confusion. They asked if I would blog how I work with these two products, and what advantages there are to this way of working.

Before I start - this is specific to these two Microsoft products. If you’re a fanboi of another product, that’s great. Awesome. Go for it. You don’t have to use these. There’s no law about it or anything. It’s all good. I use the products you see below because I evaluated lots of them, and these work the best for me - not because I work at Microsoft. But do what makes you happy.

Let’s start with what each of these products do. Live Mesh synchronizes files to various locations. You can create a file on one PC, save it, and then when you fire up another PC that file will be copied from the original location. It’s a mirror of the file, and it exists in both places. You can change the file on the second location, and it will be copied back to the other system, stepping right on top of it.

SkyDrive is a storage system. You can store lots of data in there - larger than most of the other free offerings.

Office Live Workspaces allows you to integrate SkyDrive into your local copy of Microsoft Office, so that you can create, save and edit a document and it will be stored in SkyDrive, and not only that, it will keep a local, synced copy so that you can work offline. But it also has a web-based subset of Microsoft Office. You can create, edit and work with Microsoft Office documents with no software installed at all. From Linux, Mac, a cell phone, whatever has a browser. In fact, we’ve released one of my favorite products, OneNote, in iPhone and iPad flavors, which also buffer down the file as if you had a PC and Microsoft Office.

I rely on these each of these products every day. Here’s how I use them.

I use Live Mesh to copy my entire “Data” directory - files, music, everything - from my home “server” to my work and other systems. Since SkyDrive has a limit, I only send certain files to SkyDrive using Mesh. Just the ones I need access to from non Microsoft-OS devices. Of course, this means I have to leave my home server turned on - which I do anyway since it’s my media server, web server, TV, etc. But everything else I sync to about four computers running Windows.

For my OneNote files - quickly becoming the center of my universe - and anything else I want to access from anywhere, all the time, I use SkyDrive and Live Office. Here’s how that works.

If it’s an MP3, Visual Studio Code, a training video or whatever my customer needs, I save it in SkyDrive, mark it public, and send them the link. Done. Any device that can render these can access the file over the web. Since I play in a group on Sunday, I even put my music there (I use MuseScore) and then I can pop the music up on my netbook right at the pulpit and leave the paper at home.

For OneNote or other Microsoft Office documents, I create the document first in Office Live. Once the file is open, and before I even type in it, I click the button marked “Open in OneNote” (or Word, or Excel, or whatever) and from them on I have that file linked in the local system, and a shadow copy for working offline. I can also work with that document from the web using my Linux or Apple OS’s if needed. I recently attended a very Microsoft-hostile environment, so everything from the presentation to the code review for Windows Azure I did from Live Office and my SkyDrive, all from my Linux Laptop.

As I’ve always said - use what works. This arrangement gives me the ultimate flexibility. I have my data from Live Mesh synchronized on multiple systems. More than once I’ve deleted something I needed, or changed something. I simply boot up the other device without being connected to the web, copy the old version off, and then let it connect and sync. I also back up my home server once a week to a set of local drives, so I have offsite and onsite backups. I can work from anywhere I have a browser, or someone that will let me borrow a device. I have all my presentations ready to present from any system, even if mine breaks.

Hopefully this helps - and hopefully it inspires you to write a blog entry on how you use your favorite cloud products. There are always multiple ways to do things, and I love to learn.

Stand-Up Cloud Computing

2012-01-09T14:33:28Z

When I was very young, I asked my uncle for career advice. He went silent, thinking for a bit, and then said: “People who work sitting down make more than people who work standing up.” I’m not certain how true that really is, but my career as a technology professional has led me to work in a seated position for most of my life.

Turns out, that’s a bad thing. Although I consider myself pretty fit, eating right, sleeping well and working out several times a week in addition to a morning walk with the family each day, I always look for low-barrier ways to stay healthy. When I first moved to the Pacific Northwest to work for Microsoft, I noticed several folks working at tall desks with no chairs. Some even had treadmills. I chalked it up to the ethos here; and certainly not something I would do.

But this year that changed. I noticed that my back was a little stiffer when I got done with my 12-13 hour days of work. For the last couple of years, I’ve worked from home, so I don’t attend meetings (at least in person) as often or have to walk very far to do almost any part of my job. I start work around 6 in the morning, and sometimes get so focused that I don’t moved for many hours. I read an article on how bad sitting really is, and after further investigation thought I might give one of those stand-up desks a try.

The research led me to believe that you don’t actually have to use a stand-up desk per-se, you can also use an alternate chair or just get up every so often. But I wanted to try this out, and figured that I would be more likely to take a break and sit every hour than I would to remember to stand every hour.

The Before

My office desk is fairly typical, but I do have a decent office chair. That’s after going through probably six or seven chairs in the last few years. I have good lighting, a speakerphone, a web cam and two monitors. I also have the typical flotsam and jetsam of desk clutter, although I’m neater than some.

This arrangement has suited me well since I’ve been working at home. I had something similar in an office environment, although I didn’t always have the option of a decent chair. I didn’t go through the trouble of bringing one of my own in; I just put up with whatever I got, or could “appropriate” from an empty office or conference room.

The Build

My criteria were fairly simple: the experiment had to cost less than 100.00, and be at the proper height and size to hold my keyboard, trackball, phone and monitors so that I could type with my elbows staying at a 90 degree angle.

After researching standing desks, 100.00 was going to be impossible, not even for a used one. I visited several thrift shops in the area (I do that a lot anyway to donate and to buy) and didn’t find anything that worked. Of course, when you’re faced with finding cheap furniture, you naturally turn to the most amazing store on the planet.

Before I left, I measured the top area and height of my desk, and wrote down acceptable measurements based on how I high I stood, the stuff I needed the top to hold, and the distance I needed for my typing to be done at the right height. Measurements in hand, I headed to the store.

I found a coffee table - a really cheap one (19.00) called (oddly) LACK and brought it home to begin the surreal process of assembling something bought at Ikea.

Happily, this was REALLY simple. Four lag bolts hold the legs on, and eight screws punched into the wood (or at least wood-like) to attach the shelf. My original thought was that I would move the shelf up higher than the Ikea instructions, and then use the pull-out tray from my desk to put the keyboard and trackball on. However, that was not to be. On investigation I found that the tray was not hung underneath the desk, but attached at the sides. That meant I had to either buy another tray, or place the keyboard on top, necessitating standing two inches higher.

The After

Researching trays, I found they were terribly expensive. These things used to be everywhere, so I was surprised that they aren’t as easy to get as they once were. Off to the thrift store to see what they had. I found an older tray, but it looked flimsy. I then found a child’s plastic picnic table. The plastic was strong, and 1.5 inches thick. I figured I needed some padding to stand on anyway, so I bought the table, pulled off the legs, and wrapped some padding in an older rug. This brought the total cost of the build to 25.00 and 2.00 for an espresso and a cinnamon bun at Ikea (I would be burning these calories off with my new desk, after all).

I re-routed all my cables, and everything fit correctly.

My Early Conclusions

The first day was easy. I thought - well I should have done this a LONG time ago! My back wasn’t that sore, and I didn’t feel that tired.

Then I woke up the next morning. My feet were sore, although not terribly. The second day, I had to sit down each hour. Not just wanted to sit down - needed to. I play in a group at Church on Sunday, so I put my guitar in the office and spent 5 minutes each hour (roughly - sometimes I have calls that are longer than that) and practice a little sitting down. That helped a lot.

I’ve now been at the desk for four days, and I don’t need the breaks as often. I also find I need to shift around a lot, which of course burns even more calories and is better for me. I honestly think the treadmill desk might be easier than a standing one. We’ll see if I go that far someday.

The verdict so far? Glad I’ve done this. If it doesn’t work out, I’ll just re-purpose the coffee table and go back to sitting - although I’m pretty stubborn and will probably stick with this for a while. I’ll let you know if I change back, and why.

My executive assistant hasn’t changed her office arrangement at all. She still keeps her (Ikea) chair just like she’s had it since she started working with me, and dutifully stays at her workstation for the entire 12-13 hours each day. We do, however, take our lunchtime walk still. She burns her calories that way, and thinks it’s better than just standing around all the time.

Valentine’s Day and Your Career

2012-01-03T14:53:17Z

The new year has begun. It’s traditional to make “Resolutions” at this time, but as I’ve mentioned before, I don’t do that. I make goals instead. I like things to be measurable, and I hold myself accountable to those goals - some of which deal with my professional life.

But you might not buy into all that. Perhaps you’re the kind of person who doesn’t buy Valentine’s day cards, or take your significant other out for dinner on Valentine’s day. After all, it’s a manufactured, made up holiday from the greeting-card companies, right? Somebody just decided to come up with a day to make you do something you don’t normally do.

Here’s a tip: do it anyway. Buy the flowers. Jump into the hype. Yes, it’s a made-up holiday. Yes, they’re making money off of you. But take that person out for the nicest dinner you can find. Treating someone you love in a special way on a periodic date is shown to increase the bond in a relationship, simply because it’s a ritual date that others keep. The ritual is the magic.

What does this have to do with New Year’s, or your career? Everything. Not to burst a bubble here, but the universe is not aware of human timekeeping mechanisms. The New Year is just as artificial as Valentine’s day. In fact, many other cultures don’t even count the 1st of January as the New Year. But it’s OK - just like Valentine’s day, you can use the “start of the new year” as a time to focus on something you need to do.

It’s pretty simple to do this - but of course simple != easy. Goals need to be realistic - so sit down sometime this week, and follow this process:

Write down where you want to be in a year in your career. Make it specific. An award, a position, a company, a raise. Write it down.
Write down a few books you want to read that will help you get there. Blog about these books.
Write down the people you need to talk to, inside your company and out. Send an invite out to these people to chat. Do that this week.
Write down the things you need to accomplish for that goal in your job.
Tell others you are doing these things, and what you expect.
Implement your plan.
Review your plan and adjust as needed each month.

Yes, the “new year” is artificial, like Valentine’s day. So what. Use it to get where you need to go.

Happy New Career.

Book Review (Book 7) - Think Stats

2011-12-30T14:11:17Z

This is a continuation of the books I challenged myself to read to help my career - one a month, for year. You can read my first book review here. The book I chose for December 2011 was: Think Stats, by Allen B. Downey.

Why I chose this Book:

I originally chose another book for this month, but changed to this one after a difference in focus (sort of) in my technical career. That brings up a couple of interesting points right away. The first is that it’s OK to change a list - remember that the purpose of reading these books is to gain information that gets you closer to your professional goals. When you develop your list, you have a certain amount of knowledge, and as you read more, experience more, and are exposed to more, you get different information. When that happens, adapt.

The second point is that your goal itself may change. I am focusing on “Big Data” this year and with the changes we’ve made in Windows and SQL Azure at Microsoft, this fits neatly with my professional goals personally and the company I work for. Actually, my goals in technology haven’t changed in the 27+ years I’ve worked in IT, in roles from electronics, programming, consulting, management, architect and in my current technical role here at Microsoft. I think that it has always been about data - everything in IT is an interface to data. And I have always wanted to be at the center of that. Data Science involves not just the sourcing, administration and movement of data, but in applying scientific (with an emphasis on mathematical) disciplines to get at the meaning the situation needs.

So that brings me to this choice. My friend Jeremiah Peschka found this resource for a role I am VERY interested in - the “Data Scientist”. It’s a combination of high-end mathematics, Data Analysis and Big Data. The resource is a series of books from O’Reilly for that very title. You can find that here.

Personally, I find the grouping of books a little cobbled together. They are all fine books, but I’m not certain how they lead you through the series of knowledge required for the topic, but that’s a post for another day. Within that series of books is the one I’m reviewing today. I started (since there is no implied order in the books) with the “Data Analysis” book, but it seemed to start in the middle of some topics I needed to research, so I switched to reading this one, and chose it as my December book.

Another note here - December is a tough month. Since so many people take vacation time during this month, most of my clients try to get as much work in before the Holidays as possible. Since they are all doing that at once, it makes for a lot of overtime. Also, I travel to see family, which of course puts me out of pocket for a while myself. So staying on track with the books - especially one that makes heavy use of computing, math and focus is hard. So it’s tough to maintain your goals all of the time - but keeping in mind why you do this is the important thing. It will keep you on track.

What I learned:

This book focuses more on what the title says - it’s more about being mindful of the way you use statistics than the statistics themselves. It’s assumed you know not only the basics of statistics (I used these free lessons as a refresher, along with some of my old stats books) but how they are used.The author doesn’t stop to explain a lot of stats he uses, but periodically he does show why a given formula works the way it does. This is very useful, and helps with understanding the point of using one method over another. He also does a great job of using statistics to verify other statistics.

Although it should be obvious, the meaning of the data is essential. We think about this when we deal with the result of data processing, but not necessarily when we work with the sources. For instance - as the author explained some central tendency, smoothing and so on using statistical methods, he introduced some numbers and asks you to guess the central number from the set. Dutifully you work out the answer, but in time he reveals that it’s a series of numbers on a die - which of course can only be whole numbers. The point is that you’re so focused on getting the right answer, you don’t define what the real problem is first.

Another great tool - and a fascinating study that I need to look into further - is the fact that you can often make at least educated inferences into data you might not imagine. For instance, he talks about the example of a series of train cars, numbered sequentially. You see a train car numbered “60” - can you guess with any certainty how many train cars the company has? Fascinating stuff.

He includes a glossary at the end of each chapter. I found this a great approach for summarizing the information in one place, and really helpful in making sure I understood everything before moving on. I didn’t always, so I had to re-read parts of the book and freshen up my stats knowledge along the way as well.

He uses Python as the language of choice - which I found a bit unusual. Most of the stats profession uses something more like the R language, which I’ve also started learning, and one of the other books in this series includes R as a primary subject. Because the author uses Python, he includes references to a series of libraries you add into it to work through the examples. Python certainly is a Data Scientist’s tool, just normally not for statistics. The author uses great examples and assignments, but doesn’t really follow up on those. I guess I’d rather see those introduced earlier in the chapter and explained better. He tends to jump around a bit, and his references are to Wikipedia, which isn’t always as reliable or thorough as it can be. But these are small quibbles. It’s a good book, and a I learned a lot reading it. In fact, I have lots of concepts to unpack based on what I read.

Windows Azure Storage (WAS) Internals - Achieving Consistency

2011-12-13T17:05:50Z

Windows Azure Storage has three primary components - a Queue, a Binary Large Object (BLOB) store (two types of these), and Table Storage.

Storage of data on-premises is fairly well understood - but there components of it that you may not consider. When you move to a distributed architecture, certain factors should be taken into account, such as consistency. Consistency means that when you store a datum it should be available in the same bit format across the calling mechanism. In other words, if you store a picture with a certain name, whenever you call that name that particular picture should show up. That might sound obvious - but when you begin to scale horizontally, it’s a big consideration. Systems are spread out over multiple physical racks, which are further separated into separate “fault domains” each with its own power, networking and so on, and in Windows Azure, the storage is replicated to ensure high-availability.

Some “cloud” systems relax the consistency target to allow for the highest speed throughput. This might allow inconsistent reads, meaning that the datum recorded in the naming system would be available yet, or that it might allow an older version of the datum to be read. In Windows Azure, we took the position that the consistency is of the highest importance. We achieved this through constructs such as the Location Service (LS), Stream, Partition and Front-End layers, and separate replication engines. Of key importance in a system that allows high consistency is in the naming and object access protocols - in fact, these turn out to be some of the most pivotal.

Windows Azure Storage has a complex arrangement to ensure this high consistency. You can read some very deep internals here. And a video of the talk held at an ACM conference is here.

How Microsoft helps you NOT break your Windows Azure Application: Storage Services Versioning

2011-12-06T15:42:57Z

One of the advantages of using Windows Azure to run your code is that you don’t have to constantly manage upgrades on your platform. While that’s a big advantage indeed, it immediately brings up the question - how do the upgrades happen? Microsoft upgrades the Azure platform in periodic increments, and the components that are affected are documented.

This brings up another question - upgrades mean change, and change can sometimes alter the way you might implement a feature. What if you have taken a dependency on some feature in your code that has been altered by an upgrade? Windows Azure does have an Application Lifecycle Management (ALM) Process, which I’ll reference at the end of this post. But beyond that, there are some features we’ve put into place that will help you manage many of these changes. One of those is being able to set the version of storage features you would like your code to use.

Windows Azure is made up of three main component areas: Computing, Storage and a group of features called the Application Fabric. You can use these components together or separately, depending on what you would like your application to do. In this post I’ll deal with the version control in the storage subsystem - in other posts I’ll explain how to track and in some cases control the versions of the other components you work with.

When you send a request to a Windows Azure resource, you’re actually using a REST call. That’s a three-part call to the system that has a request (called a URI), a header, and a body of code you want to send. So a typical call, such as to a table, might look like this example, which changes the properties of a Blob:

URI:
PUT http://myaccount.table.core.windows.net/?restype=service&comp=properties HTTP/1.1

Header:
x-ms-version: 2011-08-18
x-ms-date: Tue, 30 Aug 2011 04:28:19 GMT
Authorization: SharedKey
myaccount:Z1lTLDwtq5o1UYQluucdsXk6/iB7YxEu0m6VofAEkUE=
Host: myaccount.table.core.windows.net

Body:
<?xml version="1.0" encoding="utf-8"?>
<StorageServiceProperties>
    <Logging>
        <Version>1.0</Version>
        <Delete>true</Delete>
        <Read>false</Read>
        <Write>true</Write>
        <RetentionPolicy>
            <Enabled>true</Enabled>
            <Days>7</Days>
        </RetentionPolicy>
    </Logging>
    <Metrics>
        <Version>1.0</Version>
        <Enabled>true</Enabled>
        <IncludeAPIs>false</IncludeAPIs>
        <RetentionPolicy>
            <Enabled>true</Enabled>
            <Days>7</Days>
        </RetentionPolicy>
    </Metrics>
</StorageServiceProperties>

(Source of this code)

You can see that I’ve highlighted a portion of the header block - that’s where you set the version of the Storage Services you would like to use. You can find a list of the features introduced in each version here. It’s not a requirement of adding that element to the header, but it’s best practices to do so.

You don’t have to use REST calls directly, however. It’s more common to use the API in the Software Development Kit to just change the property in your IDE environment - the setting you’re looking for there is the Set Storage Service Properties call.

Interestingly, rather than a breaking change you might run into an unexpected behavior if you are not aware of these parameters. In some code I recently reviewed a newer feature from the storage system failed when it was called. On inspection I found that the developer had used an older codeblock from a previous version of the storage system - he was not aware you can set the version of storage in the call. We changed the header to the latest version, and everything worked as expected.

References:

The Storage Services Versioning and the changes for each version:

http://msdn.microsoft.com/en-us/library/windowsazure/dd894041.aspx

Windows Azure Application Lifecycle Management:

http://msdn.microsoft.com/en-us/library/ff803362.aspx

http://channel9.msdn.com/posts/Windows-Azure-Jump-Start-03-Windows-Azure-Lifecycle-Part-1

http://channel9.msdn.com/Events/TechEd/Australia/Tech-Ed-Australia-2011/COS201

Windows Azure Best Practices: Affinity Groups

2011-11-29T05:43:00Z

When you create a Windows Azure application, you’ll pick a subscription to put it under. This is a billing container - underneath that, you’ll deploy a Hosted Service. That holds the Web and Worker Roles that you’ll deploy for your applications. Along side that, you use the Storage Account to create storage for the application. (In some cases, you might choose to use only storage or Roles - the info here applies anyway)

As you are setting up your environment, you’re asked to pick a “region” where your application will run.

If you choose a Region, you’ll be asked where to put the Roles. You’re given choices like Asia, North America and so on. This is where the hardware that physically runs your code lives. We have lots of fault domains, power considerations and so on to keep that set of datacenters running, but keep in mind that this is where the application lives.

You also get this selection for Storage Accounts. When you make new storage, it’s a best practice to put it where your computing is. This makes the shortest path from the code to the data, and then back out to the user.

One of the selections for the location is “Anywhere U.S.”. This selection might be interpreted to mean that we will bias towards keeping the data and the code together, but that may not be the case. There is a specific abstraction we created for just that purpose: Affinity Groups.

An Affinity Group is simply a name you can use to tie together resources. You can do this in two places - when you’re creating the Hosted Service (shown above) and on it’s own tree item on the left, called “Affinity Groups”. When you select either of those actions, You’re presented with a dialog box that allows you to specify a name, and then the Region that names ties the resources to. Choose a specific region - not one of the "Anywhere" choices.

Now you can select that Affinity Group just as if it were a Region, and your code and data will stay together. That helps with keeping the performance high.

Official Documentation: http://msdn.microsoft.com/en-us/library/windowsazure/hh531560.aspx

Book review (Book 6) - Wikinomics

2011-11-22T15:35:02Z

This is a continuation of the books I challenged myself to read to help my career - one a month, for year. You can read my first book review here. The book I chose for November 2011 was: Wikinomics: How Mass Collaboration Changes Everything, by Don Tapscott

Why I chose this Book:

I’ve heard a lot about this book - was one of the “must read” kind of business books (many of which are very “fluffy”) and supposedly deals with collaborating using technology - so I want to see what it says about collaborative efforts and how I can leverage them.

What I learned:

I really disliked this book. I’ve never been a fan of the latest “business book”, and sadly that’s what this felt like to me. A “business book” is what I call a work that has a fairly simple concept to get across, and then proceeds to use various made-up terms, analogies and other mechanisms to fill hundreds of pages doing it.

This perception is at my own – the book is pretty old, and these things go stale quickly. The author’s general point (at least what I took away from it) was: Open Source is good, proprietary is bad. Collaboration is the hallmark of successful companies. In my mind, you can save yourself the trouble of reading this work if you get these two concepts down.

Don’t get me wrong – open source is awesome, and collaboration is a good thing, especially in places where it fits. But it’s not a panacea as the author seems to indicate. For instance, he continuously uses the example of MySpace to show a “2.0” company, which I think means that you can enter text as well as read it on a web page. All well and good. But we all know what happened to MySpace, and of course he missed the point entirely about this new web environment: low barriers to entry often mean low barriers to exit.

And the open, collaborative company being the best model – well, I think we all know a certain computer company famous for phones and music that is arguably quite successful, and is probably one of the most closed, non-collaborative (at least with its customers) on the planet. So that sort of takes away that argument.

The reality of business is far more complicated. Collaboration is an amazing tool, and should be leveraged heavily. However, at the end of the day, after you do your research you need to pick a strategy and stick with it. Asking thousands of people to assist you in building your product probably will not work well.

Open Source is great – but some proprietary products are quite functional as well, have a long track record, are well supported, and will probably be upgraded.

Everything has its place, so use what works where it is needed. There is no single answer, sadly.

So did I waste my time reading the book? Did I make a bad choice? Not at all! Reading the opinions and thoughts of others is almost always useful, and it’s important to consider opinions other than your own. If nothing else, thinking through the process either convinces you that you are wrong, or helps you understand better why you are right.

The Data Scientist

2011-11-15T16:00:18Z

A new term - well, perhaps not that new - has come up and I’m actually very excited about it. The term is Data Scientist, and since it’s new, it’s fairly undefined. I’ll explain what I think it means, and why I’m excited about it.

In general, I’ve found the term deals at its most basic with analyzing data. Of course, we all do that, and the term itself in that definition is redundant. There is no science that I know of that does not work with analyzing lots of data. But the term seems to refer to more than the common practices of looking at data visually, putting it in a spreadsheet or report, or even using simple coding to examine data sets.

The term Data Scientist (as far as I can make out this early in it’s use) is someone who has a strong understanding of data sources, relevance (statistical and otherwise) and processing methods as well as front-end displays of large sets of complicated data. Some - but not all - Business Intelligence professionals have these skills. In other cases, senior developers, database architects or others fill these needs, but in my experience, many lack the strong mathematical skills needed to make these choices properly.

I’ve divided the knowledge base for someone that would wear this title into three large segments. It remains to be seen if a given Data Scientist would be responsible for knowing all these areas or would specialize. There are pretty high requirements on the math side, specifically in graduate-degree level statistics, but in my experience a company will only have a few of these folks, so they are expected to know quite a bit in each of these areas.

Persistence

The first area is finding, cleaning and storing the data. In some cases, no cleaning is done prior to storage - it’s just identified and the cleansing is done in a later step. This area is where the professional would be able to tell if a particular data set should be stored in a Relational Database Management System (RDBMS), across a set of key/value pair storage (NoSQL) or in a file system like HDFS (part of the Hadoop landscape) or other methods. Or do you examine the stream of data without storing it in another system at all?

This is an important decision - it’s a foundation choice that deals not only with a lot of expense of purchasing systems or even using Cloud Computing (PaaS, SaaS or IaaS) to source it, but also the skillsets and other resources needed to care and feed the system for a long time. The Data Scientist sets something into motion that will probably outlast his or her career at a company or organization.

Often these choices are made by senior developers, database administrators or architects in a company. But sometimes each of these has a certain bias towards making a decision one way or another. The Data Scientist would examine these choices in light of the data itself, starting perhaps even before the business requirements are created. The business may not even be aware of all the strategic and tactical data sources that they have access to.

Processing

Once the decision is made to store the data, the next set of decisions are based around how to process the data. An RDBMS scales well to a certain level, and provides a high degree of ACID compliance as well as offering a well-known set-based language to work with this data. In other cases, scale should be spread among multiple nodes (as in the case of Hadoop landscapes or NoSQL offerings) or even across a Cloud provider like Windows Azure Table Storage. In fact, in many cases - most of the ones I’m dealing with lately - the data should be split among multiple types of processing environments. This is a newer idea. Many data professionals simply pick a methodology (RDBMS with Star Schemas, NoSQL, etc.) and put all data there, regardless of its shape, processing needs and so on.

A Data Scientist is familiar not only with the various processing methods, but how they work, so that they can choose the right one for a given need. This is a huge time commitment, hence the need for a dedicated title like this one.

Presentation

This is where the need for a Data Scientist is most often already being filled, sometimes with more or less success. The latest Business Intelligence systems are quite good at allowing you to create amazing graphics - but it’s the data behind the graphics that are the most important component of truly effective displays.

This is where the mathematics requirement of the Data Scientist title is the most unforgiving. In fact, someone without a good foundation in statistics is not a good candidate for creating reports. Even a basic level of statistics can be dangerous. Anyone who works in analyzing data will tell you that there are multiple errors possible when data just seems right - and basic statistics bears out that you’re on the right track - that are only solvable when you understanding why the statistical formula works the way it does.

And there are lots of ways of presenting data. Sometimes all you need is a “yes” or “no” answer that can only come after heavy analysis work. In that case, a simple e-mail might be all the reporting you need. In others, complex relationships and multiple components require a deep understanding of the various graphical methods of presenting data. Knowing which kind of chart, color, graphic or shape conveys a particular datum best is essential knowledge for the Data Scientist.

Why I’m excited

I love this area of study. I like math, stats, and computing technologies, but it goes beyond that. I love what data can do - how it can help an organization. I’ve been fortunate enough in my professional career these past two decades to work with lots of folks who perform this role at companies from aerospace to medical firms, from manufacturing to retail.

Interestingly, the size of the company really isn’t germane here. I worked with one very small bio-tech (cryogenics) company that worked deeply with analysis of complex interrelated data.

So watch this space. No, I’m not leaving Azure or distributed computing or Microsoft. In fact, I think I’m perfectly situated to investigate this role further. We have a huge set of tools, from RDBMS to Hadoop to allow me to explore. And I’m happy to share what I learn along the way.

Developing a Cost Model for Cloud Applications

2011-11-08T16:30:49Z

Note - please pay attention to the date of this post. As much as I attempt to make the information below accurate, the nature of distributed computing means that components, units and pricing will change over time. The definitive costs for Microsoft Windows Azure and SQL Azure are located here, and are more accurate than anything you will see in this post: http://www.microsoft.com/windowsazure/offers/

When writing software that is run on a Platform-as-a-Service (PaaS) offering like Windows Azure / SQL Azure, one of the questions you must answer is how much the system will cost. I will not discuss the comparisons between on-premise costs (which are nigh impossible to calculate accurately) versus cloud costs, but instead focus on creating a general model for estimating costs for a given application.

You should be aware that there are (at this writing) two billing mechanisms for Windows and SQL Azure: “Pay-as-you-go” or consumption, and “Subscription” or commitment. Conceptually, you can consider the former a pay-as-you-go cell phone plan, where you pay by the unit used (at a slightly higher rate) and the latter as a standard cell phone plan where you commit to a contract and thus pay lower rates. In this post I’ll stick with the pay-as-you-go mechanism for simplicity, which should be the maximum cost you would pay. From there you may be able to get a lower cost if you use the other mechanism. In any case, the model you create should hold.

Developing a good cost model is essential. As a developer or architect, you’ll most certainly be asked how much something will cost, and you need to have a reliable way to estimate that. Businesses and Organizations have been used to paying for servers, software licenses, and other infrastructure as an up-front cost, and power, people to the systems and so on as an ongoing (and sometimes not factored) cost. When presented with a new paradigm like distributed computing, they may not understand the true cost/value proposition, and that’s where the architect and developer can guide the conversation to make a choice based on features of the application versus the true costs.

The two big buckets of use-types for these applications are customer-based and steady-state. In the customer-based use type, each successful use of the program results in a sale or income for your organization. Perhaps you’ve written an application that provides the spot-price of foo, and your customer pays for the use of that application. In that case, once you’ve estimated your cost for a successful traversal of the application, you can build that into the price you charge the user. It’s a standard restaurant model, where the price of the meal is determined by the cost of making it, plus any profit you can make.

In the second use-type, the application will be used by a more-or-less constant number of processes or users and no direct revenue is attached to the system. A typical example is a customer-tracking system used by the employees within your company. In this case, the cost model is often created “in reverse” - meaning that you pilot the application, monitor the use (and costs) and that cost is held steady. This is where the comparison with an on-premise system becomes necessary, even though it is more difficult to estimate those on-premise true costs. For instance, do you know exactly how much cost the air conditioning is because you have a team of system administrators? This may sound trivial, but that, along with the insurance for the building, the wiring, and every other part of the system is in fact a cost to the business.

There are three primary methods that I’ve been successful with in estimating the cost. None are perfect, all are demand-driven. The general process is to lay out a matrix of:

components
units
cost per unit

and then multiply that times the usage of the system, based on which components you use in the program. That sounds a bit simplistic, but using those metrics in a calculation becomes more detailed. In all of the methods that follow, you need to know your application. The components for a PaaS include computing instances, storage, transactions, bandwidth and in the case of SQL Azure, database size. In most cases, architects start with the first model and progress through the other methods to gain accuracy.

Simple Estimation

The simplest way to calculate costs is to architect the application (even UML or on-paper, no coding involved) and then estimate which of the components you’ll use, and how much of each will be used. Microsoft provides two tools to do this - one is a simple slider-application located here: http://www.microsoft.com/windowsazure/pricing-calculator/

The other is a tool you download to create an “Return on Investment” (ROI) spreadsheet, which has the advantage of leading you through various questions to estimate what you plan to use, located here: https://roianalyst.alinean.com/msft/AutoLogin.do?d=176318219048082115

You can also just create a spreadsheet yourself with a structure like this:

Program Element	Azure Component	Unit of Measure	Cost Per Unit	Estimated Use of Component	Total Cost Per Component	Cumulative Cost

Of course, the consideration with this model is that it is difficult to predict a system that is not running or hasn’t even been developed. Which brings us to the next model type.

Measure and Project

A more accurate model is to actually write the code for the application, using the Software Development Kit (SDK) which can run entirely disconnected from Azure. The code should be instrumented to estimate the use of the application components, logging to a local file on the development system. A series of unit and integration tests should be run, which will create load on the test system.

You can use standard development concepts to track this usage, and even use Windows Performance Monitor counters. The best place to start with this method is to use the Windows Azure Diagnostics subsystem in your code, which you can read more about here: http://blogs.msdn.com/b/sumitm/archive/2009/11/18/introducing-windows-azure-diagnostics.aspx This set of API’s greatly simplifies tracking the application, and in fact you can use this information for more than just a cost model.

After you have the tracking logs, you can plug the numbers into ay of the tools above, which should give a representative cost or in some cases a unit cost.

The consideration with this model is that the SDK fabric is not a one-to-one comparison with performance on the actual Windows Azure fabric. Those differences are usually smaller, but they do need to be considered. Also, you may not be able to accurately predict the load on the system, which might lead to an architectural change, which changes the model. This leads us to the next, most accurate method for a cost model.

Sample and Estimate

Using standard statistical and other predictive math, once the application is deployed you will get a bill each month from Microsoft for your Azure usage. The bill is quite detailed, and you can export the data from it to do analysis, and using methods like regression and so on project out into the future what the costs will be. I normally advise that the architect also extrapolate a unit cost from those metrics as well. This is the information that should be reported back to the executives that pay the bills: the past cost, future projected costs, and unit cost “per click” or “per transaction”, as your case warrants.

The challenge here is in the model itself - statistical methods are not foolproof, and the larger the sample (in this case I recommend the entire population, not a smaller sample) is key.

References and Tools

Articles:

http://blogs.msdn.com/b/patrick_butler_monterde/archive/2010/02/10/windows-azure-billing-overview.aspx

http://technet.microsoft.com/en-us/magazine/gg213848.aspx

http://blog.codingoutloud.com/2011/06/05/azure-faq-how-much-will-it-cost-me-to-run-my-application-on-windows-azure/

http://blogs.msdn.com/b/johnalioto/archive/2010/08/25/10054193.aspx

http://geekswithblogs.net/iupdateable/archive/2010/02/08/qampa-how-can-i-calculate-the-tco-and-roi-when.aspx

Other Tools:

http://cloud-assessment.com/

http://communities.quest.com/community/cloud_tools

Windows Azure Use Case: Supplementing Infrastructure

2011-11-01T14:22:01Z

I’ve explained before that Windows Azure is a Platform as a Service - at its simplest, that means that you write software and Azure runs it for you. But what if you are a shop that normally buys “off the shelf” software, and the only software you write is an internal utility here and there - can you still use Windows Azure?

Absolutely. Windows Azure is made up of several components, such as computing, storage and other objects you can call in code. And as such, some companies have extended software and hardware you can use that is backed by Windows Azure. In some cases that’s simply software you can use like you would any other - making Azure a “Software as a Service”, which I’ll write about later.

But in other cases the software a vendor provides a utility you can use for your infrastructure. For instance - “BlobShare”, a free codeplex offering, allows you to create a place where you can upload and download files securely. It allows you to hook in your local Windows accounts (without sending names and passwords over the Internet) or even Google, LiveID or Yahoo logins to access the files. You can think of it as a “corporate DropBox”. Download it and read more about it here: http://blobshare.codeplex.com/ and learn how to hook in Active Directory here: http://blogs.msdn.com/b/vbertocci/archive/2011/10/31/blobshare-sample-acs-protected-file-sharing.aspx

Another option along these lines is using a hardware appliance to store data in Azure. This has a particular attraction, since some of these appliances (such as the offerings from StorSimple) do more than just act as a storage target. The device plugs into the network as storage, moves the data to Windows Azure, de-duplicates (thus saving storage costs), encrypts, acts as a backup device, works with SharePoint and more. No coding is required to use their solution, and it can even act as a Disaster Recovery site.

Storage isn’t the only way to use Windows Azure in place of infrastructure. Using the Application Fabric Service Bus, you can perform some functions of data access and transfer between companies that used to take a VPN setup - which is no longer required. the caching function in the Service Bus can even relieve the need to upgrade a server for performance. And of course SQL Azure allows you to access a SQL Server database without any server at all.

So if you’re thinking that Windows and SQL Azure aren’t for you, think again. Start with the problem you have, and see what options you have for solving it.

Book Review (Book 5) - The Cloud of Unknowing

2011-10-31T17:23:23Z

This is a continuation of the books I challenged myself to read to help my career - one a month, for year. You can read my first book review here. The book I chose for October 2011 was: The Cloud of Unknowing, Anonymous: The role of faith in life. Once again, this is out of order, but the book came in from a hold at the library so I’ll do this one now.

Why I chose this Book:

This book is probably the most far afield for many of my readers – some folks don’t hold a faith, others have faiths that are different from mine. That’s fine – I think this is still an intriguing read.

However this is a religious work – if you’re not into that sort of thing, it’s completely OK. As I’ve mentioned early-on, a book list for your career can include many kinds of books. Faith is such a part of my life that I find it impossible to separate from my day-to-day efforts. To that end, a little about this book is in order.

The book was written in the 14th century by an unknown, anonymous author, probably a Carthusian monk in Europe. It’s a work involving the deep thoughts around the intersection of intellection over contemplation. Contemplation is what ancient Catholics called meditation, or focus. It’s a really deep work involving philosophical history, specifically in the Christian tradition. Even as old and philosophical as it is, many contemporary writers, singers (Like Leonard Cohen) and others have referenced it, used it words, or developed entire works around it.

What I learned:

So what did I learn? Actually quite a bit, specifically on the role in contemplation in life. As a technologist, I tend to stay incredibly busy and the temptation is always to be very scattered. But lack of focus is often the enemy of getting things done correctly. Focusing on what really matters and being in the moment is a powerful tool in professional life. It has taught me that now, more than ever, I need to decide what I will not give a lot

As I mentioned, this is a religious work – it has far more value to me there than in the pure business sense, although I find those two linked irrevocably. The way I treat others in both personal and business relationships is my character – which I strive to improve every day.

I highly recommend you read something that is internal like this in your career development. Money, success and fame are not all that there is – and those things are not separated from who you are as a person.

Bonus Rant: Conference Speakers and the Golden Rule

2011-10-20T15:07:34Z

Fair warning: This isn’t a technical post. This is my opinion, and not Microsoft’s.

I’ve been a public speaker at events since 1980. Interestingly, even though I’ve done this, it’s never been my paid job. I’ve spoken on technical topics at groups as small as 5 people to over 10,000, in
local and global events. I’ve been on TV, radio, on stage and on webcasts. I teach at college as well.

In all of that time, I’ve come to realize that speaking is the easy part. To be sure, I put an extraordinary amount of work into my presentations. I learn the topic, research speaking styles and communication
methods, fret over my demos and practice relentlessly. But I reiterate that this is the easy part.

The hard part is putting the events together.

Most folks have no idea how hard it is to create, advertise, pay for, staff, run and tear down a technical event. Even small ones. And the folks that do it get none of the glory, none of the praise, and usually all of
the criticism. These folks deserve way more respect than they get.

And the worst offenders in not offering this respect are the speakers.

Let me say that again – some speakers are rude, arrogant and disrespectful of the event planning groups that run their sessions. They have unrealistic expectations, and I’ve seen some actively fight the requirements these teams impose. From not delivering slides on time, to not following the rules the event planners lay out, to not even calling when they don’t show up. Showing up and saying "I'm still working on my slides" with a smirk doesn't impress me as a listener at all - it just makes me think you're dismissing my time as unimportant as a listener. Your content should be nailed way before the event, not the day of. Practicing on the day of presentation is expected - changing it indicates unprofessionalism. Sorry, but there it is.

I don’t bring up problems without suggesting some solutions – that’s my military training coming through. :) Here’s what can fix this issue:

All speakers should have to demonstrate they have assisted in the logistics of an event somewhere. Or you don’t speak.
All speakers should follow the rules. Don’t get your slides in, don’t follow the rules? You don’t present. There are LOTS of people wanting
to speak.
All speakers should follow the golden rule. Sure, you’re special, just like everyone else. Understand that the event planners are there – with you – to make the event a success. Treat them like you want to be treated.

No, I’m not pointing anyone out. No, no one asked me to write this post. I’ve just been on both sides of the fence, and you really have no idea how difficult the logistics of an event are until you put one on. Want to find out if I’m right? Volunteer at a local even like SQL Saturday, ask to help out at PASS, TechEd, whatever. The point is, walk a mile in the shoes of the event planner. It will make you a better presenter.

Big Data and the Cloud - More Hype or a Real Workload?

2011-10-18T14:57:36Z

Last week Microsoft announced several new offerings for “Big Data” - and since I’m a stickler for definitions, I wanted to make sure I understood what that really means. What is “Big Data”? What size hard drive is that? After all, my laptop has 1TB of storage - is my laptop “Big Data”?

There are actually a few definitions for this term, most notably those involving the “Four V’s” Volume, Velocity, Variety and Variability. Others disagree with this definition. I tend to try and get things into their simplest form, so I’m using this definition for myself:

Big data is defined as a large set of computationally expensive data that is worked on simultaneously.

Let me flesh that out a little. To be sure, “Big Data” has a larger size than say a few megabytes. The reason this is important is that it takes special hardware to be able to move large sets of data around, store it, process it and so on. (large set)

If you store a LOT of data, but only use a small portion of it at a time, that really isn’t super-hard to do. It’s mainly a storage issue at that point. But, if you do need to work with a large portion of the data at one time, then the memory, CPU and transfer components of the system have to adapt to be responsive - new ways to work with that data (game theory, knot-algorithms, map-reduce, etc.) need to be brought into play. (computationally expensive)

Once that data is loaded into the processing area (memory or whatever other mechanism is used) it must be worked on in parallel to come back in a reasonable time. You have two options here - you can scale the system up with more internal hardware (CPU’s, memory and so on) or you can scale it out to have multiple systems work on it at the same time using paradigms such as map/reduce and so on. Actually, when you lay this out in an architecture diagram, scale up or out doesn’t actually change the logical structure of the process - in scale out the network becomes the bus, and the nodes become more RAM and computing power. Of course, there are changes in code for how you stitch the workload back together. (worked on simultaneously)

So back to the original question. Is Big Data, as I have defined it here, a workload for Windows and SQL Azure? Absolutely! In fact, it’s probably one of the main workloads, and I believe it represents the latest, and perhaps also the earliest frontier of computing. Jim Gray, a former researcher here at Microsoft and a hero of mine, was working on this very topic. I believe as he did - all computing is simply an interface over data.

Microsoft has multiple offerings on the topic of Big Data. In posts that follow from myself and my co-workers, we’ll explore when and where you use each one. Whether you are a data professional or a developer, this is the new frontier - don’t wait to educate yourself on how to leverage Big Data for your organization.

Hadoop on Windows Azure and SQL Server - Microsoft’s partnership to include Hadoop workloads on Windows Azure and SQL Server/Parallel Data Warehouse (PDW)

LINQ to HPC - Microsoft’s High-Performance Computing SKU of HPC is now in Azure

Windows Azure Table Storage - A key/value pair type storage with full partitioning that is immediately consistent, able to handle huge loads of data and works with any REST-compatible language

Other offerings - Including the new Data Explorer, Project Daytona (with a Big Data Toolkit for Scientists and researchers), Power View and more.

The era of Big Data is here. And you can use Windows and SQL Azure to bring it to your organization.

Creating a Distributed Computing System Using a Windows Azure Queue

2011-10-11T14:12:42Z

The Windows Azure Queue component, like all Windows Azure components (Roles, Storage, App Fabric, SQL Azure) can be used by itself or with other Windows Azure components. That’s why I refer to Windows Azure as “Distributed Computing” rather than “cloud”.

Having a distributed off premise queue has a lot of use-cases. An interesting use-case is a company that wanted to harness the power of all of the PC’s and laptops in the company when they were not being used throughout the day. A developer wrote a screen-saver program that connected to an Azure Queue, pulling work off of the queue and placing an entry when it was done. In essence he had a partially connected distributed work relay system, and since he used a Windows Azure Queue, the system worked from anywhere in the world.

He uses an on-site central server (which was actually only a workstation-level system) that holds the computations in a scatter/gather paradigm. The computations are broken into less-than-8K chunks, so that it fits within a message. The server connects to a Windows Azure Queue, and places the message marked for computation. It also scrubs the Queue for completed work, and as part of the process puts that kind of message into a mapping function (queues are not guaranteed a message order).

The workstations that are not being used (even those systems at remote workers and travelers) connect to the same Windows Azure Queue when the system is not being used for a period of time, when the screen saver kicks in. It then takes one message from the queue, computes the information, and then sets a new message for the server to pick up with the answer. The workstation then deletes the message.

The Server picks up the completed work, processes it and then deletes that queue message. He also added logic to process messages for computation on the server as well, when the server function of adding work is not required.

There are a few caveats here. This works because of the mapping function on the head server. Order is not guaranteed, so he includes a number for the function step as part of the message body, which cuts the size a bit. Also, he’s careful to watch the encoding, since Azure will hand binary back in Base64 format.

He’s found that there are enough systems to ensure that the messages are cleared every few days – important, since the Windows Azure Queue ages out after seven days. Also, he’s careful to use the CloudQueue.PeekMessage function when he wants to monitor the system – that function ensures that the message status doesn’t reset as “read” when he accesses it.

This is a great example of using the “cloud” as what it is intended to be – a distributed architecture you can use as needed to solve a business problem. It’s not an “all or nothing” proposition, but instead it is simply another set of components to use where you need them.

Windows Azure Use-Case: Mobile Development

2011-10-04T15:11:50Z

One of the most natural development use-cases for the Windows Azure platform is with mobile devices, such as smartphones, tablets and even embedded computers in cars and other devices.

The reason this paradigm works so well is that most mobile devices are not as powerful as larger computers - they simply don’t have the room to power the device, keep it cool, and hold the components that they would like. So leveraging the computing power, data and networking capability from a massively distributed system is a natural fit.

Mobile devices, by definition, are often constantly connected to the internet. That means that you can access Windows Azure components from these devices whenever they are connected, and you don’t have to open firewalls into your own computing environment. In fact, you don’t have to have an infrastructure at all to set this all up - the user has the client (mobile device) the Internet is your network, and Windows Azure runs your code. Nothing on-premise is needed. Of course, using a hybrid model you can include components from on-site into your design as well.

So what are the main components of a mobile platform? There’s a great video by my friend Wade Wegner that you can watch here that covers these components step-by-step, but in a simple outline, here they are:

Client development environment and deployment

Each mobile device has an operating system and environment that it works with. Windows Azure has full toolkits you can use to develop against the major platforms in use today. You’ll need the Windows Azure Software Development Kit first (free) and a development environment (even the free Visual Studio Express edition is fine) and then you simply download the toolkits here:

Windows Phone: http://watwp.codeplex.com/

Android: https://github.com/microsoft-dpe/wa-toolkit-android

iOS (Apple): https://github.com/microsoft-dpe/wa-toolkit-ios

Security and authentication

From there, you need to log people in and out of the service you provide. You can certainly write your own authentication into Windows Azure if you wish, but it’s often far easier to allow your users to connect with an identity they’ve already established. This takes you out of the business of storing profiles, names and passwords.

You can use multiple providers - and some of the easiest to work with and most popular are Windows Live, Facebook, Google and Yahoo. Using the Access Control Service (ACS) feature in Windows Azure, you can “point” to those providers and the user will be presented with those login screens. From there, you trust the provider to hold the names and passwords identifying the user. You can learn more about that process here: http://www.wadewegner.com/2010/08/use-social-web-providers-in-less-than-5-minutes/

Of course, you’re still able to write your own login logic, and you always control access to the assets the user can see.

Networking and connectivity

With a mobile device, the carrier that the user has is responsible for the “last-mile” to the device. But with Windows Azure, you are using the some of the largest connections to the internet available - and you have three regions around the world to deploy your applications to. You’re able to select which region you want, so you can provide everything from code to data to the device in a reliable way.

One caveat here - you need to ensure your code has logic to deal with connectivity issues. You don’t control the provider the device uses, so you should make decisions about caching, retry logic, disconnect messages and so on based on what your application does. Different parts of your application might require a specific strategy around what needs to be constantly connected and what can display on it’s own from the storage on the device. There are always tradeoffs for each of these decisions.

Platform components

Windows Azure is a platform - which means that it isn’t a single unit or thing, it’s a group of components that work together. You can choose any or all of these components based on what your application needs to do.

Computing - With Windows Azure, you can call on standard computing resources to provide the front-end for your application, or to do the processing for your application logic. You might choose to embed the full client experience on the device, and simply have Windows Azure compute data or perform other logic. Or, you might choose to develop the entire application, front-end and all, on Windows Azure. This approach is sometimes the least “rich experience” method, but the most device-independent. Using the Windows Azure platform you can scale (even programmatically) from very small when the demand is low to thousands of compute resources based on higher demand, and back down again.

Storage - Windows Azure has multiple types of storage, from NoSQL-like table storage to binary large object storage for documents, music, videos, and so on. Your application can use storage independently - perhaps backing up the application’s data into Windows Azure and nothing else. Or, you can use storage with the computing resources. There is also a message queue that is part of the Windows Azure storage, allowing a fully REST-ful development paradigm which allows for scale. Queues are also interesting for a combinatorial approach - passing computing work between devices and then re-combining those. In fact, using this paradigm you could have your mobile device submit jobs to a super-computing fabric.

The binary storage also has a Content Delivery Network (CDN) feature that moves the data closer to the user in an “edge” methodology.

Application Fabric - The Application Fabric component of Windows Azure is actually made up of yet more components. This is the area that holds the ACS mentioned earlier, and it also has a full Service Bus component for inter-process communication. You can also leverage a cache in this component to speed up your applications.

SQL Azure - For mobile development, working with location-aware applications is a primary feature. SQL Azure is a Relational Database Management System (RDBMS) that has a Spatial Data capability, where you can work with not only Geodesic information on the mobile device, but geometric data as well.

Resources

There are many resources for Windows Azure development for mobile devices. The primary one is here: http://www.microsoft.com/windowsazure/scenarios/mobile-applications/ You can find code, videos, whitepapers, the toolkits and more from that link. In addition, be sure and follow Wade Wegner’s blogs and postings for more information.

Book Review (Book 4) - Predictably Irrational

2011-10-03T16:07:26Z

This is a continuation of the books I challenged myself to read to help my career - one a month, for year. You can read my first book review here. The book I chose for August 2011 was: Predictably Irrational, Revised and Expanded Edition: The Hidden Forces That Shape Our Decisions, by Dan Ariely.

Why I chose this Book:

This is another non-technical book, or even a business book, but I believe that understanding the behavior of others - and more importantly, my own behavior. I’ve heard of this author before, and when I read the book reviews they spoke about the fact that people often do things that don’t make sense - even when they know that the actions don’t make sense.

What I learned:

I started this book out of the order I had originally selected, since I had put it on hold at the Seattle Public Library and it came in today. Interestingly, it came in an e-book format, which I really like. I can read and take notes at the same time on the Tablet-PC I use (a Lenovo X220).

You have to read the introduction for this book to have its full impact. The author was burned over 70% of his body, and as he slowly recovered, he became an observer of those around him. He was able to watch as intelligent people behaved in ways that were intuitive, but did not produce the goals they really wanted. In other words, their actions (and even thoughts) went against what they really wanted to have. The key to using this book is to recognize this behavior in yourself and others.

The author backs everything up with references to studies, data and other supporting evidence, but does not show that data in the book. If you’re the anal type, you might want more evidence right in the book. For my part, that wasn’t an issue.

I also learned that I need to be self-aware. I read a little in these books on my list each night, and after each reading I was a little more surprised - and sometimes even disbelieving that I would act this way. But in fact, I do - as do we all. It’s quite an eye-opening experience to read this book.

Another interesting topic was the fact of “relativity” - or where you start from in a negotiation. It’s why people who move from a low-cost area (like I did from Florida) to a high-cost one (Like Seattle where I live now) are so shocked by prices, and push back on them so hard. I’ve seen this play out when people are asked to use a pay-for-use model like a cloud system, when they are used to an all-you-can-eat license for on-premise system. Even when you add up all the costs of an on-premise system and show someone how much they spend, they don’t want to believe that. The relative pricing they are used to makes them believe that a cloud model is too expensive.

I have to recommend this book to everyone in the tech industry - or in fact everyone that wants to advance in their career. I found myself over and over not wanting to believe what I was reading - that I would certainly never act in a way that doesn’t make sense. But I do - and we all do. Recognizing these patterns gives you an edge in working smarter, and understanding the people you work for and the ones that work for you.

Raw Notes:

I take notes as I read into OneNote - my favorite note-taking tool. These might not make sense in isolation, but here’s what I wrote down as I read:

People do not always act in their own best interests
Relativity helps with comparison issues
Important to focus on staying objective
Arbitrary coherence and anchoring are two behavior motivators
People often look for a “middle choice” - hence low/med/high pricing models
Theories of motivations are important to investigate further
Primary decisions set coherency
Bundle with free - Free is a powerful motivator
Observation of other’s behavior leads to behavior control

Query SQL Azure and On-Premise Data in One Codeset

2011-09-20T16:26:23Z

I was asked recently how to query SQL Azure along with an on-premise data source. There are actually lots of ways to do this, but a quick and simple method is to use LINQ - or Language Integrated Query. There are always pro's and con's for any data access methods, but I'll show this one as an example, one I've used in other places. It's important to understand what you're doing, and why.

Note that the key here is in three parts - the connections, the DataRelation object space within the .NET environment, and the LINQ query that presents the data. I'll show this example in C#, and keep in mind that you don't even have to call the Azure SDK - this is all straight Tabular Data Stream (TDS) layer that SQL Server uses. Although I'm querying from SQL Server as well as SQL Azure, you could also query Oracle, XML, text files, even a Excel Spreadsheet. To keep it simple I'm querying two data sources, but you can include more in your tests.

For this example I'll tie together two Purchase Order databases, assuming that a PO number is shared between them - a key relationship that may not exist for you. Be careful on that front.

I'm using a special connection string for SQL Azure, since it requires a different format for the server name, login name and of course it needs to be encrypted. You can see that below. As always, test systems only, your mileage may vary, don't run wth siccors.

//Set up two connections, more are possible
SqlConnection companyConnection = new SqlConnection("Data Source=FirstServer;Server=tcp:[serverName].database.windows.net;Database=PurchaseOrders;User ID=[LoginForDb]@[serverName];Password=myPassword;Trusted_Connection=False;Encrypt=True;");
SqlConnection lineItemConnection = new SqlConnection("Data Source=SecondServer;Integrated Security=SSPI;Initial Catalog=PurchaseOrders");

SqlDataAdapter headerAdapter = new SqlDataAdapter("SELECT * FROM Company", companyConnection);
//Note: if this is to be a different source such as Oracle or text files, set up a different
//connection information above and possibly a different connection below
SqlDataAdapter lineitemAdapter = new SqlDataAdapter("SELECT * FROM LineItem", lineItemConnection);

// This is the combined dataset
DataSet companyLineItems = new DataSet();

// Go get data from each dataset from above
headerAdapter.Fill(companyLineItems, "companies");
lineitemAdapter.Fill(companyLineItems, "lineItems");

// Join them
DataRelation relation = companyLineItems.Relations.Add("CustLineItems",
companyLineItems.Tables["companies"].Columns["companyID"],
companyLineItems.Tables["lineItems"].Columns["companyID"]);

// Display the combined data
foreach (DataRow pRow in companyLineItems.Tables["companies"].Rows)
{
Console.WriteLine(pRow["companyID"]);
foreach (DataRow cRow in pRow.GetChildRows(relation))
Console.WriteLine("\t" + cRow["LineItem"]);
}
Using Language Integrated Query (LINQ)

Another method is to use Language Integrated Query (LINQ). LINQ is a single data query language used in .NET programming languages. It's completely integrated into the .NET framework, so developers use this construct quite often. I have an entire article on this technology here, so I won't repeat that information in this article.

Here's a snippet of an example of two text files in C# (the SQL Server examples are longer, to be more clear I'm just using text files here):
class PurchaseOrder
{
    public string CompanyName { get; set; }
    public string CompanyAddress { get; set; }
    public int ID { get; set; }
    public List<int> LineItems { get; set; }
}

class PopulateCollection
{
    static void Main()
    {
   string[] POHeaders = System.IO.File.ReadAllLines(@"../../../POHeaders.csv");
         // File example, Bucksoft,1234 Mockingbird Lane,1001

string[] LineItems = System.IO.File.ReadAllLines(@"../../../LineItems.csv");
// File example, 1001, Product 234, Product 237, Product 341, Product 795

        // Merge the data sources - This is the main part of the work
        IEnumerable<PurchaseOrder> queryNamesLineItems =
            from nameLine in POHeaders
            let splitName = nameLine.Split(',')
            from itemLine in LineItems
            let splitLineItemLine = itemLine.Split(',')
            where splitName[2] == splitLineItemLine[0]
            select new PurchaseOrder()
            {
                CompanyName = splitName[0],
                CompanyAddress = splitName[1],
                ID = Convert.ToInt32(splitName[2]),
                LineItems = (from itemAsText in splitLineItemLine.Skip(1) // Do not show the key again
                              select Convert.ToInt32(itemAsText)).
                              ToList()
            };

List<PurchaseOrder> companys = queryNamesLineItems.ToList();

        // Display each company's name and items
        foreach (var company in companys)
        {
            Console.WriteLine("The items are {0} {1} {2} {3} {4}", company.CompanyName, company.CompanyAddress, company.LineItems);
        }

        //Keep console window open in debug mode
        Console.WriteLine("Press any key to exit.");
        Console.ReadKey();
    }
}

Rip and Replace or Extend and Embrace?

2011-09-13T12:20:05Z

As most of you know, I don’t like the term “cloud” very
much. It isn’t defined, which means it can be anything. I prefer “distributed
computing”, which is more technically accurate and describes what you’re doing
in more concrete terms.

So when you think about Windows and SQL Azure, you don’t
have to think about an entire product – you can use parts of the system
together or independently to accomplish what you need to do. You can use the
computing functions, storage, and more and more I see folks leverage the
Service Bus to enable current applications to expose things to the web.

And that brings up the point of this post. Once you decide
that a distributed architecture works to solve a problem, you’re faced with a
decision: should you completely re-write your architecture to take advantage of
the current systems or should you just fold in new code that makes the data or
function available to the web?

Of course, the answer is always “it depends” on the situation
– and it does. But unless you’re fixing a problem with current code, I usually
advocate a migration approach. That means at the very least retaining the
business logic (again, unless it’s not currently working) and as much of the
code as you can. In fact, if you follow this paradigm, you’re on your way to
making a Service Bus out of the functions you currently have. You can expose
the results of a system rather than opening the system up. Let’s take an
example.

Assume for a moment that you have an order-taking system
on-premise. That system performs many functions, one of which might creating a
Purchase Order. Your system might be enclosed, meaning that it has an
application that talks to a middle-tier, and then from there to a database
system. A query is generated from a screen, and passed along to eventually
compute, store and return a Purchase Order Number, along with other
information. Imagine now that you wire up the code not only to return the PO
number to the client, but to make that number available on an endpoint –
actually really not that hard to do.

Now you can make that PO number available to the web using
Azure. You could restrict who can make that call to the system, or open it up
to a broader audience. Or instead of the PO Number, you could make a product
list available. And you can go further than that – EBay, for instance, uses the
OData protocol (which is very cool in and of itself) which you can query from
the web. You could compare your company’s product catalog to what is on EBay,
and list the items you have there if there are no competitors in that space.
And on and on it goes.

So the point is this – where you can, retain what works.
Fold in systems like Azure where they make sense. Extend and Embrace.

Plan for Diagnostics in Cloud Computing From the Git-Go

2011-09-06T14:11:22Z

“Git-Go” is something we say in the South that means “right at the start”. I’ve seen several applications for on-premise systems that don’t have much in the way of diagnostics - the developers rely on a debugger, the event logs on the server and client workstation, and most of all, the ability to watch the system from end-to-end.

This approach is a mistake for an on-premise system, and it’s definitely a problem for a distributed architecture. You simply do not own all of the components from end to end in a cloud environment, nor are you always able to attach a debugger or other remote monitoring tools to the various areas within the code path. So you need to make sure that from the very outset of your design that you build in diagnostics. My personal preference is to build a system such that a control file turns on deeper information gathering from the system, up to a minimal level.

When I do that, I set a high level of logging, a medium level, and a moderate level. I normally use the deepest level of information during the testing and acceptance phase of the deployment, then switch to moderate and then the least level of information gathering. Also in my design I often set an error condition to begin gathering the deeper information along with the exception, where possible.

There are decisions you need to make as to where to store the diagnostics (many operations in the cloud cost money), how often you collect them, and so on. You can get a quick overview on using the diagnostics that come with Windows Azure here: http://www.azuresupport.com/2010/03/getting-started-with-windows-azure-diagnostics-and-monitoring/ This is where you should start first. More detail on that: http://msdn.microsoft.com/en-us/library/gg433048.aspx

My friend Dave Pallman has a great tool he’s released for free: http://davidpallmann.blogspot.com/2009/03/azure-application-monitor-now-on.html

If the issue is in storage apps: http://social.msdn.microsoft.com/Forums/en-US/windowsazuredata/thread/d84ba34b-b0e0-4961-a167-bbe7618beb83

If you have System Center, this is the quickest and easiest way to implement the monitoring – really handy: http://pinpoint.microsoft.com/en-us/applications/windows-azure-application-monitoring-management-pack-release-candidate-12884907699

Book Review (Book 3) - Favorite Folktales from Around the World

2011-08-31T15:34:58Z

This is a continuation of the books I challenged myself to read to help my career - one a month, for year. You can read my first book review here. The book I chose for August 2011 was: Favorite Folktales from Around the World, by Jane Yolen.

Why I chose this Book:

No, this isn’t a technical book or even a business work, but I believe that being able to tell a story is at the heart of almost everything I do in my career. From explaining technology to a broad audience to getting the point of a use-case across during an architectural design review, a story is a succinct way of ensuring that the germ of an idea is passed from the speaker to the listener.

This book has a short introduction to each block of stories, gathered from cultures around the world as the title suggests. My thought was that studying stories would help me create my own.

What I learned:

Being from the South of the U.S., I grew up in a culture of storytelling. Everyone I knew told stories for just about everything, so analogies and so on come naturally to me. But I have never stopped and actually dissected a story, to find out why it worked, and why I chose a particular story or story style for a given situation.

I have a little confession here - I rarely just read the books I’m listing here on my blog without adding other references to them. I keep a dictionary handy in case I want to know the etymology of a word, since I believe word choices are important. I read other reviews on the work. And I try and find similar works to read at the same time to get a different perspective. While there are lots of anthologies of stories, I found a lack of quality work on using storytelling in the way I’m thinking. I was actually surprised at that. I only found a few books on storytelling, and most had to do with how you use a story, not how to tell or create one. It’s as if folks thing you already know how to fashion a story that will resonate, and you just need to know how to use them. Or they think you’re only telling stories to small children, but I think everyone loves a good story. Perhaps one day I’ll research this more and write my own book on how to tell a story.

The book itself is arranged in groups of types of stories, such as “magical beings”, “truths and lies”, “smart and ignorant”, “the powerful and the weak” and so on. The author gives a short (too short, in my mind - would LOVE to hear more of what she thought) introduction to the type of story, and then multiple stories on that genre followed. Some are short, others are much longer, but all deal with the same topic, which at times made for repetitive reading. Did you know, for instance, that there are probably thousands of versions of “Cinderella”? You’ll read several in this book.

As I read I learned that many stories have the same structure, that is, introduction and development of character, a “stressor” (situation or person) and a resolution. Even cultures that are comfortable with duality (like China and India) have a resolution to the story, whether positive or negative. This is interesting, because it seems a story should always be “round”, or complete. It sounds pat, or trite, but it’s just true. The main character can’t disappear from the storyline, the plot can’t change to something else entirely and so on. That, of course, deals with short stories - ones you can remember and tell again, not written, longer stories.

I’ll end this review with one of my favorite stories from the book. Most are at least a couple of pages long, but this one struck me because it’s touted as the world’s shortest horror story - and that succinctness really resonates. See if you agree:

“He awoke with a start, and reached for the lamp - and it was placed into his hand.”

Scary.

Raw Notes:

I take notes as I read, calling that process “reading with a pencil”. I find that when I do that I pay attention better, and record some things that I need to know later. I’ll take these notes, categorize them into a OneNote notebook that I synchronize in my Live.com account, and that way I can search them from anywhere. I can even read them on the web, since the Live.com has a OneNote program built in. I took a TON of notes during this reading, but I don’t think it makes sense to post them this time. There were so many, and due to the structure of the book they were so random, that I don’t think reading through them would help much past what I’ve posted in the “What I learned” section. I’ll see if the raw notes from the next work are the same or if it makes sense to add them here.