Client plus Cloud - All Comments

re: TIP-ING SERVICES TESTING BLOG #1: THE EXECUTIVE SUMMARY

公直 — Tue, 09 Oct 2012 11:13:37 GMT

nice to see such good sharing.

re: Microsoft Cargo Container Data Center comes Online

abia john — Thu, 16 Feb 2012 06:39:06 GMT

very simply explained. It is indeed an art to read & stop new visitors with your attractive writing style. I am really impress from your posted information. Thanks for sharing.

<a href="www.customsclearanceworld.com.au/Services.aspx">shipping services</a>

re: Penultimate Day of School 2011

Seth Eliot — Mon, 20 Jun 2011 16:36:32 GMT

Congrats Ken. query={Penultimate kosher} brings up this blog post as #1 result on Bing :-)

re: AN UNEXPECTED PLUG FOR “HOW WE TEST SOFTWARE AT MICROSOFT”

Mack — Thu, 14 Apr 2011 17:26:02 GMT

Writing a book is one thing selling it is something else. This truth was taught to me in the school of hard knocks. I help others with my lessons learned. http://teamebookpublishers.com

re: 1+1 Redundancy Just Isn't Good Enough

R.K.Johnston — Wed, 27 Oct 2010 16:21:50 GMT

Wow, this is a super old Blog post. I'm a software guy, not a power guy but as I'm now in Bing and we have dealt with DC power I'll see what I can do to answer your question.

1+1 is a way of expressing the relative risk footprint. If your service is to have even three 9s of availability you can't really go with 1+1 redundancy at any point in your all up service or DC architechture. But how you derive 1+1 is relative. In Bing we use cargo containers for thousands of servers. Each Cargo container has some redundancy in power - essentually pulling power from two different systems within the data center with each on seperate diesel generator backups. So I would say that is 1+1 at the cargo container level.

The thing is we have a lot of cargo containers and we do not run a single major service of Bing (say Bing Video) inside of a single cargo container and for an important service element it always has instances in multiple DCs. In that way our services have greater than 1+1 redundancy despite the cargo container chokepoint.

Here is an example I've read about in the past 12 months happening at Twitter, Google, Facebook, Yahoo, and Bing. All have had errant deployments that hit a large % of production all at once. What typically happens here is that a change that is considered safe is made to say one or more Data Center's worth of machines all at once. The change causes all of those machines to fail. They may be only 25% of the total machines supporting the service but they are substantial enough that they cause cascading failures across the entire service. In this sense the automated deployment has created a risk point.

One more example of risk chokepoints tends to be networks at a datacenter or regional hub. Edge routers for a DC are often pairs to provide 1+1 redundancy. One is taken out of rotation for an OS upgrade or a config change and the other is left active. Far too often during maintenance the one in active rotation fails while the other is in maintenance. Another way I have seen failures is when the one that was in maintenance is put back into rotation as primary and the secondary router is immediately taken offline for maintenance. Well the recently upgraded router looks good for a few minutes and we go into upgrade on the other one. While we are in the middle of the upgrade on the second router the one we recently put into active rotation begins to show failure symptoms. Now we don't have our failover option anymore because we are in the middle of upgrading. Boom, outage. That's not such a big deal though if you have multiple data centers and you can fail traffic over to another redundant system. Of course you have to be as big as Bing to have that level of redundancy.

Wow, long answer. Maybe I should have just made it a blog post. Anyhow, hope it helped a little.

re: 1+1 Redundancy Just Isn't Good Enough

Graham Davies — Wed, 27 Oct 2010 10:51:05 GMT

We need to add more power to our small data center, I recieved a quote for 1 x 60KVA DSTS ( Digital Static Transfer Switch ) it has 2 inputs from 2 differant UPS's, is this 1+1?

http://blogs.msdn.com/tomholl/

TrackBack — Fri, 30 Apr 2010 19:32:36 GMT

re: Telemetry Myopia: The Toyota “Fail Safe” system Debacle and Google cavalier attitude toward Privacy

simon hornby — Tue, 23 Mar 2010 21:03:13 GMT

Hi Ken, I also wanted to add that telemetry is something which will be adopted by toyota soon.

re: Telemetry Myopia: The Toyota “Fail Safe” system Debacle and Google cavalier attitude toward Privacy

simon hornby — Tue, 23 Mar 2010 21:02:19 GMT

Dear Ken,

This is one of the best and most comprehensive posts I've read on the entire Toyota issue - if only there was more common sense in the world. thankyou.

re: Telemetry Myopia: The Toyota “Fail Safe” system Debacle and Google cavalier attitude toward Privacy

JavierCaceresAlvis — Thu, 18 Mar 2010 15:02:55 GMT

Hi Ken,

"foolish statements" - I agree.

Every time I read your blog and book, I learn something new.

I made some testing on the MSDN forums sign in system and you can’t imagine some surprises I found:

http://speechflow.spaces.live.com/

Thanks!

Javier Andrés Cáceres Alvis