Wednesday, January 02, 2008

Perrow and Normal Accidents

While reading the book 'Deep Survival' (most kindly given to me at the UKOUG conference in Birmingham by Sir Graham Wood of Oracle after the fire in my house) I happened on a description on page 107 of a book called 'Normal Accidents' by a fellow named Perrow (get it? per row - a perfect name for database nerds).

Perrow's theses is that in any tightly coupled system - in which unexpected interactions can happen - accidents WILL happen, and they're NORMAL.

Also, he states that technological steps taken to remedy this will just make matters worse.

Perrow and IT systems
=====================
I have freely translated Perrow's thoughts into the following:

IT systems are tightly coupled. A change - a patch, a new application, or an upgrade - to a layer in the stack can cause accidents to happen, because they generate unexpected interactions between the components of the system.

This is normal and expected behaviour, and any technological gear added to the technology stack in order to minimize this risk will make the system more complex and therefor more prone to new accidents.

For instance, I find that two of the most complexing things you can do to an IT system are clusters and SAN's.

These impressive technologies are always added in order to make systems more available and guard against unexpected accidents.

Hence, they will, in and by themselves, guarantee other normal accidents to happen to the system.

Complexing and de-complexing IT systems
=======================================
So you could say that it's a question of complexing or de-complexing IT systems.

I have found four situations that can complex IT systems (I'm being a bit ironic here):

1. To cover yourself (politics).
2. Exploration.
3. SMS decisions.
4. Architects.

1. Reason One: To cover yourself (politics)
===========================================
You might want to complex systems in order to satisfy various parties that you depend on or who insist on buying certain things they've heard about at vendor gatherings:

"Yes, we've done everything humanely possible, including buying state-of-the-art technology from leading vendors and asking independant experts to verify our setup".

This is known as CYB (Cover Your Behind).

2. Reason Two: Exploration
==========================
Ah, the urge to explore unknown territories and boldly go where no man has ever gone before...

Because you can.

The hightened awareness thus enabled might be A Good Thing for your system and your customers.

It could also create situations that you and others find way too interesting.

Reason Two is often done by men, because we love to do stupid or dangerous things.

3. Reason Three: SMS decisions
==============================
A third reason for complexing IT systems could be pure ignorance in what is commonly referred to as Suit Meets Suit (SMS) decisions - where a person of power from the vendor side with no technical insight talks to a person of power from the customer side with no technical insight.

These SMS situations tend to cause considerable increases in the GNP (just like road accidents and fires) of any country involved because of all the - mostly unneccessary - work following.

The costs to humans, systems and users can be enormous. Economists tend to love it.

4. Reason Four: Architects
==========================
A fourth reason for complexing IT systems can be architects. Don't get me wrong: There are many good IT architects. The very best ones, though, tend not to call themselves architects.

One of my dear friends once stated that an architect is often a developer that can't be used as a developer any more. Very funny.

However, what I have witnessed myself is that the combination of getting further away from the technical reality and getting closer to the management levels (the C class, as it were) tend to make some architects less good at making architectural decisions after a while.

That's where the vendors get their chance of selling the latest and greatest and thus complexing new and upcoming systems.


Summary: The end of reasoning
=============================
Four reasons must be enough. There are probably more, but I cannot think of them right now.

Anyway, imagine what savings in costs and worries you can obtain by moving just a notch down that steep slope of complexity in your system.

You might be able to de-complex your system to a degree where it becomes
absolutely rock solid and enormously available.

That should be our goal in the years to come: To help our customers de-complex their systems, while of course trying everything we can to support those who chose to complex theirs.

14 Comments:

Blogger Niall said...

Moans

I believe you missed out another good economic argument known as Competitive Pressure as in

"All our competitors are doing this, when can ours be implemented."

Cheers

Niall

4:11 AM  
Blogger CJ said...

Darling, what a nice, compact post. What an 'AHA Erlebnis' is this.

The SMS decision:
One of my customers, a world-wide spread multinational, many, many plants running decentralized nice systems based on Oracle EE, Data Guard, DR sites, most of the time everything works fine, got this CIO who met an IBM guy at the golf course. Now they consider seriously to move to DB2. Not to centralize (they're amidst rolling out new software everywhere), but because it's better, cheaper. Yes, any manager isn't worth his salary if he doesn't manage fundamental changes, no matter what. Three generations of managers further we'll see everything changed to what it is now. It's just like fashion: If you wait long enough your current suit will be fancy again.

The end of reasoning.
I have this one customer, a rather smallish company in headcount (half of the dozen in the IT dept), seriously big in turnover. They have knocked their main system down to two Oracle databases, each running half of the load but capable of one-and-a-half of it, Oracle SE. Two buildings, 30km apart. Truly 24x7, I reckon they make 100% for several consecutive years now. That includes HW/SW/DB upgrades! Application can dynamically switch to 'the other' database, under water the databases synchronize by simply forwarding transactions to each other. Math: 20 minutes of downtime costs serious money, but less than a so-called 'true' HA investment. Application: home-grown, HA woven into the whole stack. HW: simple X86 boxes, DAS in stead of SAN, no cluster. no complexity, but also no outages. Savings: not investing all the top of the bill HW/SW, Result: no outages for several years.

Do the math, de-complex.

Carel-Jan

3:17 AM  
Blogger Daniel Fink said...

I am going to have to reread that book.

An absolutely brilliant post! Very thought provoking.

7:42 AM  
Blogger Daniel Fink said...

I was just at an Avalanche Awareness Clinic and the instructor had a quote that is appropriate.

"There are no new accidents, just variations on recurring themes."

11:58 AM  
Anonymous Anonymous said...

> Architects
They do not operate the systems. They do not even stay on the project till the implementation phase. They just fire-off their slide-ware.. and forget.
They have simply no motivation whatsoever to "design as lean as possible", keep it super simple and performant.
The "Architect Role" as it is now usually defined is the ultimate danger to successful implementation and operation of the system.
...
How many times I have heard them recommending things like Oracle RAC (or at least fail-over clusters).. just to end-up the discussion with unsubstantiated claims like "customer wants it.."

1:35 PM  
Blogger Alex Gorbachev said...

Well said Mogens!
IMHO, CYB is the most common so it deserves to be the first.
BYC (Because You Can) has one positive attribute - it stimulate adoption of new techniques and technologies. Or not. I wish that BYC would be the most common but life is life.

By the way, there is one more reason that seems rather obvious and natural to me - ignorance that is not admitted. It's especially "popular" amongst consultants who tend to make bold statements just to get that contract.

7:48 PM  
Blogger weddingdresses said...

I just have this energy level that I've never had before. The only thing that's changed is I've started wearing my Power Balance and nike running shoes
Power Balance wholesale

11:27 PM  
Anonymous price per head online said...

Thank you very much for letting me comment, tell it is a pleasure for me to have the opportunity to read this article. Today I read a lot and I assure you that all day had not read anything as interesting as this, I found it very effective information. THANKS

7:35 PM  
Anonymous manhattan locksmith said...

Thanks, Gannett Web site has collected data "the newsroom of the future." It will benefit all the people. appreciate the idea. and Yours truly

4:52 PM  
Blogger petersmith said...

These impressive technologies are always added in order to make systems more available and guard against unexpected accidents.

regards
Pawnbrokers

10:31 PM  
Blogger petersmith said...

The "Architect Role" as it is now usually defined is the ultimate danger to successful implementation and operation of the system.
Pawnbroker

11:30 PM  
Blogger 柯云 said...

2016-01-08keyun
louis vuitton purses
cheap toms shoes
ray bans
michael kors outlet
louis vuitton handbags
michael kors outlet clearance
oakley sunglasses
michael kors outlet
cheap oakley sunglasses
cheap toms
hollister clothing
north face
ray ban sunglasses outlet
coach outlet
cheap ugg boots
michael kors outlet sale
north face jackets
swarovski crystal
hollister kids
louis vuitton handbags
louis vuitton purses
ugg boots
uggs outlet
ugg boots sale
ralph lauren outlet
oakley sunglasses cheap
air force 1 trainers
gucci handbags
uggs for women
fitflop clearance
ugg outlet store
cheap toms
louis vuitton handbags
coach outlet store
discount ugg boots
ugg australia
air max 95
chanel outlet
chanel purses
nfl jerseys

11:53 PM  
Blogger oakleyses said...

oakley sunglasses, michael kors outlet, louboutin outlet, prada handbags, air max, burberry outlet, louis vuitton outlet, ray ban sunglasses, tory burch outlet, cheap uggs, michael kors outlet, michael kors outlet, chanel handbags, burberry outlet, oakley sunglasses, kate spade outlet, uggs outlet, oakley sunglasses cheap, prada outlet, longchamp handbags, gucci outlet, air max, louis vuitton, ray ban sunglasses, nike shoes, jordan shoes, nike free, polo ralph lauren, michael kors outlet, polo ralph lauren outlet, ray ban sunglasses, oakley sunglasses, michael kors outlet, christian louboutin, tiffany and co, louboutin, tiffany and co, rolex watches, longchamp outlet, louis vuitton outlet, longchamp handbags, ugg boots clearance, michael kors outlet online sale, uggs, ugg outlet, louis vuitton outlet stores, louboutin shoes, replica watches, louis vuitton handbags

5:00 PM  
Blogger Mobile app development company said...

Excellent post. Thanks for sharing. Mobile Application Developers in India are proficient professionals specializing in crafting, designing, and upkeeping applications for mobile devices, including smartphones and tablets.

2:55 AM  

Post a Comment

<< Home