Wednesday, January 02, 2008

Perrow and Normal Accidents

While reading the book 'Deep Survival' (most kindly given to me at the UKOUG conference in Birmingham by Sir Graham Wood of Oracle after the fire in my house) I happened on a description on page 107 of a book called 'Normal Accidents' by a fellow named Perrow (get it? per row - a perfect name for database nerds).

Perrow's theses is that in any tightly coupled system - in which unexpected interactions can happen - accidents WILL happen, and they're NORMAL.

Also, he states that technological steps taken to remedy this will just make matters worse.

Perrow and IT systems
I have freely translated Perrow's thoughts into the following:

IT systems are tightly coupled. A change - a patch, a new application, or an upgrade - to a layer in the stack can cause accidents to happen, because they generate unexpected interactions between the components of the system.

This is normal and expected behaviour, and any technological gear added to the technology stack in order to minimize this risk will make the system more complex and therefor more prone to new accidents.

For instance, I find that two of the most complexing things you can do to an IT system are clusters and SAN's.

These impressive technologies are always added in order to make systems more available and guard against unexpected accidents.

Hence, they will, in and by themselves, guarantee other normal accidents to happen to the system.

Complexing and de-complexing IT systems
So you could say that it's a question of complexing or de-complexing IT systems.

I have found four situations that can complex IT systems (I'm being a bit ironic here):

1. To cover yourself (politics).
2. Exploration.
3. SMS decisions.
4. Architects.

1. Reason One: To cover yourself (politics)
You might want to complex systems in order to satisfy various parties that you depend on or who insist on buying certain things they've heard about at vendor gatherings:

"Yes, we've done everything humanely possible, including buying state-of-the-art technology from leading vendors and asking independant experts to verify our setup".

This is known as CYB (Cover Your Behind).

2. Reason Two: Exploration
Ah, the urge to explore unknown territories and boldly go where no man has ever gone before...

Because you can.

The hightened awareness thus enabled might be A Good Thing for your system and your customers.

It could also create situations that you and others find way too interesting.

Reason Two is often done by men, because we love to do stupid or dangerous things.

3. Reason Three: SMS decisions
A third reason for complexing IT systems could be pure ignorance in what is commonly referred to as Suit Meets Suit (SMS) decisions - where a person of power from the vendor side with no technical insight talks to a person of power from the customer side with no technical insight.

These SMS situations tend to cause considerable increases in the GNP (just like road accidents and fires) of any country involved because of all the - mostly unneccessary - work following.

The costs to humans, systems and users can be enormous. Economists tend to love it.

4. Reason Four: Architects
A fourth reason for complexing IT systems can be architects. Don't get me wrong: There are many good IT architects. The very best ones, though, tend not to call themselves architects.

One of my dear friends once stated that an architect is often a developer that can't be used as a developer any more. Very funny.

However, what I have witnessed myself is that the combination of getting further away from the technical reality and getting closer to the management levels (the C class, as it were) tend to make some architects less good at making architectural decisions after a while.

That's where the vendors get their chance of selling the latest and greatest and thus complexing new and upcoming systems.

Summary: The end of reasoning
Four reasons must be enough. There are probably more, but I cannot think of them right now.

Anyway, imagine what savings in costs and worries you can obtain by moving just a notch down that steep slope of complexity in your system.

You might be able to de-complex your system to a degree where it becomes
absolutely rock solid and enormously available.

That should be our goal in the years to come: To help our customers de-complex their systems, while of course trying everything we can to support those who chose to complex theirs.


Blogger Niall said...


I believe you missed out another good economic argument known as Competitive Pressure as in

"All our competitors are doing this, when can ours be implemented."



4:11 AM  
Blogger Carel-Jan said...

Darling, what a nice, compact post. What an 'AHA Erlebnis' is this.

The SMS decision:
One of my customers, a world-wide spread multinational, many, many plants running decentralized nice systems based on Oracle EE, Data Guard, DR sites, most of the time everything works fine, got this CIO who met an IBM guy at the golf course. Now they consider seriously to move to DB2. Not to centralize (they're amidst rolling out new software everywhere), but because it's better, cheaper. Yes, any manager isn't worth his salary if he doesn't manage fundamental changes, no matter what. Three generations of managers further we'll see everything changed to what it is now. It's just like fashion: If you wait long enough your current suit will be fancy again.

The end of reasoning.
I have this one customer, a rather smallish company in headcount (half of the dozen in the IT dept), seriously big in turnover. They have knocked their main system down to two Oracle databases, each running half of the load but capable of one-and-a-half of it, Oracle SE. Two buildings, 30km apart. Truly 24x7, I reckon they make 100% for several consecutive years now. That includes HW/SW/DB upgrades! Application can dynamically switch to 'the other' database, under water the databases synchronize by simply forwarding transactions to each other. Math: 20 minutes of downtime costs serious money, but less than a so-called 'true' HA investment. Application: home-grown, HA woven into the whole stack. HW: simple X86 boxes, DAS in stead of SAN, no cluster. no complexity, but also no outages. Savings: not investing all the top of the bill HW/SW, Result: no outages for several years.

Do the math, de-complex.


3:17 AM  
Blogger Daniel Fink said...

I am going to have to reread that book.

An absolutely brilliant post! Very thought provoking.

7:42 AM  
Blogger Daniel Fink said...

I was just at an Avalanche Awareness Clinic and the instructor had a quote that is appropriate.

"There are no new accidents, just variations on recurring themes."

11:58 AM  
Anonymous Anonymous said...

> Architects
They do not operate the systems. They do not even stay on the project till the implementation phase. They just fire-off their slide-ware.. and forget.
They have simply no motivation whatsoever to "design as lean as possible", keep it super simple and performant.
The "Architect Role" as it is now usually defined is the ultimate danger to successful implementation and operation of the system.
How many times I have heard them recommending things like Oracle RAC (or at least fail-over clusters).. just to end-up the discussion with unsubstantiated claims like "customer wants it.."

1:35 PM  
Blogger Alex Gorbachev said...

Well said Mogens!
IMHO, CYB is the most common so it deserves to be the first.
BYC (Because You Can) has one positive attribute - it stimulate adoption of new techniques and technologies. Or not. I wish that BYC would be the most common but life is life.

By the way, there is one more reason that seems rather obvious and natural to me - ignorance that is not admitted. It's especially "popular" amongst consultants who tend to make bold statements just to get that contract.

7:48 PM  
Anonymous james said...

Subject Line: Beat Long Poll Lines with Absentee Ballots from
Many state and local election officials are encouraging voters to use Absentee Ballots to avoid the long lines and delays expected at the polls on November 4th due to the record-breaking surge in newly registered voters.
Voters in most states still have time to obtain an Absentee Ballot by simply downloading an official application form available through, a completely FREE public service from the nonprofit State Democracy Foundation.
Read More:

5:53 AM  
Anonymous Online forex info said...

Nice post! yesterday i found another great post about forex. Here is the link
online forex info

10:36 AM  
Anonymous Industrial Accidents said...

Great post! yesterday i found another great video post about Industrial Accidents. Here is the link
Industrial Accidents

1:44 PM  
Blogger guangming said...

wholesale nike shoesShop a great selection of authentic Nike shoes&Nike Air Max with reasonable price for the entire families at shoes 100% quality guaranteed and smooth customer service.UGG Women's Classic Cardy Boots 5819 are available with colorful knit uppers (composed of a wool blend) and a sheepskin sock liner for extra comfort.ugg boots It is detailed with three oversized wood buttons, allowing it to be styled buttoned up, australia uggslouched down, slightly unbuttoned, or completely cuffed down. They have a light and flexible EVA outsole along with a suede heel guard provides durable wear all season long. That is why it is one of several styles that have been all time favorites with women.

8:03 PM  
Blogger pandorasell said...

Clip flees and this Chrismas costs buy pandora jewelery coming near. Surprise! Most belated cheap pandora jewelery at Discount Price I plan to explore the net because an allow Christmas gift discount pandora jewelery .In the online search appendage, I unexpectedly read a rattling occupying pandora bracelets and unbelievable story, and I hence besides chose pandora silver bracelets blimey Christmas Day empowers.A-list gross revenue from colligates from london sweetheart cheap pandora bracelets Decades ago in London, a small restaurant enjoyed good business concern and attacted a lot returned discount pandora bracelets buyers.called for to express thanks to these pandora bracelets sale ,

8:35 AM  
Blogger weddingdresses said...

I just have this energy level that I've never had before. The only thing that's changed is I've started wearing my Power Balance and nike running shoes
Power Balance wholesale

11:27 PM  
Anonymous Anonymous said...

It's Friday morning and I'm on my way away from San Francisco after a splendid week of OOW, good guys, a few beers, and a lot of tech talk.Cheap Soccer Shirts | Cheap Football Shirts | henry 12 arsenal jersey | messi jersey 10 argentina | fabregas jersey arsenal | david villa barcelona youth jersey | benzema soccer jerseys | cristiano ronaldo real madrid jersey | cheap rooney soccer jerseys | kaka soccer jersey | real madrid shirt | new fc barcelona jersey | argentina jersey 2012 | Brazil jerseys wholesale | cameroon jerseys wholesale | england jerseys sale

6:57 PM  
Anonymous price per head online said...

Thank you very much for letting me comment, tell it is a pleasure for me to have the opportunity to read this article. Today I read a lot and I assure you that all day had not read anything as interesting as this, I found it very effective information. THANKS

7:35 PM  
Anonymous manhattan locksmith said...

Thanks, Gannett Web site has collected data "the newsroom of the future." It will benefit all the people. appreciate the idea. and Yours truly

4:52 PM  
Blogger petersmith said...

These impressive technologies are always added in order to make systems more available and guard against unexpected accidents.


10:31 PM  
Blogger GuildWars2Items said...

Death, old age, are words without a meaning, that pass by us like the idle air which we regard not Diablo 3 Gold Sale, It takes strength to do what must be done when the work is unpleasant and uncomfortable Diablo 3 Gold, it takes strength to persist in the face of obstacles, when it would be much easier to simply give up Buy Diablo 3 Gold.
buy c9 gold you laugh at me for being different, but I laugh at you for being the same, the past is gone and static. Nothing we can do will change it. The future is before us and dynamic. Everything we do will affect it cheap c9 gold, don't go around saying the world owes you a living. The world owes you nothing. It was here first c9 money.

1:31 AM  
Blogger petersmith said...

The "Architect Role" as it is now usually defined is the ultimate danger to successful implementation and operation of the system.

11:30 PM  

Post a Comment

Links to this post:

Create a Link

<< Home