The only thing you should NOT ask Icelanders (and maybe Norwegians) to do...

I love to visit Iceland. Not because of the landscape (We Do Not Use landscapes), but because of the people there. Man, they're funny and good to talk to.

This time (last week, for two days only) I detected a deep, slow-burning anger in the folks up there, that I haven't felt before.

One of them - a very stylish, suit-dressed bartender in his 40's, who broke his principle about not talking politics with guests (hey, the bar was empty), said that he suspected there would be real riots in the fall of 2010 if the few guilty bastards were not punished for real in court.

The handful of real bastards have been driven out of Iceland, by the way: People simply spat on them when they met them on the street AND painted their houses and cars red at night. One of the bastards re-painted his house in its original color. Guess what happened the following night.

They all now live in the UK or Florida.

Fine and good. Now you know.

But this post is about the one thing you should never ask an Icelander to do: A list.

You see, back around year 900 several Norwegian vikings sailed to Iceland. They pretty much remembered everything: Clothes. Food. The ship. Oares. Live animals. Tools. Pen & Paper. You know - all the usual stuff for a 400 years voyage.

But whoever was in charge of The Norwegian Iceland Travel List forgot one thing, and whether they discovered it quickly (i.e. on the journey) or when they had settled in in their small mud huts, I do not know. But man, they must have told The List Guy a thing or two upon suddenly remembering what they had forgot:

Women.

Anyway, they apparently decided to do something about it, because DNA-tests of Icelandic women some years ago confirmed that they originate from the British isles.

The British scientist who found this out, and who was interviewed on Danish Radio, commented on the fact that Icelandic women look Pretty Damn Good by saying (rather drily): "They probably didn't take the ugly ones."

So now you know.

Oracle Closed World - now closed

It's Friday morning and I'm on my way away from San Francisco after a splendid week of OOW, good guys, a few beers, and a lot of tech talk.

We ran OCW four times from Monday to Thursday, and it was really good presenters we had talked into showing up:

Monday: Jeff Needham on processors and how Oracle runs on them. Opteron good. Nehalem good. A reporter named Kate was present in order to write about OCW. Code: 41.

Tuesday: Jonathan Lewis showing why the crowd were not experts. Ouch. Code 43.

Wednesday: Jeremiah Wilton about the Cloud, and especially the Amazon Cloud. He seems to know a good deal about Amazon. Code 24.

Thursday: Uri Shaft on counting eg. NDV in the optimizer, and some compression theory - and then Dan Norris & Greg Rahn about the Database Machine. Code 42.

And Kate's funny article about OCW appeared in the daily conference newspaper on Thursday. She got all the technical and non-technical stuff right - very impressive! She also gave away the secret location (Thirsty Bear on 661 Howard, upstairs), but thankfully only on the very last day of OCW :).

I truly enjoyed it, and so did several others, so we'll probably do it again next year.

Apart from that, it also appears that the guys from Miracle who were here with me (Morten Tangaa, Jesper Haure, Kaj Christensen, Claus Sørensen) got good things out of the conference.

While I remember it: Thank you to Victoria Lira, Lillian Buziak, and Justin Kestelyn for allocating a reporter for OCW, for managing the whole ACE Director thing, and many other favors that make the conference work.

Extra! Extra! Oracle Closed World today.... on Cloud

We had planned not to have any OCW presentations today in order not to steal Larry's audience from his planned keynote, but we're doing it anyway.

It's at 1200 hours, NOT 1300 hours as usual.

More details via text messages later, including todays codeword. If you want text messages from me for the OCW sessions, send me a text/SMS on +45 25277100.

Cloud computing is 'hot'. So is Larry when he talks about it on YouTube. Funny as Hell, actually.

There are at least these two videos. They are partly overlapping, but that doesn't matter- you'll want to see him do this standup routine a couple of times, trust me:

http://www.youtube.com/watch?v=0FacYAI6DY0
http://www.youtube.com/watch?v=8UYa6gQC14o

Which is why today, at the secret location, Oracle Closed World will present a couple of guys that know everything about 'the cloud'.

Mogens

Oracle Closed World - an underground conference...

I'm here in San Francisco for the Oracle Open World conference along with four other guys from Miracle, the two crazy Miracle Finland guys and some other crazy people - we've rented a couple of big apartments as usual, and are doing work, beer and other essential stuff together.

Last year at Oracle Open World (OOW) my friend Iggy Fernandez, who edits the NOCOUG (Northern California Oracle User Group) magazine/journal, suggested an Oracle Closed World conference, where REAL, TECHNICAL presentations would take place underground in secret locations, using secret passwords, and what have you.

Well, it's here. Monday, Tuesday and Thursdag at a secret location we'll do deep and very technical presentations about various topics. The secret location (which is indeed underground) has the capability to serve beer, by the way.

Let me know if you're interested in hearing more about OCW - email me on mno@MiracleAS.dk or text me on +45 2527 7100.

Mogens

Things you never wanted to know about SAN's...

Here's some information you will try to forget after reading. It explains why SAN's always cause trouble, why "a firmware upgrade" is really a complete change of an OS and therefor really dangerous (and impossible to plan or test for) and more.

From now on, think of the firmware in a SAN as a whole OS, just bigger. Scary, right?

My question to this very smart guy I know was this:

"Could you repeat what OS'es are used in what SAN's for me? And how many code lines the ExaData is using?"

=======================================================================

oh god...that is a huge question... First, Exadata software is small (less than about 100MB of bits)...but that is an unfair comparison to the glut of stuff in a full-featured array... Exadata has Linux underneath it, but then we execute about 98% User, 2% Kernel so really, the only thing we get from Linux is scheduling and I/O... Exadata is small because it doesn't do any of the fat stuff arrays like Clariion do (e.g., snapshots, remote mirror, etc).

Netapp's is called OnTap and it is a heavily developed BSD (Net/1 to be exact). It is huge and full featured as you can tell by how many add on packages it support, but just in protocol provisioning it is huge. Consider the fact that it can support front-end FC yet the LUNS are actually files in the WAFL filesystem! Wild.

Clariion OS is called FLARE and it sits on top of a full Windows distro (XP). EMC NAS (celerra) is called DART which is written from scratch.

HP StorageWorks Clustered Gateway is Linux +hundreds of thousands of very specialized PolyServe code.

EMC DMX OS is called Enginuity...DMX cpus are Power and I have no idea what the origin of this OS is. If I were a betting man I'd bet that it is scratch like DART.

IBM DS83XX is full blown AIX plus more (this is the old SHARK array) in fact, it is a cluster of AIX boxes in there...

I don't know what HP calls the stuff that runs inside EVA ...

as you can see...it is very confusing.
=======================================================================

Oracle Discoverer - help people write ugly code :)

There's been a discussion going on among some of my friends about all this horrible-looking (and often badly performing) auto-generated SQL coming out of Discoverer and other tools. Here are some of the comments made during the discussion, and some of my memories of how I got started with Oracle with the help of my good friend Mogens Egan...

=======================

Me:

"Oracle Discoverer - helping developers write ugly code for more than a decade."

=======================

NN:

"no no no!

the real beauty of discoverer (and similar tools) is not that it lets developers write ugly code, but it lets people who don't know what code is (business users), write code and share it with other users who also don't know what code is. It's entire purpose in life is to let people who don't know what they are doing, do it. developers do what they do with some understanding and can, sometimes, be educated. accountants and hr people can't."

=======================

Me:

"This brings me back. From 1987 to 1990 I was in a bank, sharing an office with Mogens Egan (the father of Morten Egan) and basically creating a datawarehouse (although we didn't know it) for internal users in the bank.

Our strategy was this:

1. Every night (or once a week or whatever) we would transfer data from the banks mainframe system via a SNA gateway to our VAX. The data came from IMS databases and was delivered as flat ASCII files (one physical record = one logical record) which often resultet in very very long records, of course, since IMS is hierachical. We would then load it into tables and let the users access it.

2. I would hold one- or two-day courses where I'd teach the attendees (who had probably only used a PC for a very short time) how to log onto the VAX using Smarterm, how to use VMS basic commands (including the editor), how to use SQL and SQL*Plus, how to create default forms in Forms 2.3 - and some other stuff.

3. Mogens Egan's idea was that it was better to turn users/experts (SME's in todays jargon) into "programmers" than vice versa. And then it should be our job to fix run-away jobs (read: SQL that performed bad or messed up things for others).

A rather anarchistic approach, you could say. But man, it worked. In three years we had 1000 users, some of who turned out to be natural super users, who started creating systems that helped their co-workers.

Since they were not officially named super users they couldn't demand to be given time to develop something they thought could be useful - they were by natural selection only allowed to spend time on something their co-workers thought useful.

Mogens and I are still in contact with many of those users. The machine is now an Alpha cluster, the data it manages runs a rather large banks' trading stuff, and all that - but its name is still Samson. And the super user we created back then is still called Supermule, which is the Danish name for Super Goof. With the introduction of English-speaking consultants in the last 10 years it has proved a minor mistake - they all ask "What's a super mule?"

So yes, we had many incidents of run-away jobs where the poor user had issued a SQL statement without the proper where-clause, etc. But then we would discover it, kill it, help the user - and all of the victims of this bad SQL knew it could be their turn one day, so they didn't get mad or upset.

That playground which we created back then generated a lot of Oracle-lovers who are still around in various higher positions, and perhaps it would have been even easier for them back then if we had had Discoverer.

So I think you're absolutely right: Discoverer will help computer-illeterates write really bad code even faster. But at least it gets them to use Oracle, and it creates wonderful problems that finances our fantastic lifestyles.

Mogens

PS: In the World as a whole, I think Discoverer had a presence (penetration) of about 2% of customers. In Denmark it was 20% due to my ex-wife Laila (Nathalie's mother), then product sales rep for Discoverer, who insisted that every single customer should have this product, like it or not. And notice how well Miracle is doing here. Perhaps there's a relationship."

===================================

SAPPU

SAP is a huge, mysterious, expensive animal.

In my very private opinion it is probably the worst ERP system you can buy today. Hence, most whiteshirts will choose it.

To compensate for the fact that it's old and silly technology, it's also exceedingly expensive. Introducing SAP to your company is the only reliable way to tell whether your company is so financially strong that it almost resembles a monopoly.

But what I really hate about SAP is that it removes people from the Oracle database field. I think most of us have experienced the following scenario:

A colleague or a bunch of colleagues are selected to help implement SAP. Until then they've been ordinary DBA's, fixing stuff, running databases and leading normal family lives.

Then they go away for EXTENSIVE training over a LONG period of time. In between the 42 week-long classes they have to take (per year, of course), they usually rest with their families and might show up for short, social functions among their (still) colleagues. But they have these myserious, far-away eyes... you can't quite reach them.

Then you get the famous message by mouth or email stating:

"We're now almost ready to , so for the next transition period of , I'll be working half of my time with my old stuff and half of my time with SAP before moving to full-time SAP obligations."

AND THAT'S THE LAST YOU EVER HEAR FROM THEM!

I think, but I could be wrong, that they're sucked into a place and time in space that the rest of us can't see or in other ways sense.

A Harry Potter-like parallel universe.

From which, mind you, they never return.

Numerous are the good Oracle DBA's who disappear from the real Oracle world this way.

Now it appears that in their parallel universe (the SAPPU, it could be called) they're slowly corrupted into thinking about Oracle databases the way the real (the few, the remaining) Oracle DBA's thought about databases in the 80's.

They're forced to unlearn all the right things they had learned. A kind of communist re-schooling or indoctrination. So sad.

Here follows a real mail thread from some friends of mine that know more about Oracle than most. They shall, of course, remain nameless - we still have no idea about the powers and general abilities of our disappeared friends in the SAPPU, and so a certain degree of fear for the unknown make us cautious....

======================================================================
Oracle Person One said:

I find myself once again embroiled in an SAP/Oracle Issue.

It seems I broke something in the production database by applying security patches, even though the 5 development and production servers have no such issues. (No, we can't afford a full system to test patches on - sigh...)

There are a number of ORA-7445 and ORA-600 errors being issued when DML is attempted, along with requisite trace files. (Yes, there is a an active SR already)

The SAP Basis team see's ORA-3113 and ORA-3114 errors.

I think I finally have them convinced to stop trying to 'fix' the ORA-3113...

Which brings me to the point: The usual method of troubleshooting SAP problems, as practiced by the SAP team, and nearly every SAP person I have worked with (not a terribly large sample - maybe I am just really lucky), goes something like this:

1. Search the SAP support site for every note containing ORA-600, ORA-7445. There are a few as you can well imagine.

2. Call a meeting to discuss which of the actions in these notes should be taken. Whether or not the contents of the note actually match the problem at hand seems to be irrelevant.

3. Ask the DBA to run a script (recommended by SAP support) to check for some problem or another in the data. This script will launch full table scans for every table in the database... Using a for/next loop. Fortunately for me, the script is broken as is. ... and I can't seem to find the problem with it.

4. Rinse and repeat - effectiveness is unimportant, only looking busy is important.

To memorialize this method, I have created a link to the following short, but accurately portrayed method of this troubleshooting methodology:

http://tinyurl.com/sap-troubleshooting-method
===============================================================
For that Person Two commented:

Make that, 'development and test'
===============================================================
Person Three need this off his chest:

So, regarding recent security patches that might cause havoc:

The one that prevents old client retries in clear text when the encrypted handshake is rejected affects some connection attempts.

“Repairing” the permissions of the archive log destination can ultimately get the archiver stuck far enough behind to toss a 7445 (I think) I didn’t look it up and since my databases never have problems I get rusty on the error messages. (tongue firmly in cheek). If memory serves (see previous sentence) one of the security patches suggests repairing the permissions on the archive log directory without telling you to make sure the ownership is correct.

What are your vintages? I’ve only done SAP stuff ONCE (and walked away quickly and quietly having proved that a certain physical reordering solved all their stated performance issues on the load testing system only to be informed that any manipulation of the data outside of SAP was not allowed.)
===============================================================
Person One felt he finally had someone to talk to who understood him:

Yup, that is SAP SOP.

So far I have refused to do that particular operation, except in a couple cases where it retrieved a lot of empty space due to archival of data.

The 'make work' analogy from "The Longest Yard" (the old one with Burt Reynolds and Eddie Albert) was used in reference to the "SAP Reorg" mentality when they last asked me to do a db reorg.

They didn't get it.

For those of you that don't already know it, the 'make work' for the prison workers in the facility where Reynold's character was incarcerated consisted of the ollowing:

Morning: shovel mud out of the swamp.
Afternoon: shovel mud back into the swamp.

By the classic definition of work, nothing was accomplished in the end.
But the inmates were still sweaty, tired, thirsty and hungry.
===============================================================
Person Five then finally could say this to a friendly crowd:

I wholeheartedly agree with both of you.

I spent almost 4 years as part of team that supported SAP. There were 3 dba's on the team, and both of the other two received their primary dba training from SAP. Their method was to manage EVERY database as if it were an SAP database.

I volunteered to do every non-SAP upgrade and rearranged everything back Oracle standards when they weren't looking. :) Took about 2 years to get to them all but it was well worth the trouble.

As for the SAP databases, patch application and upgrades always had different results on dev, test or prod. It was utterly baffling.
===============================================================

Ah well, based on all this, it is really no wonder that SAP is so wildly popular and has won the whole upper(ERP market) over less complicated, cheaper, more technologically advanced - and way more agile - competitors.

If you can sit in the local CEO club and claim: "We actually managed to pay for the WHOLE SAP implementation with our own money and we're still functioning in several departsments..." your fellow CEO's will know that you have more money than God or AIG's dealers.

You will have the uttermost respect from them, as they scramble to try and explain why their predecessors in their respective companies chose to implement something different from SAP. Cheaper, of course.

But one day, when they have gained enough financial strength....

Here's one final salute to all those lost colleagues from the Oracle database space. We'll always remember you. You'll never be forgotten. Long live your memory.

Wherever you are. In whatever lifeform.

Oracle Open World 2001 in Berlin: The Truth (Finally)

It’s time that we admit it. We did horrible things at OOW in Berlin. We’ve not told anyone for all these years, but the pressure is building inside. So I’ve decided to come clean.

We had just started Miracle, so we were only about eight folks or so in total. So we decided to go to the conference in Berlin all of us. We rented two or three apartments and also invited our friends (customers) to stay with us.

We drove down there in a few cars and found out upon arrival that the apartments were empty except for the mattresses on the floor. Oh well, easier to find your way around.

I’m still not sure why Peter Gram or someone else decided to bring along our big office printer/scanner/copier, but the guys quickly set up the network, the printer and the laptops, and then we just sat around, worked on the laptops, drank beers and talked about all sorts of Oracle internals.

I went down to registration and got a badge, so that was good. Then someone (forget who) came up with the idea that we should simply copy my badge so the rest of the guys could get in for free.

It wasn’t because we didn’t have the money or anything. Oh no. It was just because it sounded stupid and a little risky. So that’s why you’ll find pictures here and there (including in my office) of the guys copying and modifying badges.

The biggest challenge was that the badges had an “Oracle-red” stripe at the bottom.

But Oracle Magazine had a special conference edition out which had a lot of “Oracle-red” on the front cover, so it was just a matter of using the scissors in the Swiss army knife.

It worked perfectly for the whole conference and we were very proud, of course.

It was also the conference where I was introduced to James Morle by Anjo Kolk, our old-time friend from Oracle. I had placed myself strategically in a café/bar between the two main halls in the conference center which meant that everybody came walking by sooner or later. So I met lots of old friends that way. And a new friend named James Morle, who was in need for an assignment – and we had a customer in Germany who badly need his skills, so he ended up working for Mobilcom for half a year or more.

So the next bad thing we did was to crash the Danish country dinner. Oracle Denmark might not have been too fond of us back then, because they thought we were too many who had left in one go. Nevertheless, we thought it was not exactly stylish of them not to invite us to the Danish country dinner – as the only Danish participants.

Our friend (and future customer) Ivan Bajon from Simcorp stayed with us in the apartments and he was invited to the country dinner. So we found out where it was, snooped around a little, and then simply climbed a rather high fence and gate-crashed the dinner.

That was fun. The Oracle folks there were visibly nervous when we suddenly stormed in, but what could they do in front of all the customers, who very well knew who we were? So we sat down at the tables and had a good evening with all the other Danes there.

We had lots of fun during those few days in Berlin, had many political debates and beers, and went home smiling but tired.

To my knowledge we’ve not faked badges or gate-crashed country dinners since.

There have been a few suggestions since then that the badges we copied were actually free to begin with, but that can't possible be. I strongly object to that idea.