Saturday, September 09, 2006

A Year of WTFs

Each year, the Oracle WTF organisation celebrates its anniversary by inviting a fellow Oracle professional to share an evening of fine Eastern European lager and disappointing bar snacks in a pub where you can't hear each other properly, and this year it was the turn of former newbie (but now of course senior expert) DBA Lisa Dobson.

Actually that's not quite true - I realised today it was just over a year ago that we started the blog, and the other evening some of us met up with Lisa who happened to be in town, and if I'd thought of this a bit earlier I could probably have pretended it was all planned. Anyway it was nice meeting Lisa, and we enjoyed the story about the colleague who accidentally deleted the entire production server, as luck would have it a week after the server room air conditioning took out the only other server by dripping a surprisingly large amount of water into it. That rather outdid my story about the day we turned up for work at a client's site and found nobody could log in, because as it happened someone had stolen the servers during the night. Or the other one, if I'd remembered to tell it, about the high street retail chain whose backup system involved a PC and a timer plug. (I shouldn't laugh - it actually worked quite well.) It was also reassuring to hear we are not the only ones who don't understand a single bloody word of those "Oracle" blogs about installing a Java framework in JDeveloper to implement service-oriented BPEL with a right-click and some XML. Umm, neat.

It seems we're also not the only ones to have been shocked and saddened by Doug Burns' recent senseless killing spree. This monster must be stopped. (...is probably what Doug said as he reached for his copy of "Expert One-On-One Oracle").

Anyway, one year, 63 posts, and some lessons learned:

  1. Post one of Tom Kyte's pet hates like WHEN OTHERS THEN NULL (or WHEN OTHERS THEN DBMS_OUTPUT, which isn't much better quite frankly) for a mention on his blog and a surefire thousand hits. We need one about IM Speak now b/c im sure u will C loadsa hits ;-)
  2. Post something involving security for a mention on Pete Finnigan's site. We also get a steady stream of visitors googling for "forgot system password +oracle", "how to get password of a user in oracle" and so on, and being directed to Umm, I Forgot My Password, part 2. (In future just put it on a Post-It under your keyboard like everyone else.)
  3. Titles involving common Oracle error codes seem to do well. A lot of visitors come here from a Google search for SP2-0552 (guys, you have a variable with a colon in front of it somewhere). We need some posts about ORA-0600 or TNSNAMES.
  4. A lot of people seem to be searching for an explanation of joins and end up at our Joins Explained, heaven help them, where a Mr Sanders Kaufman explains things like "Left Joins are joined on the left columns of two tables. Right Joins are joined on the right columns of two tables." I'm glad we got that cleared up.
  5. Google Analytics Rock. So (while we're at it) do Statcounter, Feedburner and CoComment.

We were recently offered some sponsorship by price comparison website Shopzilla (or rather, they emailed me a week ago and I remembered it in the pub). The consensus among those present was that we should not accept it, although personally I think all that would change if we could get The Daily WTF's Beanbag Girl.

We were also kindly offered a spot on Pythian Group's Logfile Of The Vanities or whatever it's called (or rather, they emailed me a month ago and I seem to have deleted the message, sorry guys, meant to get back to you) but I didn't take them up on it, partly because there didn't seem much point in writing a piece telling people about Oracle blogs they already read (surely?) via blogs.oracle.com, and also because it's explicitly for DBAs, and with the possible exception of Padders we are all developers (even though these days "developer" is often assumed to mean someone who is into aspect-oriented JDeveloper BPEL plug-ins). Our main thing is SQL, PL/SQL and data modelling, and AFAIK none of those have much to do with the DBA role. In fact, thinking about it I'm not sure why they asked. And is that Pie-thian as in Pythagoras and Python, or Pith-ian as in Gryffindor and Slytherin, anyway?

32 comments:

Scott Swank said...

I don't know what you have against BPEL. I things it's a terrific solution in search of a problem. 'Course I have to admint that it's ads in Craig's List are always kind of pitiful.

Sensitive Service-Oriented Workflow language in search of corporation. Likes long walks on the beach and vanilla chai.

Y'all stop bye if you're in Vegas.

Dave Edwards said...

"...there didn't seem much point in writing a piece telling people about Oracle blogs they already read (surely?) via blogs.oracle.com..."

Log Buffer covers not just Oracle, but any technology relevant to database professionals. It is intended to apply a human, rather than a mechanical, filter to the database blogosphere, and to relate a point of view -- that of its editors.

"In fact, thinking about it I'm not sure why they asked."

The more perspectives, the better. As the duties of DBAs may often involve data modelling, I think your point of view would be of interest to administrators. Also, it's good to at least occasionally stick our heads outside our own silos and see the landscape from a broader perspective.

"And is that Pie-thian as in Pythagoras and Python, or Pith-ian as in Gryffindor and Slytherin, anyway?"

It's Pith-ian as in Apollo and Delphi. More specifically, our name relates both to the Oracle at Delphi, who was also called the Pythian Priestess, and to an early ancestor of the Olympic Games called the Pythian Games, which held competitions in music and poetry as well as athletics. Some of our branding (such as "database symphony") comes from that.

Thanks for following up. Of course, the invitation still stands, to you and to anyone else who would like to try their hand at Log Buffer.

Cheers,
Dave Edwards.
Log Buffer

William Robertson said...

Thanks for stopping by.

Regarding DBAs being involved in data modelling, I don't know if it's just the Oracle world or the UK industry, but I've been in this business 17 years, most of it consulting and contracting in one form or another, and I have never heard of a DBA doing data modelling, or for that matter writing code. Don't they have database administration work to do? (Does a Unix admin model Java apps?) I remember one (he was pretty good, too) who asked to be reminded how you got today's date in SQL.

Then again I'd never heard of anyone saying "Bee-Lob" until Eddie Awad's bizarrely great post a few months back so perhaps I've just lived a sheltered life.

Doug Burns said...

Dear, dear, dear ... Are you just *trying* to wind me up? ;-)

>>and I have never heard of a DBA doing data modelling, or for that matter writing code.

You know, I'm really starting to get wound up about this. I thought this whole DBA/Developer thing was just a bit of fun, but I'm astounded to find that people really believe that because I'm a DBA I don't end up writing loads of SQL (and PL/SQL for that matter) because a) the developers sometimes aren't up to the job and b) I need it for myself.

As for data modelling, isn't that much more a DBA job than a developer's? (maybe 50-50.)

I'm utterly confused by this definition of 'DBA' that seems to be floating around that seems to amount to database creation, backup and recovery. I'm perfectly prepared to accept it's me that's missing something here, but I just don't recognise myself in that image. Frankly, I wouldn't be interested in this software installation, recovery, create user (etc, etc.) role.

As for :-

>> It seems we're also not the only ones to have been shocked and saddened by Doug Burns' recent senseless killing spree.

My position here is both clear and consistent. It was a spider. It walked across my living room carpet. I have no feelings towards it other than fear. It's dead.

Well, there is one more thing. I'd welcome personal contact details from the various spider apologists so that next time I can preserve my books in pristine condition whilst you all carry them far away from my house in beer-mats and glasses ....

P.S. A small (polite) request ... Any chance of opening up comments to people without them needing a Blogger account?

P.P.S. Don't listen to a word that Lisa Dobson says, particularly about me!

Doug Burns said...

>> I remember one (he was pretty good, too) who asked to be reminded how you got today's date in SQL.

Erm, no he wasn't! ;-)

William Robertson said...

Of course there is way more to the DBA job than database creation, backup and recovery. I certainly don't meant to have a go at DBAs and I apologise if it sounded like a criticism. There is the whole proactive instance tuning thing for a start. It's just that surely requirements for new functionality come into the development team. The DBA team is in a whole different department, and I've never seen an organisational structure where design requirements get handed across to the DBAs then come back to development.

Perhaps there are more front-end driven applications out there where "developer" means Java/C++/Web developer and there are database specialists looking after the database, and in that setup perhaps they would be schema-designing code-writing DBAs. (I wish them luck.)

Regarding the guy who needed reminding how you got the current date, you're probably right. Now I think about it, he was also the guy who set PARALLEL 8 for every table in the warehouse one day (though to be fair, when everything ground to a halt he was quite quick to figure out why). He was way better than the developers on that project though.

> Any chance of opening up comments to people without them needing a Blogger account?

I've changed the comment setting. If we get flooded with spam it's going back on. (I'd prefer to avoid that tedious CAPTCHA thing if humanly possible.)

Doug Burns said...

I certainly don't meant to have a go at DBAs and I apologise if it sounded like a criticism.

>> Don't worry, I was only having a laugh ;-)

It's just that surely requirements for new functionality come into the development team.

>> Agreed

The DBA team is in a whole different department, and I've never seen an organisational structure where design requirements get handed across to the DBAs then come back to development.

>> I've definitely seen that in some places where DBAs would review the overall approach before the head-long rush into development. The initial requirements must be handled by development, but why shouldn't the people with specific database expertise be involved in the initial design stages? I tend to think we might end up with more supportable applications if that were the case more often. However, I need to qualify that by saying that developers seem to be far more database-savvy than they used to be, so maybe there's less need.

>> It's also increasingly clear to me that there are a whole bunch of people who are neither coders or DBAs, but somewhere in between.

I've changed the comment setting. If we get flooded with spam it's going back on. (I'd prefer to avoid that tedious CAPTCHA thing if humanly possible.)

>> As a victim of the most tedious spam, I understand your concern and you'll undoubtedly be inundated with it if you have word verification off, but I would much prefer people not have to sign up for a Blogger account just to leave a comment. Just my opinion, of course, but Blogger is not my favourite service :-(

Paul said...

It has been my experience that in some shops, DBAs do the data modeling, and even write the database interaction code, literally vending as an API to the development team. In other shops, the DBA has little to no involvement at this end of the business at all, except sometimes a QA review opportunity at certain junctures.

Both approaches can be successful. I myself have never had the opportunity to work as a DBA in the UK, but I can vouch for widely diverging strategies shop-to-shop throughout the USA and Canada.

Doug, being from the UK, seems to have a similar experience to mine, with substantial DBA involvement in data modeling work. So there may be some regional differences, and there may not, I'm not sure on this point. However, there is no doubt in my mind that most developers have by nature of their work focused their expertise on their development language, not their database platform target. As a result, the DBA can play a valuable role in a developer support function in those shops, essentially facilitating the successful application of the database platforms' strenghts. A great example of the sorts of skills that can be applied can be found in Kyte's Expert one-on-one book. The target audience is the developer. But in the typical shop, the people who know this stuff, and thus have an opportunity to contribute and teach it, is the DBA.

Cheers,
Paul

Paul

Paul Vallee said...

Sorry - that broken Paul blogger profile above is me.

Paul

Anonymous said...

Your solution is not going to work if you have someone with no business knowledge, no understanding of the data, and no background in data modelling building your data model.

Data modelling is a profession and a field of knowledge in itself, not something that you learn by osmosis because you've been an Oracle DBA for X years and know your features inside out.

Maybe a "DBA" might have some implementation ideas (clusters, partitioning, IOTs, indexing etc.), but actually "modelling the data", that has to be a job for an analyst/database programmer.

Now maybe you've lucked out and got some DBA there who also understands the finer points of the business and has great application knowledge. In my experience that would be very very very rare, and add another few "very"s if the organistion is one of any size or complexity.

You wouldn't have a nurse design your operating theatre because he/she has assisted a few heart bypass operations.

OraWTFGuy said...

I'm utterly confused by this definition of 'DBA' that seems to be floating around that seems to amount to database creation, backup and recovery.

Likewise, I'm astounded that the online Oracle community worships at the temple of Tom Kyte (an Oracle developer), enjoys the blogs of people like David Aldridge (an Oracle developer), yet considers an Oracle developer to be someone who codes pretty screens in J&*!, C++, C# etc and *sometimes* writes a poorly performing dynamic SQL statement without bind variables and insisting on 23 indexes.

There is a whole raft of developers out there, mostly in DWH/DSS I'd imagine, who do nothing but build Oracle database applications in SQL and PL/SQL. In the UK we call this being a "back-end Oracle developer". And in my experience, like William's, this includes nearly all of the developers I know. We model, we load, we code, we tune, we extract, we interact with business. And in the behemoth financial organisations and telcos that we work in, there is *always* a central operational DBA function. And it *never* touches the application, because each DBA is assigned about 48 Oracle, SQLServer, Sybase and DB2 databases to "look after".

My guess is that in the consultancies and software houses out there, the DBA/developer roles are much more integrated. But in the large commercial in-house teams out there, the developers completely own the applications and the DBAs support the infrastructure. And there are no pretty screens.

But I wouldn't feel too bad about this misconception. Oracle University doesn't think there's any such thing as a developer that "just does SQL and PL/SQL inside the database". To Oracle, a developer is someone who understands WTF BPEL or ADF stands for.

P.P.S. Don't listen to a word that Lisa Dobson says, particularly about me!
We'll ignore all the nice things she said about you then ;o)

PPS from me. Great post William. A year already. So much bad code out there and so few people willing to share it.

Scott Swank said...

Entertainingly the following just hit my email inbox.

I'm particularly intrigued by the requirement for 10+ years of Oracle server-side development. Why on earth wouldn't 5 years of solid development suffice. While I don't want to pick a particular cutoff threshold, I think that most of us would agree that understanding grows only incrementally past a certain number of years.

Then they ask for 2+ years of J2EE. Do they mean JSPs, Servlets, JSF, ADF, JDBC, EJBs, JTA, etc. And then do they want you to understand Tomcat, Resin, Jetty, WebSphere, Oracle 10gAS, etc. It's a bit like asking for 2+ years of "database".

---

Job Title : Oracle Developer (Lead)
Location : Tampa,FL
Duration : 6 months
Pay Rate : Open

Job Description:
-10+ years Oracle, including PL/SQL. Data warehouse/ETL a big plus.
-2+ years of J2EE experience
-Development Lead Experience

William Robertson said...

> most developers have by nature of their work focused their expertise on their development language, not their database platform target

Perhaps if the developers are not Oracle developers, say they are C++ guys or whatever, you might need someone with database experience and solid arse-elbow differentiation skills to do the data modelling, or risk ending up with some horrific EAV agile persistence dump, and in the absence of any Oracle developers, that might fall to the DBA. I've never worked on a project like that so I wouldn't know. That is certainly not how it works at sites like T-Mobile or HSBC from what I've seen, where the DBA role is as OraWTFGuy describes it and the idea of them doing application design and development is unheard of. That would be my job.

William Robertson said...

> posted by OraWTFGuy
btw, hope the next name you come up with is more original than the last two ;)

Covenant said...

'and we enjoyed the story about the colleague who accidentally deleted the entire production server'

funnily enough, that reminds of a story concerning one of the esteemed contributors to the oracle-wtf....

actually he didn't delete the ENTIRE database, just some of those datafile thingies.

Covenant said...

Scott said '
Then they ask for 2+ years of J2EE. Do they mean JSPs, Servlets, JSF, ADF, JDBC, EJBs, JTA, etc. And then do they want you to understand Tomcat, Resin, Jetty, WebSphere, Oracle 10gAS, etc. It's a bit like asking for 2+ years of "database".'

this is also a pet hate of mine. especially when they start listing stuff like sql*plus.

'so you have been driving for 10 years, how many years steering wheel experience do you have?...uh..huh...what about accelerator pedals, have you used them?...ok...have you got any experience with hardbrakes...sorry, oh yes I meant HANDbrakes, silly me'

aaarrgghh!

or the jobs that want 'in-depth' knowledge of Oracle, SQL Server, .Net , J2EE ( howver that is defined ), web services.

...or the stupid have you got 'full lifecycle experience' ..er..no Im not dead yet!

ahem. time to stop typing now.

Covenant said...

William said 'Of course there is way more to the DBA job than database creation, backup and recovery.'

I couldn't agree more. who else would come up with all those meaningless naming standards. like the DataWarehouse team at a major UK supermarket who dicated that all table and names should be 6 characters long and contain no vowels. ( I kid ye not !)

Seriously though, in my experience database design is never done by the DBA team. maybe it's a UK thing. or maybe it's just a control freak thing!

William Robertson said...

> I couldn't agree more
I'm with you there, I think, but I'm not sure what this has to do with the DBA role. In your example, wasn't it the Warehouse team that came up with the meaningless naming standard? Or are you saying the DBAs could have stopped them?

Covenant said...

I of course meant to say the 'Datawarehousing DBA team', the developers thought it was madness!

Doug Burns said...

I was going to say 'I turn my back for 5 minutes', but it's more like 3 days.

Don't get me wrong, I've worked in more sites where DBAs *aren't* involved in design other than physical implementation detail. I've also worked at sites (my current site is an example) that have a dedicated data modelling team, albeit a small one, and I've worked at sites where DBAs are involved. What I was arguing against is the concept that all significant sites work on the 'DBAs sitting in a corner writing documentation' model. Even at those sites, you have different levels of DBA doing different jobs and I get very concerned people might think that when I say 'DBA' in my sig file, it means I don't do database design or development and don't (constantly) rewrite cr*ppy back-end code.

However, I would say that we're all in a fortunate position when you work at a site with good back-end database developers and modellers. And, if you're so right about the level of knowledge of the *average* back-end developer (and I have to work with all levels), what could possibly explain the complete trash that results and how am I meant to explain developers coming to ask the DBAs how to do things?

The problem here is that we're assuming Tom Kyte, Mark Rittman, David Aldridge and you chaps are representative. Maybe if you were, I'd find I had too much time on my hands once I've finished the backups ;-) Trust me, I don't!

I know quite a few terrific database developers but they're sadly still a minority (something on which this very blog depends!)

(Oh, and for what it's worth, I've been a back-end developer in the recent past. What a joy that was, to work at a site that understood that concept. Clearly I'm capable of doing both jobs, as I suspect would be quite a few DBA colleagues.)

P.S. I've *finally* got round to adding you to my blog-roll. Sorry it took so long ;-)

Covenant said...

I think you are confusing roles with job titles.

you may be a DBA in the org chart, but if you are writing SQL or designing models then you are fulfilling the 'role' of a developer.

DBAs ( the role ) perform Database Adminstration ( the clue is in the name )

Developers ( the role) develop things ( SQL code, models etc ).

Hope this helps!

William Robertson said...

> the 'DBAs sitting in a corner writing documentation' model.

Now that's another thing I've yet to see ;)

Doug Burns said...

>> DBAs ( the role ) perform Database Adminstration ( the clue is in the name )

Thanks for that. Always wondered what it stood for. Looks like I'd better stop writing SQL and PL/SQL then ;-) Pray tell me, what happens when I'm better at writing SQL than the developers? (Not *all* developers, I might add!)

Sounds to me like I'm a DBA/Developer but now I'm all confused again. Which team do I work in, seeing as the teams appear to be very seperate?

>> Hope this helps!

I'm learning a lot, so thanks. At this rate, my existential crisis will become a mere blip.

Scott Swank said...

Here are things that I think are the exclusive purview of a DBA. If you're doing them I figure you're doing DBA work no matter what your title is.

1. Backup/recovery
2. Installation/upgrades
3. Patching
4. Instance-level performance monitoring & tuning

Here is work that I associate soley with database developers.

1. Application design
2. Data modeling
3. Application-level & statement-level performance/tuning

'Course lots of folk have jobs with one title and job descriptions that overlap with the other. Further, at far too many shops the DBAs are the only folk who are considered professionals and paid accordingly. Consequently they're often the only folk who know a damn'd thing about how the database works.

William Robertson said...

I wonder whether Unix admins have this conversation.

> Which team do I work in?
For development, whichever one gets development requirements assigned to it.

I'm not saying you or database administrators in general cannot or should not do non-administrative work, in fact I think it is extremely valuable to involve DBAs in the process. I'm just describing the companies I've worked in, where "administrator" tends to mean what it says. However if database development is actively discouraged, as it seems to be on sites like this one, any database-related tasks would presumably fall to the DBA.

If you're having an existential crisis, how do you think I feel when I see articles, blogs, training and conference programmes that assume I must be either a DBA (if I know anything about the database) or else a Web/Java/C# developer?

Covenant said...

Doug,

'What's in a name? that which we call a rose By any other name would smell as sweet'

not that many of the DBAs I've met smell all that sweet though ;)

Doug Burns said...

Scott said ...

(some pretty smart stuff that I agree with, then ...)

3. Application-level & statement-level performance/tuning

When you have really good developers. I'm sure that's a minority of sites I've worked at, though. Most DBAs I know are pretty good at SQL tuning and I seem to spend an inordinate amount of my time tuning SQL.

Actually, the *really* good developers understand the whole set-based 'thing' but I truly struggle to get most to see that. Think about it - isn't the whole basis of this blog the lack of understanding of Oracle shown by some developers?

I'll tell you what, though, most developers I've worked with are delighted to learn more about Oracle, but it's not their management's focus.

William (as usual) said some more smart stuff, and ...

However if database development is actively discouraged, as it seems to be on sites like this one, any database-related tasks would presumably fall to the DBA.

Maybe that's what it is. I work at lots of different companies - big and small - and there are times when being a DBA at a big company can be a *little* boring because your involvement is limited.

If you're having an existential crisis, how do you think I feel when I see articles, blogs, training and conference programmes that assume I must be either a DBA (if I know anything about the database)

Yep, excellent point and I know that's the way it is. Maybe that's more evidence to my argument, though - that the heavy weight Oracle knowledge at many (but definitely not all) companies is in the DBA team? I swear I *love* it when I meet Oracle-strong developers (I have witnesses) but they aren't all like you chaps. I promise.

I like Tom Kytes use of 'Database Technologist' (I've probably got that wrong, but it's something similar). You're either 'into it' or you're not. This blog is for people who are 'into it' - DBAs and Developers.

Hey, the more back-end development teams there are, the more likely I am to switch ;-)

covenant said ...

not that many of the DBAs I've met smell all that sweet though ;)

Okay, you win ;-)

William Robertson said...

> William (as usual) said some more smart stuff,

Wow, thanks for the compliment! Don't think it'll stop me winding you up about spiders though ;)

Doug said...

Well, this post has become a lot more relevant in the past few days.

It looks like I'll be starting a Senior (wait for it ...) developer contract in London in a matter of weeks.

So we should grab a beer and I can tell you stuff about that Lisa Dobson that would make your hair curl. (Well, perhaps not yours, but you know what I mean)

William Robertson said...

You're on.

Lisa said...

My lawyers are on standby.....

Doug said...

You have lawyers and stuff? Crikey ... I wish you'd said something earlier.