The evils of over-engineering

My reply to a great Slashdot comment: http://it.slashdot.org/comments.pl?sid=264749&threshold=3&commentsort=0&mode=nested&cid=20164481

Ugh, Java frameworks...Somebody needs to drag the people who make these things in a room, erase their memories, and make them use what they have created...

The end result is you end up with what should be a fairly simple task (like OO-relational mapping) have 400 page manuals because it ends up doing every little thing that people want to do. In the time it takes you to choose the right framework, download and install the binaries, wade through the required config files, sift through the quickstart, and actually get familiar with how it is done, you could have just written and tested the tedious JDBC code to load and unload an object from the database.

You know, this seems to work both ways. Ever worked with a homebrew framework that didn't have good exception-handling or some type of passable tracing? The other end of the spectrum from these IBM Java guys are the people who still hardcode strings everywhere in the system and hand-code ALL of their stored procs and data access layer.

There are actually two problems here:

1. The Principle of Least Threshold


2. Enterprise Hubris

From the top, Javaheads like your above description are trapped in the Enterprise way of thinking. And hey they're building apps that will process more transactions in one day than many other small business apps will ever process. They have the right to be a little proud, the problem is that they're not very good at "scoping downwards". These frameworks all scaled up and came together, but now they have 400 pages of documentation and 12 different config files and everything is interdependent b/c they didn't want to lose functionality.

Even written a good business logic layer (BLL)? Nor have I, nor have like 90%+ of programmers, but it's ok, most of them never need it. The BLL is just useless abstraction for the typical database front-end that we develop. BLL just gets smushed in with Entities and UI b/c it's pointlessly complicated to develop on all but the biggest systems.

On the flip side, you get the Principle of Least Threshold, which is my term for the phenomenon where the perception of quality is limited by the highest level quality one has experienced. There are millions of small developers out there (VB 6 was the second most-used langauge at one point), who have never worked with any thing close to these monstruous Struts frameworks. So they don't understand them and they don't need half of the functionality and they can't spend 4 months figuring out how the system works, so they just build their own system one piece at a time and keep re-creating the parts of the wheel that they actually need.

I'm living this, our Data Objects framework doesn't have tracing. Nobody cares but me, b/c I'm the only one who knows what a godsend it is to have instrumentation code. Just yesterday, wasted 2-3 hours diagnosing a problem that would've taken 10 minutes if we'd had minimal tracing (queries?).

Of course, the real solution lies between the two camps. The Javaheads with the arcane infrastructure and architecture knowledge need to enable mechanisms to scale the complexity of their frameworks. The other guys need to start pushing from the other direction. But most of all the leaders in the field need to learn to embrace differences and change (man that sounds corny).

When anyone starts spouting off about good software or bad software, they must at least identify the scale of the software with which they are working (and when was the last time we saw that). Unacceptable mistakes in an Enterprise Framework are just ignored in a 12-user system. In fact nearly every piece of code posted to the web is unacceptable in an Enterprise environment, b/c it fails to catch errors or instance tracing or define attributes or use fully qualified types or validate inputs or even comment in a way that is compatible with JavaDoc/NDoc/Sandcastle.

A twenty line class in an Enterprise app could have 80 lines of associated "overhead code" that just plugs the class into the framework. Is this elegant and beautiful or just clunky and cumbersome? That answer could vary by system size, by programmer skill or even by personal taste. But no one seems to want to acknowledge this reality. The small guys call the Enterprise guys "clunky" and they in turn call the small guys "clumsy". So in the end they just ignore each other and throw rocks in each other's general direction and nobody stands and up and says "Hey guys, it's a continuum, it's not black and white, our solutions can only be inspirations for yours."

We need new leaders. The guys in my CoDe & MSDN magazines aren't sitting around handing out three different solutions of different complexity for the same problem. They're writing one-solution articles to solve problems that happen at all kinds of levels. How do you store user-specific data in a web app? Well in this article we'll talk about how you can solve this problem using my framework that doesn't actually work with your chosen method of storing data... if stuff isn't in an SQL Server-persisted Session then you're SOL... blah... blah... blah..." thanks guys. These are the same guys that write articles showing drag 'n drop data connections and then wave their hands about storing encrypted connection strings in the config files or using parameters to protect from injections.

Point is, we as programmers need to get off our high horses and actually learn about other people's problems so that we can learn to develop beautiful solutions to the problem rather than just solutions.