The Cost of Innovation

Early in October, I walked into the office and passionately proclaimed "We're going to migrate to Jboss 5.1". I'd been toying with the idea and researching its new features for a while. The new features (better enterprise level messaging, monitoring framework etc) combined with some limitations of the current version (4.2.x) made migrating an increasingly attractive option, some would say even a necessity. From a risk perspective, 5.1 couldn't be any worse than 4.2, just like 4.2 couldn't have been any worse than Geronimo 1.x.

For those lost in this version jargon, let's assume your application is a Lexus LS 450, then the application server is it's engine. So now picture you're trying to either upgrade the engine to a newer one from Lexus, or completely change the engine and retrofit it with that of a Cadillac. Imagine the mess, the grease, and the tinkering involved with something that is supposedly just a matter of nuts and bolts. Now add to that the malleability of software, and you might get an idea of the complexity.

In any case, the first task was to make the code base deploy on the new server, i.e., get the engine to sit inside the car. This itself was a huge undertaking for 5.1--itself a completely redesigned server--suddenly become a strict enforcer of the J2EE spec (the framework on which most enterprise grade Java applications are built). However, we persevered through the mess, the lack of documentation, the incomplete instructions, the cryptic user groups and got it to deploy--a week later. So the engine was in the car, now on to the road test. Here's where things got really interesting.

Before we go further, it's necessary to know that we're talking about a very scalable, stable enterprise application which as you read this is creating hundreds of campaigns per hour, processing between a few hundred to a couple thousand calls per minute, and serving hundreds of thousands of API requests per day. In short, we're talking about a fine-tuned, race-track tested performance automobile. It's definitely not a see-ma-what-i-made application.

Anyway, we run the first of our performance tests and the application fails badly. Hiccups like these are to be expected in any migration. Everything we read about 5.1 seemed to suggest there would be work involved making the source code either compliant with the new server or better equipped to tap into its many features. This of course took time, which we gladly invested. To continue the auto analogy, to help the new engine perform better, we were making changes to the chassis.

Regardless of the code changes the new system failed our (admittedly very stringent) performance tests. Perhaps it was our zealous belief in 5.1 as being a kick-ass platform, but we assumed that the problem is hardware. So we ended up building a server stack that mirrored our production configuration and re-ran the tests. The app performed better, but yet not at the level desired. It was at this point, that we began the entertain the idea that maybe, just maybe, the problem was in Jboss 5.1. There was only one way to test this--run the same code (with all its modifications) on the current version and compare. To our surprise (and some relief) the older version blew every performance test out of the water. Not only did it run fast, but it also ran a lot more efficiently i.e., no hogging up 400% of the CPU etc.

Disappointment would be an understatement. Not only had the next generation of application server platform failed miserably, we had spent almost a month on an initiative that wasn't going to work. Jboss 5.1 was turning out to be Redhat's version of Windows Vista. It was over-engineered, under-documented, and in the end grossly under performing version of a famed line of products. This makes more sense when you realize that it took the Jboss team a full two years to come up with this new release. What's more, despite Jboss 5.1 being around for a while now, until a week ago, Redhat officially supported only version 4.3. In other words, you couldn't pay Redhat to support version 5.1.

This is neither here or there, for the call was still mine, and I had basically put the entire company's efforts on hold by placing a strategic bet on this new platform. Vijesh ended putting his project on hold and becoming my primary partner in this effort. Punit had to put his stealth project on hold because the platform was changing. Paul, Will, Bill, Jacob & Becky had to put their bug and feature requests on hold because the platform was changing. The entire company's growth was put on a holding pattern because of this experiment. The question then is, whether it was worth it.    

Experimentation is the necessary cost of innovation. Any firm, let alone a technology company, shouldn't shy away from experimenting (as Vijesh put it, "We're a technology company, this is something we do!"). Very often we see companies get so comfortable in the status quo, the If-it's-not-broken-don't-fix-it mentality. This is a very dangerous for it leads to entrenchment and a vested interest in the status quo. It's not to say that things always need to be changed, but they definitely need to be questioned regularly. Strategy unchallenged becomes dogma, suffocating diversity ultimately killing the organization it serves.

The answer then is yes, I still believe it was worth it, and so does the entire team. Who knows, six months down the road, I might waltz into the office and say, WebSphere will be our vehicle to the future.

Quote of the Day:

                          "Life is "trying things to see if they work" "
                                                                      --Ray Bradbury

Comments

[Sherry Main]

First, I love Ray Bradbury.

Second, great post. I tell my team, friends the same thing all the time... that failure doesn't mean wasted time. It's not an easy to concept for some to grasp, especially the antsy.

Third, great car analogy.

[jthinaka]

Thanks! I agree, it's tough for some the grasp and that's why we have to keep at it, on message, all the time.

[dan]

yeah, you'll probably have to move to it at some point anyhow, so the knowledge gained will be useful then, and in addition you may be able to start coding to avoid some of the complications. Were there things that had to be changed for the same reason a lot of times where maybe it should have been compartmentalized in one place?
I'm a fan of code review and refactoring, so to me it still looks like an opportunity to improve the existing code base on the existing platform even if you back out of the 5.1 specific changes.