They Will Not Listen

See: TheyWillNotListenResolution

I am working on a project right now that is nearing code complete. The system is written in Java, is well factored, and does what the customer wants. We have a nice, automated build/test system using Ant and JUnit, and we even have an automated functional testing system built into our processing framework. I'd say this project has been fantastic compared to many others I've worked on.

However, there is one serious concern, and I'm not entirely sure it can be solved. The performance is atrocious. This is a trading system, and is comprised of a user-interface (web-based) and a processing engine. The engine does matching, validation, and various MQ Series message sends. Oh yeah, our database is DB/2 on a mainframe running OS/390.

Now, we've profiled this thing with JProbe and our own metering and found that our Java code is fine. In fact, our Java code can run at extremely high TPS.

Our problem is DB/2. It is horrible. But the problem really isn't DB/2, it's the customer. They simply will not listen to us when we tell them that the database is the problem. We've been telling them this for six months, and they refuse to understand. This customer is a COBOL shop. To them, DB/2 is "fast". And it is, if you're within IBM's "sandbox". But, using their JDBC/ODBC bridge, performance is pathetic.

We've had IBM out. We've had TopLink out (oh yeah, we use TopLink too). We've had other consultants come in who've worked on systems like this before. They all say the same thing: Don't use the database like that.

We've tried everything, and performance just gets worse. Early on I built an entire prototype using POET, which is an OODBMS. It was a screamer. They are resolutely against using an OODBMS -- they're too mysterious.

So, my question: How do you get a client to come to terms with reality? We've been saying this for six months and I'm afraid the project is going to collapse if something doesn't happen.

Okay, I didn't really make myself clear here. We've tried "everything". We've looked at type-4 versus type-2. We've looked at the ODBC interface and even stepped through the ODBC code with IBM. We've used IBM's internal DB/2 Connect tracing mechanisms to determine where all of the time is being spent. We've used JProbe on our Java code, not to mention our own homegrown metering. Our Java code is 'not' the problem. We've run simple JDBC only tests, just to see what the "high water mark" is. We've had TopLink gurus (three of them) out to our site from WebGain. They basically told us to use straight SQL. Still, this doesn't help. We've had other consultants who've built systems like this before come out and give us their wisdom. What was it? Do it in memory. All of it. Use as many machines as possible, and give each machine a specific "function". Like each machine handles a certain product grouping, or a certain firm. We've had IBM's own DBA's try to help us. In the end? Our tables (objects) are too complicated, we have too many columns of data, and we are doing too many joins. Especially those of the self-referential variety.

I've moved a lot of the technical discussion to TheyWillNotListenDiscussion. Jim's post below is what I'm looking for. Although, I would say that number 5 isn't absolute. The solutions that are available tend to uncomfortable to this customer, even if they still include using a relational database.

What I'm trying to understand, from those who've done this before, is how to deal with the people in this kind of situation. I'm very comfortable with the technical issues (and I'm not trying to be cocky here). -- JeffPanici

Interesting page, Jeff. Let me see if I understand the situation:

Your application is too slow.
Profiling has indicated that the problem is DB/2.
Talking with experts indicates that DB/2 can't handle the kinds of queries and joins you're using.
TopLink necessitates that your current queries and joins be used.
You're convinced that the only technical solution is to replace DB/2 with an OODB.
IT won't accept point number five.
You're frustrated. You'd like us to suggest a social or political solution, not a technical solution.

Did that sum it up correctly?

(PS: When it's all over with, please submit your experience to OptimizationStories!)

--JimLittle

Yes. Does anyone have any recommendations on this? This customer is your average, large, corporation. They feel safer doing what the "herd" does rather than strike out on their own. But it is also simpler than this. This particular group's exposure to technology really begins and ends with the mainframe. They use some other tools, but -- by and large -- they're a COBOL, VSAM, DB/2 shop. This is what they can see. This is what they know. However, we're building a large, distributed, system. There are others parts of this corporation that deal with this kind of technology, but the two "hands" don't talk to each other. Or, at best, they know they're there -- they just ignore each other.

Think of it like this: Many of us on this wiki probably wouldn't be suprised if someone told us they used 100 Linux servers to run their web applications. Or that they have special servers to process each kind of product grouping they offer to their customers. This customer hears these statements, but doesn't understand/believe them. They only see one thing: The mainframe. To them, distribution means buying a few Sun servers and "replacing" the mainframe. We've worked diligently to try and shatter this way of thinking, but it has been difficult.

So, the real question is: How do you migrate a group like this? -- JeffPanici

Jeff, I've read this page and the discussion page. I'd like to take a stab at the social/political solutions you're looking for.

It appears that there are two distinct parties within the 'customers' group. I would suggest starting to think more in terms of personal motivations of each person rather than sticking to the UsVsThem? tack. I had a similar experience running up against a brick wall when I tried to get people to listen to me about XP at my previous job. In my case, I realized after the fact that I was being too abrasive and was effectively shoving a big wedge between myself and management. Somebody said (and I think it's on Wiki somewhere), "People hate change." The first reaction is, "Yeah, well, everyone knows that!" To which the proper reply is, "No, you don't understand, people really really hate change."

I would say the first step is to break out of the logical-argument mode. When it comes to emotional topics people are not logical. The IT manager in this case is not being logical; however he's being very stubborn, which hints to me that there's a deeply rooted emotional reason for resisting the change*. The other reason to stop trying to be logical is that nobody likes to be 'proven wrong'. You could have a signed letter from the CEO of IBM stating that DB2 is not meant to handle this kind of data structure, and the lowly IT manager will not budge. In fact, he'd probably become even more adamant. The reason is that he hasn't been given a chance to save face. If he finally agrees that you were right and he was wrong, he'll look like a fool for holding up the project for so long. And heck, what's he got to lose by resisting you? If this project fails, he's still got his job and can still show his face amongst the people he works with. It's you whose contract/reputation/etc. is on the line. But you know what? The IT manager doesn't give a damn about your problems. You've got to find a way to either a) get the IT manager to care about getting the project done, or b) go over his head. Which one is more difficult depends on your circumstances. And maybe a little of both would work too.

I would strongly suggest buying and reading HowToWinFriendsAndInfluencePeople. It's the book that finally made me understand why I wasn't able to make any progress changing my previous employer's work process. It's a nice small book and shouldn't take more than a day or two to read the first time through. Start thinking more on a personal level about individual people's motivations, and you'll start to see why people are doing what they're doing.

In the mean time, from what you've described it sounds like the quickest way to resolve this is to remove the IT manager's decision making override. Put it in a dollars and cents way and let the person above the IT manager know that there are two options (or however many you can come up with). A) The project goes on forever until the client runs out of money because "It's just not possible to do this with DB/2", or B) The project can be finished at such and such a date for $X with an OODBMS. Do not consult the IT guy, do not waste $200. Even on this, you'll probably get some resistance because the upper manager doesn't want to be put into a position of appearing to be second-guessing his IT manager. You've got to give him a rock-solid excuse to do so (i.e. "You'll lose a lot of money if you don't").

* Possible reasons that the IT manager is resisting.

Thinks it will eventually lead to losing his job (probably the underlying root reason)
Afraid that it'll screw up and he'll get blamed.
Afraid he'll have to do a lot of work.
Doesn't want his ignorance of technology to become apparent (after all, he's supposed to be the IT guy).
Afraid he can't learn new tech, and doesn't want to face facts.
Doesn't want to look like a fool for holding up the project 'for no reason' (so he holds it up to make people think there must be a reason, otherwise he wouldn't hold it up).

The premise of this page is a non-starter. It is that Jeff knows better than the IT manager or whoever is in charge from the customer's part, and Jeff is smart but that guy is stubborn and narrow minded.

... they feel safer doing what the "herd" does rather than strike out on their own. is a rather fullish statement. Maybe the consultants were doing what the "herd" does with Java and O/R mapping and TopLink. (At least you can bet that the Java/OO herd is a lot larger than the COBOL/DB2/OS390 herd ). And trying to go over the head of an IT manager and go to the business guys who don't understand the situation is rather a bad move anyways, and in the common case is dishonest and unproductive.

If the COBOL programs work well with DB2 and if it was known from the beginning that DB2 was to be used it, then it follows very simply that the consultants screwed up the project, and it is they who will have to listen. So they can try to go to the IT manager and say something along the lines :

We really screwed it. The way we built this application cannot work with DB2. We're sorry for this, but it's too late for a radical change and since you are also very interested to see the project through, maybe we can alleviate your concerns with running an ODBMS. What can we do for you to make this solution acceptable ?

What's the use in laying blame? It doesn't solve the problem. Forget who's fault it is and find a way to fix it. It's Chet's fault anyway.

It's not blame, it is called responsibility.

So what? It still doesn't solve the problem. DontLookAtTheFinger.

Neither [None of?] the other fullish [foolish?] things on this page will [solve the problem?]. But not laying blames and taking responsibility is a good starting point.

What the heck does fullish mean? And whatever it means, the point you're trying to make remains to be seen. I've seen it work, so your blanket statement that it won't seems 'fullish' to me.

See: TheyWillNotListenDiscussion