They Will Not Listen Discussion
They Will Not Listen DiscussionThis is a more threaded series of discussions over TheyWillNotListen. I've refactored them here to try and keep TheyWillNotListen on the right track. ;-> -- JeffPanici
I'm not looking at it as a code problem. I think it's more of a communication problem. A lot of traffic on this page is the Teeming Masses "not listening" and Jeff reiterating that he's tried everything.
I'd like to gently suggest that the phrase "TheyWillNotListen" presupposes failure. No one likes to be told.
Perhaps it does; this could be my frame of mind right now. I'm not really looking for failure; and, I hope I'm not projecting that. The intent of this page was to try and find if others have been faced with a similar "non-technical" problem, and how they went about solving that with their customer. There are solutions here. Heck, some of them even include using DB/2. However, many of the real solutions are outside of this customers comfort zone and I'm just not an expert at hand holding. Sure, I can say things like, "Here are your options and if you pick the wrong one it's your own damn fault." However, consultants do have a stake in their projects succeeding, especially in these tough economic times. I want to guide this customer to the right solution. How can this be done? -- JeffPanici
guide this customer to the right solution
Do you already have a solution in mind? If so, give it up because the solution must come from your customer. Help to create an environment allowing your customer to determine the solution that makes the most sense in their environment and their comfort level. This isn't just NewAge hand-holding mumbo jumbo. It's an important aspect of change and, from your comments about what lead to using Java and OO, change is partly what this project is about.
Consider using a series of facilitated meetings to allow the I/T management, your customers, and any consultants to shift their focus from blaming each other for the problems to a more positive environment of how to jointly solve the problem.
-- MarkAddleman
Can you lose TopLink? It seems like its the OO -> RD mapping that is hurting you. If you can't replace DB2 then you have to change the way you use DB2. --NoelWelsh
I agree. It seems like the problem is how often you're hitting the database, and the nature of what you're hitting it with. Ask yourself what you would have done if you didn't have TopLink, and you still had to implement on top of DB/2.
TopLink is good for pretending your RDBMS is an OODBMS. If you need to be compatible with an OODB, that's good. If you don't, then you face the reality that RDBMSes are _not_ OODBMSes and have different performance issues. The good news here is that you've got your objects that need to be persisted well-encapsulated (thanks to TopLink). So rip out the TopLink code and start optimising your code for the environment. This will probably mean doing most of those joins complicated, nasty joins on your side of the bridge. -- RobertWatkins.
What version of DB/2 are they using? Why can't you denormalize some the worst joins and use triggers to keep the data in sync? We're using DB/2 OS/390 v6.1 and DB/2 Connect v6.1sp9 on Solaris and NT boxes. We've done a lot of this, but the peformance is just not optimal. See below.
We "know", as best we can, given all of the input and technology at our disposal, that the problem is the database. My question, which is more a socio-political question, is again: How do we make them see reality? -- JeffPanici
What are their performance expectations? Can they easily be quantified?
Yes. The user-interface should respond in under 3 seconds at all times. Remember, these guys are used to 3270 interfaces where response time is almost always sub-second. The system must be able to process around 100 trades per second.
Can you port the application to use GemStonej or POET and run benchmarks? If the performance is really that bad, I suspect that it won't matter that an OS/390 is running the app in one case and a Sun/WinTel box is running it in the other case. Ask your clients if they would rather pay for rewriting the application or maintaining another production box.
Early on we built the entire system backed by POET. Performance was not a problem. The user-interface hit its target of 3 seconds or less. And the processing framework was hitting around 200 trades per second. Plus, we never investigated "federated" POET databases or reorganizing the data for better query performance, etc. because the customer said, "No OODBMS' here." -- JeffPanici
Under their technology constraints, how close are you to the targets? Has the client seen the system running under Db/2 versus Poet?
Have you presented them with the price tag associated with making the application, under their constraints, fast enough?
Well, not very. We get wide fluctuations in query peformance. I'd say the average wait is now around 10 seconds per query-type page, and it gets worse as more users jump on board. The trade processing system is even worse: At best we can do around 20 trades per second. In reality that number jumps around based on other load factors. They, indeed, did see both the POET and DB/2 version. I'll tell you an interesting story. We had all of the top brass in our little "consultants room" and everyone watched the POET demo and said, "Wow. Yeah, that's really fast. That's great." And then we showed them the DB/2 version, which ran like molasses. The actual "customers", the ones defining the requirements, said, "We want the POET version." However, the IT managers, well -- they had different plans. I kid you not, the IT management stood up and said, "The POET version is our back pocket solution, but we're going with DB/2." And here we are six months later and DB/2 hasn't magically gotten any better, in fact worse, and any kind of OODBMS is a forbidden discussion point. As for a price tag...I'm not sure on this. I think we've made it clear to them that they aren't going to be able to run this on one SunFire? 280 and "replace" the mainframe. However, I don't think they've accepted that they're going to need a lot of hardware to make this work. -- JeffPanici
Ignoring the performance issue, how close is this project to being feature complete and ready for production? If you're close, my guess is the problem will be resolved after a few nasty meetings involving your customer and the I/T managers. I/T is generally low man on the totem poll in corporate politics. The customers, who I assume are traders in this case, are probably among the company's revenue generators and therefore will have their way at the end of the day.
One more point: It might be good to make the POET system available to your customers again for a point of comparisons. Then, step back and let them fight the battle for you. After all, it's their system.
To them, DB/2 is "fast".
To be blunt, it sounds like the developers are the ones who are not listening. Give the customer what he wants now, and if he comes back later and wants some speed improvements, do it then.
We have given the customer what they want. We've tried every trick in the book: IBM's book, TopLink's book (which is really bad, BTW), other mainframers. Their DBA's have tried various index configurations. We've really given this our best effort as a team. It still does not work as desired. --JeffPanici
I know this doesn't address your question, but I'm curious what's so slow about your system? Is the object model very complex? I understand that TopLink can generate suitably monstrous queries that bring any RelationalDatabase to its knees. Is it a multi-user thing? Is DB/2 fronted by CiCs?
The object model is tree-like in nature and thus causes severe performance problems. TopLink is a great "theorhetical" tool, but it seems to lack significantly in "real world" uses. It's both a mutli-user thing (for the user-interface) and a single user thing. No, DB/2 is not fronted by CICS, but we've talked about possibly using CICS as an alternative. --JeffPanici
Have you considered that the database IS NOT the problem ? At least not a database like DB2. You can believe them that DB2 as a datbase management system is fast, that is it has the potential to process more TPM than you'll probably ever need. However that doesn't mean that the way you implemented your database, and the way you're accessing the database is also fast, it may be excruciatingly slow, for reasons that have nothing to do with DB2, until proven contrary.
Sorry but you can't complain that DB2 is slow, and your customers are certainly not going to believe you. The least you could do is to trace the problem down to the query level, and say, look for this query, and for this table DB2 executes very slow, under these conditions (concurrency from other transactions, isolation level, etc). And if you do find such a thing please share it on wiki. --CostinCozianu
Hi, Costin. On this, we agree! Db/2 is very fast and generally a pleasure to work with as far as RelationalDatabases go ;) -- MarkAddleman
We've done all of this. We've JProbed our code. We've run IBM's own, in house, DB/2 analysis tools on our database queries to determine where the time is being spent. Do you want the breakdown? How about: Our code=10%, MQ Series=20%,DB/2 on OS/390=%70. We've metered the network, and it isn't the problem. We're not dropping packets or making extra hops. And, we're not "complaining" that DB/2 is slow. We're trying to point out that all of the evidence strongly suggests that DB/2 OS/390 is our problem. Mark: I'd agree with you, if we're talking about DB/2 UDB on Solaris or NT. However, one must be careful with IBM products. They all tend to have the same name, but are truly different beasts under the hood. Especially DB/2 for AS/400 and DB/2 for OS/390. -- JeffPanici
Is DB2 a business requirement, or is "N transactions a second" a business requirement? Unless they have a business need to do online queries against the running transaction database, one has to wonder why DB2 is a business requirement.
Both DB/2 and "n TPS are a business requirement. -- JP''
DB2 is fast. But there's probably no rational reason to believe that the JDBC/ODBC driver for DB2 promises anything other than that it will work right most of the time (not that it will be fast). Also, be aware that network round-trips can kill you.
Kaching! You win the prize! This is exactly what we're saying. And, believe it or not, the network isn't the problem. -- JP
Ask the consultants who are saying "Don't use the database like that." this question: "How should we use DB2?"
They say: Don't. Almost all other consultants who've built systems like this have used Oracle or some proprietary object-like database system. We've talked with two groups who do 'everything' in memory! -- JP
Consider maintaining some data locally, possibly on a version of DB2 for your Java platform, and queue changes to the mainframe DB2 instance. Consider using queued and asynchronous communication for all updates you're required to make to the mainframe DB2 instance. Cache locally everything you're required to read from the mainframe DB2 instance. You can run Java on the mainframe too; this may help distribute work between the different platforms.
"Nothing is impossible. It's only a matter of time and money." (And willingness to try.) -- JeffGrigg
Jeff: We've suggested many of these items. Perhaps my original post didn't make this clear: They don't want to hear these things. They want us to flick a switch on the mainframe and say, "It's fixed now." -- JP
Jeff [Panici], have you spotted the real problem?? It can be anything from the windows kernel spiningn around the SNA driver (assuming you connect trhough SNA and not TCP/IP), to the ODBC driver locking on some stupid semaphore in multithreading conditions, to a hard-drive that is full, a tablespace that isn't properly maintained, to a bad database schema, to bad queries, improper indexing and tons and tons of lots of other things, including if the mainframe is too old and running on some 50 mhz processor. Before you can treat the problem (and rearchitecting your application is quite a painful treatment) you absolutely have to know what the problem is. --CC
We have painfully metered almost every inch of our "pipeline". Again, perhaps my original post didn't make this clear, but we've done all of this. We've got "the golden egg" so to speak. We know exactly what is wrong in our picture. It's the database, pure and simple. Every suggestion we make to perhaps improve this always results in them saying, "Well, our COBOL DB/2 programs are fast." We do know where the problem lies, we just can't seem to make them see it. --JP
But it's clearly not the database. Who is the "they" in the title? The customer or the developers? -- SunirShah
The pronoun "they" refers to the customers. Both the end-users and the IT managers. --JP
I was being rhetorical. Although not harshly. It's important to understand what's holding the developers up from solving the situation technically. As the problem is solvable, there is a solution. --ss
They want magic; you'll have to learn magic. (Sometimes I have to conclude that some people may prefer to have their problem than to have a solution.) ;-> -- jtg
If JDBC is slow (well, too slow), you might have to write your own bridge. As joyous as that sounds, you can't fight your way through a bad platform library. -- SunirShah
Good point. We even looked into this. A small group in Canada had done just this; however, it required a revision to the mainframe that the customer was unwilling to make. Bang, we lose. -- JP
I don't think that's the right answer. If the customer can talk to their database in COBOL, the correct answer is to use COBOL. -- SunirShah
I agree, Sunir. Do you know how many times I recommended that they stay a mainframe shop and forget any notions they have about Java and OO? However, their response, every single time is this, "If we don't look into this new technology...or, at least, make it seem like we're looking at it, we'll lose employees." To them, this is the "hot" new thing and they're worried they won't be able to keep their current staff. On the other hand, this is related heavily to the way the market was over the past few years. All of the "hype" has died off dramatically and people can't just jump ship to the next best thing. So, it's possible they'll return to doing it in COBOL. -- JP
It doesn't have to be all Java, and Java does not imply object-oriented stupidity. You will have to lose the notion of OODBMS because that is stupidity. Never force a data structure to be something it isn't. That is a CodeSmell. --ss
Wait, you're really using the JDBC/ODBC bridge? That thing's monstrously slow. Isn't there a type 4 or at least type 2 JDBC driver for DB/2, and have you tried it? -- StevenNewton
The short answer is no. The long answer is, IBM's type 2 driver is actually faster than Merants type 4. IBM is "subsuming" Merants type 4 driver and making it their own, but it requires several upgrades to the mainframe that, again, the customer is unwilling to make. And even if they were willing, it might take months. -- JP
Try telling them that COBOL had totally different approach to the solution than the one you have developed. With flexibility, price has to be paid in terms of efficiency. Since as far as I can understand here, OOP based approach is the one that is casing performance overhead. Ask them if it's flexibility verses performance what they would choose.
That sounds too close to blind object worship. If the system doesn't work, it isn't flexible. Here's a true story: Nortel's switching sofware had grown from 2 MLOC to 25 MLOC in under five years due to the exponential maintenance cost curve for LOC. Their response was to rewrite it from scratch. Their answer, this being in the days of OO hype, was to rewrite it "object-oriented." They had a really "elegant" design, I'm told, but it took 10 seconds between picking up the receiver and hearing a dial tone. In other words, the software was useless. It didn't work. It was a waste of money. They scrapped it.
Moral of the story: If the software doesn't work, it doesn't matter how nice your bubbles and arrows look on a wall chart.
Corollary: If COBOL gets the job done, and Java doesn't, that means COBOL is better than Java in this case. Don't knock COBOL. -- SunirShah
Irrelevant. Presumably the COBOL system wasn't working which is why the new system was being developed in the first place. Perhaps the techno-political constraints should have been incorporated earlier in the project, but they weren't and they're in the situation that they are in. The question is: Where do we go from here?
Exactly, that is the question du jour. I wish I knew the answer.
Sunir: I agree, that if COBOL works, then they should be using COBOL. I think it would be easy enough to rewrite a clean, modular, COBOL solution. However, there are two factions at work here. Those that think the "new" technology is the only way to go, and those that don't. For now, they customer "wants" OOP, so that's what we've delivered. To the degree that we can. -- JeffPanici
Well, let's see now. There's this: Our tables (objects) are too complicated, we have too many columns of data, and we are doing too many joins. Especially those of the self-referential variety. and there's this: The object model is tree-like in nature and thus causes severe performance problems. And this: TopLink is a great "theoretical" tool, but it seems to lack significantly in "real world" uses. And this: We've had TopLink gurus (three of them) out to our site from WebGain. They basically told us to use straight SQL.
It seems from this that your solution is clear. Use the tools you have to use in the way that leverages their strengths.
Absolutely. To use a probably over-wrought analogy, you were required to use a 40-tonne truck. You designed a solution that required a fleet of small cars, and are now complaining that the truck doesn't behave like lots of small cars.
Your code is at fault. Or rather, your design is. From what I can tell above, your design will have big problems working on top of any real-world relational database. Since you know that's what you had to work on, I think the customer is right. Does it work acceptably on top of any other relational database? (The fact that it works well with an OODBMS isn't that important, I think)
There's one more thing that needs to be presented here. Was the DB2 a technical requirement made clear from the very beginning of the project? --CostinCozianu
In a situation like this I would reach for GeraldWeingberg?'s TheSecretsOfConsulting?. You say they're not listening but maybe they think you're not listening? What's the real problem here? It's not technical. It's a people problem: either they don't want you to succeed or there's something else going on underneath the surface.
Seeing as they keep finding some way to invalidate any solution you come up with you might want to consider going to them and asking for help. Simply say "I'm stuck" what do you guys think we should do to solve this problem? Hopefully that opens the door to a shared solution. If they have a stake in the solution then they won't be as quick to nobble it.
I'd argue that the technical situation is just a symptom of some deeper inter-personal problem. Re-evaluate your relationship with the people who won't listen and see if there is a way to improve that relationship as a means of fixing the technical problem. --AdewaleOshineye
I read this and the first thing I thought was that this probably requires some simple query tuning. I've done it many times. Now, if TopLink doesn't allow you to change the structure of the queries, it is a simple issue to just eliminate TopLink and do some query tuning. I saw that suggested earlier but I didn't see a clear answer as to why that couldn't be done. --MikeCorum
EditText of this page
(last edited March 20, 2003)
or FindPage with title or text search