War Stories
War StoriesGenerally speaking, detailed descriptions of past experiences - usually with dramatic emphasis.
As always, hit the page title to get BackLinks...
Also see
Factor Soup
I had a contract gig for a very large, complex company. The project I was brought in for was to help expand and enhance a system for monitoring and studying certain kinds of activity from many different factors (viewpoints). They started out using a kind of tree-based drill-down approach, but it turned out that a tree was not sufficient for typical reasons documented in LimitsOfHierarchies. So it just grew into a sprawling mess where each screen was dedicated to a different combination of factors. To add a new factor, one had to revisit several dozens of screens to add a new factor button, for example. Plus, the buttons often didn't make sense because each screen was really multiple factors, not just the label on the button. Nor could they possibly cover all factor combinations that way.
Internally we jokingly started calling the system various nicknames such as "Cartesian Factor-O-Matic" (see CartesianJoin), "Chicken Combo Soup", "Factor-alphabet Soup", "Cartesian's Fifth Symphony", "Blender Render", etc. Anybody with the vaguest sense of factoring and generacy skills realized it was a mess, but nobody seemed to care. It was tedious to work on because every section was just another hodge-podge of factor combinations. One was just reinventing the wheel with slightly different parts each time using copy-and-paste. Shared libraries were generally discouraged because one had to do a fair amount of paper-work to justify and log the libraries.
An analysis of the patterns generally suggested that the factors be turned into meta information such that the combinations of factors are dynamically selected rather than hard-wired into the interface and navigation.
I mentioned this to the lead architect, but he generally did not care. It seems they once tried a more "generic" approach in the past, but the programmer on that project could not handle the abstraction level. He/she was otherwise known to be a very productive programmer. Thus, they were afraid that the abstraction in a more general solution would make it hard to find developers. They wanted PlugCompatibleInterchangeableEngineers for the usual reasons.
They also seemed worried about the performance of a more dynamic solution, but the bottlenecks were caused mostly by other issues in my observation. Besides, the cost of adding an extra server or two couldn't possibly be more than army of developers it would save. They were penny-wise-pound-foolish on performance.
So, they ended up with a large piece of software with an army of PlugCompatibleInterchangeableEngineers to keep it ticking. They seemed happy to do it the hard way rather than the smart way. Nobody wanted to bother to step back and study the forest, insisting instead of rushing to get more trees out on schedule. Plus, GoodAbstractionRequiresExperimentation. --top
Holy cow, Top, this sounds surprisingly like a project I endured in the casino gaming support software world. Make it work but don't actually do anything that would simplify and streamline the code, as that would confuse the more conventional coders. Just CopyAndPaste, CloneAndMutate?. Ahhh, good times. -- GarryHamilton
The people who didn't seem to care seemed happier about it all. IgnoranceIsBliss. More WarStories in AutoKeysVersusDomainKeys, PageAnchor: License_place.
Modem-Speak
We had a bank of modems and to check which modem went to which phone number (people sometimes switched them without telling us) we would have to call the number on a voice phone across the way and then run over to the modem bank to see which lights were on. Often the modem lights wouldn't stay on long enough from a mere phone call.
Rather than run fast and risky in a crowded, wiry data center, I discovered that if I whistled certain frequencies mirroring the connect sound, the modem would think I was another modem and spend a longer time trying to connect. Thus, by learning to speak modemese, I could walk instead of run.
A computer room steward saw me doing this and told his shift buddies about "the crazy lonely guy who flirts with modems". Referring to their squawky sound, somebody joked about modems being consolation partners after I got allegedly dumped by a Dalek.
(I'm tempted to create a new topic: LegacyStories? for such.)
--top
Now, there is InterestingAnalogTechnologies.
Correct:
loop(item = getNextItem(...)) {
list = listAppend(list, item, delimiter);
}
Mistake:
loop(item = getNextItem(...)) {
list = list . listAppend(list, item, delimiter);
}
// period is string concatenation
This caused geometric growth of the string, blowing the memory. I was thinking manual list appending but used the auto-appending list library operation instead. This took a while to isolate because it brought down the computer until the right "exit" debug point was found.
Another one similar to the above. A given web-based report was running about 10 times slower than I expected and I started digging into it to find out why. I inserted trace statements with time stamps at various points to narrow down the problem spots. It turned out not to be the database queries, which was a bit surprising, for the database is almost always the bottleneck. Eventually I narrowed it to a specific module that was fairly complex and I couldn't find ways to simplify it much. This complexity also made it hard to isolate the slowness further because it did a lot of different things in order to weave these diverse things together for the result. Consolidation was the main feature of the report. The deadline was approaching and I decided to just tell the customer it's far slower than expected and I couldn't improve it.
The area was still reeling from the dot-com crash and doing a good job was important for mere survival, so I was frustrated and tired from working late and really didn't want to have to tell the customer it was half-ass working at delivery time, but had no other choice. So, I began cleaning it up for production delivery by taking out all the trace and test garbage. I usually mark such code with a keyword comment like "debug" or "trace". So I was in the editor pressing the Find Next key to remove these statements as a usual practice. While doing this I happened upon one "debug" statement that I didn't remember putting in. It looked something like this:
traceString.appendString(",Foo: " + foo); // debug
It was in a function that was called several thousands of times. The app was slow because it was appending to a ballooning string used for testing. Hot Diggaty I found the son of [bleep]! People popped their heads out of cubicles to see what all the happy shouting was about. -t
Watch Your E-Mouth
I once put a snarky error message into some source code as a contractor, similar to what one might find in FunnyThingsSeenInSourceCodeAndDocumentation. My partner spotted it and warned me to immediately remove it in a serious tone. He once contracted for a place where somebody put a cuss-word in an error message that the author thought was very unlikely. Murphy's law kicked in; the client discovered it; and made the contractor's team go through every single line of code and certify that no other odd stuff remained in the code. And the offending programmer was fired.
RAM Battles
I remember the days in the mid 1990's when the difference between 4 and 8 meg of RAM was night and day. It was like a Yugo versus a Chevy sedan and people fought like mad to get more RAM.
My coworker was setting up a new PC for some PHB (PointyHairedBoss) and noticed it had 12 megs of RAM, which was a hell of a lot for those days. He secretly pulled 4 out and gave it to me so that I could have 8 instead of 4 because he saw how slow certain apps of mine were under 4 (MS-Access, FoxPro, etc.) "The PHB wouldn't know the difference between 8 and 12", he said, which is probably true. He was truly a PHB.
I felt a bit guilty, but when I saw how apps could start up without a bathroom break in between, I stopped feeling guilty. 12 was kiss-up-based allocation anyhow, not based on actual needs. The PHB mostly just used Word and email all day.
Hard Copy
A middle manager was asked to update the phone intranet roster by a higher manager. He never consulted with IT before he started. When all the changes were complete, for some unknown reason he delivered a printed copy of the HTML code of the new phone roster to the web-master, and said, "please put this up on the intranet; the deadline is coming soon". The web-master asked, "Do you have the file in electronic form?", "No, I misplaced it and doubt I can find it before the deadline," replied the manager. Just after the manager left, the web-master grumbled about having to hand key it from the printout. A coworker turned to the web-master and stated, "well, I guess that's why they call it 'hard copy'."
And, if one complains, they might get a printout like this:
101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110101001010101010100101 110010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110101001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011010101011010000101010101010010100 110011011011010110111110101001010101011010010111101010110101001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011010101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101 010011011011010110111110101001010101011010010111101010110100001010101010100101 101101011011111010100101010101101001011110101011010000101010101010010100101001 100101011011010110111110101001010101011010010111101010110100001010101010100101 010010101101011011111010100101010101101001011110101011010000101010101010010100 010011011011010110111110101001010101011010010111101010110100001010101010100101I hope the webmaster in question OCRed the printed HTML and was able to meet the deadline
It had some pen-written corrections if I remember correctly.
My coworker, I'll call Bob, used to be a computer operator in a mainframe shop. A screwy accounting programmer promoted because of connections instead of merit, and with a history of fouling up learned that he could blame Bob for "typing in the command wrong". It had no parameters, it was something like "RUN ACCT23". The stupid managers fell for it, as if the buttons were analog such that pressing Enter at the wrong angle makes the command run wrong. Bob was formally written up for the alleged transgression.
Eventually they removed him from console duty, and when other operators allegedly made the same mistake, my coworker was still blamed for "not stopping the operator from doing it wrong". The whacky programmer milked any excuse he could.
My boss's boss (BB) was not known for being available for questions; always magically out of town or on an "urgent call". BB once sent a request to put X under section A and Y under section B. We were confused about what exactly A and B were. In the past we got chewed out for asking questions that were allegedly answered in long-gone meetings such that we decided to wing it rather than ask BB. Apparently we guessed wrong and BB went on a tirade and had my boss moved and demoted for this and similar past communication breakdowns. Several months after the demotion we found out that somebody else had renamed A and B and that is largely why we were confused (my boss didn't have access to change A and B). We didn't know it was renamed. Both BB and my boss were poor communicators, but BB was the real problem in my opinion. Although my boss was a poor communicator, he was a patient listener if somebody simply spent the time to explain exactly what they want. BB on the other hand had no patience for such: everything was hit-and-run communication with BB. But BB had more power and was a better bullshitter, and so my boss took the hit for BB's poor management style.
The aspect ratio of the CEO's portrait was changed, perhaps to make him/her look thinner on the main intranet page. Nobody wants to talk about how it got that way; they just mumble in a low-but-serious voice, "don't ask, but leave it as is".
My new PointyHairedBoss and I were tasked with coming up with ways to index bunches of corporate documents for our new WAN, connected to about 30 satellite offices all over the US. (Essentially an intranet Google-like service.) This was pre-intranet and WAN's were also relatively new technology. I was looking into a self-rolled system based on an off-the-shelf indexing tool glued together with some custom programming so that a central server does most of the work and sends only the results to clients (AKA, Client/Server). But my new boss was heavily focused on this one vendor selling a LAN-based solution. It was designed before the WAN era. We asked about the largest WAN customer they set up, and after dodging the subject for a while, they admitted a 3-node WAN was the largest they've worked with, and the 3 sites were relatively near each other. I told my boss, "30 sites coast-to-coast compared to 3 semi-local is a big leap, and this is outside of their product's road-tested range and will likely fail. Rarely do vendor products work well outside of their design range because scaling is not trivial to add. And our WAN is slow."
He ignored my warnings without explaining his technical reasoning. His reasoning was something like, "They are the best product in the business for our price range, so they must know what they are doing". I moved to a different department shortly after, and 2 years later heard from an insider that the company purchased the product, but that the product failed miserably in production, and because of this, that boss was transferred out of that position. I don't know if it was technically a demotion, but they realized he was technically clueless and put him into an administrative position instead.
Another thing, I suggested we continue on the self-rolled version as a plan-B. But he asked that I stop work on it. If there is a fair amount of uncertainty, then it seems like researching a plan B is a rational position, but he was so certain the vendor would deliver. I believe he was suckered in by sales hype and ignored the technical scaling discrepancy. "Trust us, we are a well-known company in the industry and value our reputation for delivering and wouldn't have gotten this far if our product was junk, blah blah blah..." The thing is, failures are hard to research and this company may have failed many times before using the same song and dance, and my clueless boss wouldn't know about such failures." SilentFailureNoisySuccess
http://www.tonymarston.net/aboutme/disasters.html
Wow. That page contains some wonderful examples of failed projects from the perspective of someone who is part of the problem but doesn't realize it. I'd be very wary of hiring this guy. --MarnenLaibowKoser, 3 Dec 2014
See also: JustMakeItRight, CategoryStories, HumansAreLousyAtSelfEvaluation, CreepingFeaturitisWarStories, PowerDividedByComplexity, ScapeGoat, HollerithPunchCard, AynRandDesignPhilosophy, KludgeItTillItWorks, RealWorldHierarchies
EditText of this page
(last edited December 4, 2014)
or FindPage with title or text search