Creativity Run Amok
Creativity Run AmokSome professional cultures reward creativity. Academic published papers, patents, and newly applied trade secrets are viewed as being among the greatest accomplishments in many R&D settings.
If creativity is rewarded more than factors that directly contribute to end-user satisfaction and corporate success, participants will choose innovative solutions over proven solutions. This phenomenon is accelerated by the academic value system that prepares most engineering employees, a value system focused on innovation, novelty, and originality.
Again and again, I've seen untold damage done in the name of "creativity" (or its modern synonym innovation, which people do not carefully distinguish from creativity), applying new solutions that don't work when time-honored solutions would have sufficed. This creates a potential challenge for the pattern community that values experience and proven solutions. But rather than backfiring in a way that excludes patterns, it usually plays out in terms of new ideas being cast as patterns. In fact, this phenomenon is probably a major contributor to oversights of BuschmannsLaw.
Has anyone else seen this problem a lot?
-- JimCoplien
reminds me of EdwardDeBono
yup, I run a section like that in the Reuse section of my talks. It is not generally in the interest of the programmer to use previous solutions - after all, how do you get to be Senior Programmer if you don't program? Can you imagine being introduced to Jill or Joe, "our senior programmer, who is so good at programming that she/he only ever reuses existing solutions and never writes any code her/himself!" How about people who get trained in programming all through school and then are told, upon arriving at their first job, that it is really preferred that they use previous solutions and not program themselves if possible? Then they look at the Senior Programmer and decide where the truth lies in this story. So I view the problem as endemic in our entire system (industry) from the start up.
Nice article in CACM about 2 yrs ago found no correlation between anything and increased reuse except increasing the perception of its value. Not objects, CASE tools, libraries, not even reward systems! Only things that correlated were education, encouragement, internal seminars, and the perception that using previous solution was cost-effective. (...and a couple of specialized subindustries, particularly telecom!)
For the patterns community, that tells me, 1) expect resistance, 2) keep up the encouragement. --AlistairCockburn
One thing I wonder about... Is this particular to software or is it engineering in general? I don't see why innovation should be more valued in software except for the fact that reuse does not happen much. In this sense it is a vicious circle. JimCoplien's comments above seem to imply to me, that it is a problem in engineering in general. From what I've seen, there is less "let me express myself" in other engineering disciplines, mostly because there are often components hanging around that impose more structure on solutions, and building from scratch is rarely an option. -- MichaelFeathers
One significant factor, I think, is that other disciplines are more constrained by the laws of physics than is greenfield software. I see most of the flagrant innovation taking place in greenfield efforts, rather than in the contexts where it probably has the most value: longer-term maintenance. I don't know if this is the most important factor at work here, but it's certainly one of them. -- JimCoplien
That said, programmers may believe Alistair's characterization, though my own view is that it's a rationalization.
I believe that programmers program because it is more fun than reuse. Reuse is hard, especially in today's environments. You have to find the thing, you have to understand the thing, you have to hold your mouth funny while you use the thing, and then you still have to trust the thing. "Bag it, I can write it faster than that", you say, and you may be right. Even if you're not, it's more fun.
It turns out that Extreme Programmers do lots of reuse, as a result of some of the XP practices:
The reasons for promotion vary from culture to culture. Perhaps more important than promotion are fellowship positions and less official offices of esteem and respect. That's where the power is (LeadershipFromTheNonManagementSideOfTheLadder?), and it comes from programming prowess. In most of the cultures I've studied, I've found (anecdotally) that promotion these days comes from effectiveness in making one's self visible and in paper shuffling and organizing for its own sake. In fact, a pattern of sorts emerges that suggests people are as likely to be promoted for being visible in a failed project than they are to be promoted for running a successful project. Visibility is key. But this should be another thread... -- JimCoplien
Agreed. To the extent that promotion is the issue, therefore, reuse can be an advantage, cause it makes you look good which one would hope couldn't be worse than looking bad. The issue of peer esteem may well militate against reuse unless the culture supports it. Nice distinction. -- R
I can't say I've ever known anyone to get promoted on the basis of their reuse record. Your comment suggests you're impressed with the likelihood of such an event: perhaps based on your personal experience? Maybe it says something about the relative merits of our respective work culture preferences.
I've seen more promotions on the basis of innovation, perhaps because that's what research labs have traditionally valued. -- JimCoplien
I'm suggesting only that promotion is based (we hope to a large degree) on "getting things done". In software development, that usually means getting the program done, and quietly reusing what exists could help with that. Certainly in a research environment, innovation has a much higher perceived value, in spite of all that talk about standing on the shoulders of giants.
Promotion based on reuse in research, perhaps, would go to the guy who actually did something that everyone was talking about in the coffee room. Or maybe not? --RonJeffries
Cope asks whether others have seen damage done through using new and untried solutions where the tried-and-true would have done just fine.
I'd have to say yes, and no. In my many years in charge of R&D at Comshare, which was really intended to be very little r and lots of D, I would often try to build something for the future. Since we were building operating systems, language compilers, database management systems in those days, that often paid off. We'd work from the current research papers, rather than forge off on our own, and rather than use the then-standard technology. We tried to steal from the best: Peter Deutsch, Butler Lampson, Ted Codd [DrCodd], and so on.
Still, what we were doing was hard, and sometimes we delivered good stuff too late to meet management's expectations and requirements. I have been told more than once that they'd rather have it delivered than have it be perfect. And I have more than once said OK, have it imperfect, and been hammered in the marketplace and then by management over poor quality.
Where I feel things went best was when we were following the thinking of whatever guiding light we had chosen, but were building simply, reliably, and for now, not for the ages.
What I like about XP (forgive me, please, for mentioning it) is the focus on quality and simplicity, in a context of delivering sooner rather than later, and in a context of communicating clearly to the Powers That Be exactly where you are, so that they can know, can learn to trust, and can take effective action in the market. It resonates well with my successes and my failures in this long life of coding. --RonJeffries
I have seen it often enough, been guilty of it myself, and expect to be guilty of it again in the future. Saw a lead programmer determined to use This Project to extend his GUI builder from his previous, even though GUI builders are next to the last thing a mid-90's Smalltalk programmer needs.
I saw a nice ripost from management once... prior to a project on which it was evident the programmers really intended to build their own purchase tracking system (and which gave me a nice little RequirementsDocument :-)), the IT manager's technical assistant kept asking them for a report on the buy-vs-build tradeoff. The programmers were getting ready to heavily load the build side (what IT department in the late 90's really needs to write their own purchasing tracking system for their company?), when the IT manager managed to prioritize a couple of actually relevant projects and involve these people. As an expediency, they were obliged to 'simply' go out and buy the tracking system. Happy ending, this time. The estimated time budget to write that thing was about 2 work-years (estimated cost $200,000 plus lost opportunity on other projects). --AlistairCockburn
We seem to be getting further and further away from CreativityRunAmok and more into MakingReuseWork, but... In our business (both the business of the corporation at large and the business of the DoIt team) we're finding that you cannot successfully sell products or services internally based on their long-term or philosophical value. As J M Keynes said, in the long run we are all dead. Doesn't matter how much I can convince a project manager that he will save in the long run on support or maintenance. Or whether I can get him to believe in the philosophical purity of an architecture or solution I am trying to get him to adopt. Because if the project is not successful, that individual decision maker won't be around to enjoy the afterglow of long term benefits.
I have to bring solutions to the table that help projects get to market faster and cheaper the first time they use them. That's how to get both the project manager and the lead technical folks to accept a solution from the outside.
If a component doesn't promise that kind of immediate benefit, then it is not a very viable product for me to invest in building. -- BillBarnett
See also NoveltyVampires
EditText of this page
(last edited December 27, 2011)
or FindPage with title or text search