Specialization Is For Insects Discussion
Specialization Is For Insects DiscussionSee the revised page at SpecializationIsForInsects.
A programmer should be able to fix a bug, market an application, maintain a legacy, lead a team, design an architecture, hack a kernel, schedule a project, craft a class, route a network, give a reference, take orders, give orders, use configuration management, prototype, apply patterns, innovate, write documentation, support users, create a cool web-site, email efficiently, resign smoothly. Specialization is for recruiters. See also: JustaProgrammer
-- with large apology to RAH and his estate, PeterMerel
In Ancient Greece only the wealthy elite were permitted to become 'Generalists' and keeping the dumb populace as 'Specialists' ensured that they did not evolve enough personally to work out how society was organised or how the world works. Socrates remarked on how only the Philosophers could escape the cycle of birth and death once they leave this life. Only the wealthy, elite, generalists were able to become the philosophers. The poor, dumbed down masses were doomed to specialism and accordingly a limited level of development which would keep them earthbound for many, many reincarnations.
Seems not very much has altered in the interim.
DianeH
This is not only nonsense, it is dangerous nonsense. Specialization is the secret of our civilization's productivity. It is better to have five developers, each of whom know how to do their job very well, then to have five who all know how to do each part of the job a little. A good developer does NOT need to know how to market an application or lead a team, though obviously somebody does, and you will be a more valuable developer if you also know how to lead a team. Some of the best (and most valuable) developers I know do not know how to write good documentation, support users, or create a cool web-site. They either work with people who do, or avoid situations where that is needed.
I could learn to do anything on the list above if I were given enough time. However, I only have enough time to learn to do some of them. I'd rather be very good in the things I choose to do than to do a lot of things poorly.
One of the reasons that this attitude can be dangerous is that it leads people to undervalue knowledge. I know of a project where five experience IBM mainframe developers decided to build a client-server system in Smalltalk. Not only did they not know Smalltalk when they started, they had never done any PC programming or any other client-server programming. They were smart, so they refused to take classes, preferring to just read books. They worked for a year on a project that should have taken three man-months by an expert. In fact, someone who had taken my Summer Smalltalk School had to come in and rescue them, which is how I heard the story.
I found out that I had to teach an introduction to programming course using Mathematica. I had never programmed in Mathematica, though I've programmed in dozens of languages, and rarely have trouble getting programs to work when I use a new language. But there is a big difference in getting a program to work and knowing how to use a language properly! I found someone who had used Mathematica since its inception and who had taught lots of other people to program with it. I visited him nearly every day and showed him all my slides and sample programs and he corrected them and gave me a lot of insight on the proper way to use Mathematica. Things went well, but they wouldn't have if I had just relied on the fact that I am an expert teacher and a programming language expert. Instead, I knew the limits of my specialization and found a real expert.
A real expert can do things that seem like magic to the novice. Experts can be orders of magnitude more productive than novices, and they can avoid problems that would be devastating later. That is why real experts get paid so well. I tell my students to find something to become an expert in. Sure, you have to know about all the aspects of developing software. You can't be a good programmer if you know nothing about testing, debugging, or design. You have to learn something about the problem domain if you are going to build a system that meets the needs of the users. But you shouldn't fool yourself into thinking that you are as good as an expert, or that you don't need an expert.
I agree that narrow expertise is dangerous. If you are a great programmer but know nothing about user interface design then you will create wretched user interfaces quickly. We need to know what we are good at and appreciate how our skill fits into the overall scheme of things. However, that is not what Heinlein's quote or Merel's restatement of it are saying.
-- RalphJohnson
I think you're pretty much restating Cope's position below, with which I think I pretty much managed to concord, but since I'm the one out on the limb with my buddy RAH I'll try to defend the position once more. Then I'll pray an expert WikiMaster comes to save we poor novices from ourselves.
I said it first. Somebody deleted my statement, so I typed it in again. Also, Cope was too nice. I am rarely nasty, but that was the way Heinlein's quote made me feel.
"Should Be Able To" indicates, to me at least, basic preparedness, not a brain the size of a planet. All the things I mention will be required of a professional software developer over an average sized career, and if he wants to acquit himself as A Programmer then, imho, he Should Be Able To. In other words he should know enough to make a stab if he has to, and enough to know when he's in over his head. I certainly take Ralph's (and Cope's) points: along with the rest, A Programmer Should Be Able To acknowledge his limitations, recognize experts, and know when to seek expertise.
He did not say "should be able to learn". He said "should be able to do". This does not require a brain the size of a planet, but it requires a life as long as Methuselah. Most professional software developers will never market an application, hack a kernel, or route a network. Perhaps if you had a smaller list then I would not have reacted so strongly.
As to the hypothetical team of five, it seems to me that whether the experts excel or not depends partly on whether the problem domain falls within their expertise, and partly on whether they have adequate competency in the Should Be Able To stuff not to futz each other straight to hell. Like Sam, I've seen the latter too many times not to rue it. --PeterMerel
Of course. But this does not support your original statement.
Well, I haven't visited this page in a while, but here's my present feelings on this. I think specialization increases efficiency and risk both. If you're dealing with a well-understood domain, like building suburban homes, say, where surprises are few and risk is not much of a factor, then you won't be able to compete if you don't organize your team by their specialties. Carpenters form wood, electricians rig wires, plumbers run pipes and so on. But if you're dealing with an a little known domain, then specialization can easily do you in. Golden Age Pirates, for example, were constantly running aground or getting lost at sea because piloting boats was a rare specialty. They were okay so long as they stuck to the West Indies, but the moment they struck out for the open water they invited disaster.
So, yeah, I think RAH's statement and my restatement here are too strong for general use. It's when you're going where no one's gone before, or when you're trying to make the best use of only limited resources, that specialization becomes a liability. Then again I'm saying and I think RAH was saying that if you aren't willing to do such things when called on - and, yes, to go grab an expert when need be - then you're inevitably increasing risks to yourself and your enterprise. In this way efficiency, I venture, always increases vulnerability. -- PeterMerel''
Germane to specialization: BradCox's No Silver Bullet Revisited compares the division of labor entailed in building Word with that of manufacturing a pencil. It's entertaining reading: http://web.archive.org/web/20040204231718/http://www.virtualschool.edu/cox/AmProTTEF.html -- BrianFoote
In our empirical study of large projects with sustained success, we've found specialization to be a common thread. You'll find it in a dozen patterns or so at the OrgPatterns site.
That's the established, empirical pattern. From reading the above with its preponderant use of the word "should," I wonder if breadth is a wish rather than an empirically grounded harbinger of excellence. I'd be interested in hearing case studies that bear this out; I suspect that if both modes are viable, they are viable in different contexts. Having enough case studies may help discern the context. But, for now, I'm ignorant of contexts other than where specialization thrives.
There are a few people I admire who have generalized between left-brained stuff into right-brained stuff, but in the grand context of things, they still cover limited ground. One example is LarryConstantine, who's mastered family therapy, and therefore organizational psychology. But that doesn't mean he's even a good child psychologist. And it took him a lifetime of work, dedication, and concentration to broaden to where he is today. He's a model for me in that regard: I hope someday to be a master in two disciplines instead of just one.
I'll agree that we should all aspire to be as broad as we can and that we should all get our fingers in lots of things. That doesn't mean we'll excel. There is no try; there is only do, or fail (hey, Yoda was at least as cool as Heinlein...)
-- JimCoplien
Hmm. Yoda finds it's easy to locate specialists in whatever skill his team seems to be lacking. These specialists match harmoniously because their skillsets fit as neat as pieces in a jigsaw. Yoda can always swap out a specialist who isn't working out and swap in a new one. Dagoba software development, after all, is a well-oiled machine whose parts require little flexibility to achieve good results.
Here on Earth, a team of multi-skilled folk can make more informed choices quicker and follow through more thoroughly than a team of single-skilled experts. Here, problem domains twist and turn in unexpected directions, and specialists do not snap in and out like puzzle pieces. Earth software development is still the domain of artists and craftsmen whose flexibility, breadth and perspective, as much as specific skill, are the factors that make or break developments. Cf. ProgrammingInVietnam.
Actually, this is in the StarWars mythos too. Can you name a member of the cast that wasn't able to make snap repairs to the Millennium Falcon when needed? -- PeterMerel.
Well, Yoda had the supernatural materially on his side, which is even rarer among software people than is excellence. (As long as we're on Heinlein, we might as well quickly segue to ArthurCeeClarke and note that any sufficiently advanced technology is indistinguishable from magic...).
O.K., enough of Yoda and pie-in-the-sky thinking. Bring it down to earth. What success cases can we name of folks who are amazingly facile in more than three disciplines? Who has made a name for themself doing Smalltalk, Visual Basic, Java, and C++?
[Um, ObjectSpace? GemStone? Cygnus? MicroSoft?]
Leonardo da Vinci.
Specialization is for those who get things done. Universality is for snake-oil salesmen.
-- JimCoplien
...and Renaissance Men. You'll never go farther than you allow yourself.
James Parton, one of [ThomasJefferson]'s nineteenth-century biographers, gave his dazzling range of abilities a dramatic accent when he characterized his subject as a man who "could calculate an eclipse, survey an estate, tie an artery, plan an edifice, try a cause, break a horse, dance a minuet, and play the violin." And Parton was describing a young Jefferson who had not yet written the Declaration. http://www.theatlantic.com/issues/96oct/obrien/charactr.htm
As to the context: I guess that when innovation is more important than time efficiency, you'll want more universality, and the other way round. Example: Software guys who were universal enough to read (building) architecture books... -- FalkBruegmann
This drives at what I think the fundamental confusion is here. There's a difference between interest and accomplishment. I totally agree that we should have broad interests. But it's pretentious and unreasonable to be expert at all things. Few people are expert both at configuration management and programming, and there's a good reason for that - we don't want to burden programmers with configuration management details, and configuration managers with programming details.
No one can know all the details of everything. Someone who takes the Heinlein line to its normal conclusion and presumes excellence in everything will end being excellent in nothing. At best, they'll be mediocre in everything.
I think the false belief in broad excellence may be the reason for the failure of small projects and enterprises as they grow. In the business world, this is called going beyond your core competencies.
-- JimCoplien
I think I see another confusion here. I don't believe Heinlein is saying you should be expert in all things. Building expertise in anything takes time and purpose, and as you rightly point out there are few of us who can boast such in even a single field. Of course I may be wrong about Heinlein, who was known to take the odd unreasonable stand, but for me I'm referring to competency and to meta-knowledge, not to expertise.
Perhaps my experience is unusual, but I've always found that engineers with the willingness to get stuck in to whatever it is that needs doing are of great benefit in software development. I expect that if someone lacks expertise in a task they'll be the first to own up to that. But if there's nothing for it, if there's no one at hand better qualified to try it, then I expect them to learn as they go, and be eager to do so. Is this unreasonable? I don't know, but I've not found it so. YMMV.
As to selling multi-skills - I think I make a buck at it. I don't claim to be anything like a Heinleinesque character, but having worked in a lot of different application areas - military, financial, telecomms, SCADA, manufacturing, GUI, AI, etc - I find employers see breadth as a benefit. And I enjoy having more than one string in my bow because there's security in it. The same should go for development groups, no? -- PeterMerel.
I've been burned many times.
On being burned: Mostly being of the nomadic persuasion, I tend to regard the messes generated by the well-meaning-but-misguided as part of my problem domain. In fact they're often generators of consulting work for me. But no-one can be lucky enough to avoid working on occasion with folk with whom they'd not work by choice.
On experts who don't know their limitations: Seems RAH is on the same wavelength as you here. From the same book quoted at the top of this page, "Expertise in one field does not carry over into other fields. But experts often think so. The narrower their field of knowledge the more likely they are to think so."
On MBTI abuse and pop expertise: I've seen MBTI used as a weapon, and disliked it as well as you. My usual response is to point to alternatives like PCP, for which check out http://tiger.cpsc.ucalgary.ca/WebGrid/WebGrid.html , but I've lost at least one opportunity by refusing to fill out a Keirsey test. I don't regret it.
To the point, a little knowledge is dangerous, but competency is not, and the difference between interest and competency is at least as important as the one between competency and deep expertise. A child may be interested in motor racing, but we prefer he doesn't drive until he's competent. A commonly competent driver is not suited to Formula One races, but 99.9% of the time driving doesn't require Formula One expertise.
So commonly competent drivers who don't attempt Formula One races are very useful. This seems to go for other skills too, so broad competency by multiple team members in multiple skills seems very often very useful.
I think "pop" trouble might be more a matter of qualification; when a manager chooses the fellow with the flashy website over the fellow with the StrongResume, or the boffin with the buzzwords and the CASE tools over the RealGuru? with the track-record and the know-how, then disaster looms. Does this really hurt you and I? Only if we can't pick the manager with the wisdom from the manager with the gullibility.
"Pop" is becoming an issue for sorting out Patterns too - see the PatternOfBabel.
On beer: It's more the concept of beer than the substance that I like, but I'd surely value the anecdotes. -- PeterMerel
Depth versus breadth. Which should one emphasize? One can make the trade off depending on how depth and breadth are valued. If the rate of change and your learning rate are such that the value of deep expertise in an area peaks and fades faster than it takes to anticipate the trend and develop that expertise, then you have the option of being on the ground floor before the boom by luck, or developing the "meta-specialization" skill of being able to see where various specializations are needed so that you can ride the trends. In the latter approach the core competencies are accurate perception, analysis, and problem solving skills, with carefully selected secondary competencies along the way. ScenarioPlanning helps. If you go this way you are hedging. You forego deep excellence in a narrow specialty for long term growth. This is kind of like an investment strategy.
Case in point. A person might adopt the strategy of learning all that they can about Sun Java and Microsoft COM simultaneously. Doing both, they can use knowledge of the commonalities and differences that they note to create a new synthesis of knowledge about design and frameworks in general. Becoming very competent in several courses of solution to the same types of problems can have this effect. We learn by comparing and contrasting. The shifts in perception also enhance creativity. The hedge here is that one of the technologies (Java or COM) may dominate. You will be ready for either case. Also, the synthesized knowledge that you gain will be in areas that have more long term value: design and skill acquisition.
In domains of rapid change, adaptability is more valuable than narrow expertise. In you are going to be excellent at something, be excellent at learning and applying new skills. Grow or die. -- MichaelFeathers
So what is the balance? Yes, you need to grow in breadth, but you don't need to understand everything to the level of mastery or even accomplishment. Few Smalltalk programmers who trash C++ could write credible C++, and few C++ programmers who trash Smalltalk could write credible Smalltalk. Yet there is operational truth in the arguments of both (well, at least of the introspective ones - there are certainly a lot of emotional outliers on both sides).
I try to keep this balance, but I've been burned by a lot of SorcerersApprentices?. I think that's what I'm warning against here. Yes, it would be great if we could all be sorcerers in all walks of software, but that's unlikely to be. More profitable is to stand on the shoulders of others than to stand on our tiptoes. More effective is to work in a team with overall well-rounded skills than to do everything one's self.
And teams where all members are experts in everything are difficult to sustain - not impossible, but there are strong and deep cultural forces at work against it. Effective cultures evolve role specialization. This is another strong correlation from the OrgPatterns studies: one strong thread of the dysfunctional organizations we visited was the lack of strong role identification. The strong organizations can identify roles off the top of their heads. It's how effective human cultures work, too - and it may be a bit egotistical to think that we're as much better than bees than we sometimes make ourselves out to be.
[I'm in full agreement with this, and don't think it disagrees with what RAH was saying either -- PeterMerel]
According to TheThirdWave, specialization in the Industrial Revolution meant long automated factory production lines making hundreds of items, all the same. Nowadays, it means long automated factory production lines making hundreds of items, all different.
Alexander is very keen on specialization; he doesn't like to build with components that can't be tailored to the exact circumstances at hand. If you miss this, I think you've missed the point of his patterns - they're not boiler-plates; they're adaptable. At the same time, he intended that his pattern language would enable ordinary people to have a hand in designing the spaces that they live in. Thus he wanted non-specialists - non-architects - to do architecture. -- DaveHarris
I think it's a bit more subtle than that. Alexander believes in the ArchitectAsMasterBuilder?, as contrasted with the ArchitectAsMasterPlanner?. Yes, the inhabitants have a big hand in the planning. Note planning is a process, not a skill, and the result isn't a "plan," but a dwelling (like Eisenhower said, "Plans are useless, but planning is invaluable"). The skill of workmanship is still in the architect's hands. It is for lack of such specialization and artisanship, along with an inability to remove the broader social process controls, to which Alexander credits the "failure" of the Mexicali experiment. RichardGabriel talks a little bit about this in his PatternsOfSoftware book, and you find quite comprehensive coverage in GrabowsBiographyOfAlexander?.
Whether this mastery is inborn or learned is an interesting side conversation.
I actually believe that to misunderstand this is to misunderstand patterns as Alexander has actually always thought of them, though it's not very pronounced in his writings, particularly his early ones.
I think the metaphor for us is for our special non-specialists, our customers, to participate in the architecture of the systems we build for them. But specialists still build them. That, by the way, was the vision of the software folks who came together a few years ago to explore how Alexander's ideas might apply to software, and that led to the PLoPs, the books, and all that followed. -- JimCoplien
I don't disagree with any of that. Of course, calling customers "non-specialist" (or the more common "non-technical") is a bit chauvanistic because they are specialists in their own businesses. Which is why they are often the best people to make decisions about what they want. Alexander was trying to restore power to the people best able to use it. The right power, that is to say; for decisions that lie within their speciality rather than ours.
Which is also a theme of TheThirdWave. There it is sometimes called "decentralization". I agree with the earlier comment, that efficiency comes from specialization. This is especially clear in code optimization. Most (all?) optimiszation comes down to partial evaluation, which is a form of specialization. -- DaveHarris
I vote for breadth with humility. Specialize in one, perhaps two, subdisciplines, but keep abreast enough of the others that you can ask intelligent questions. No, I'm not a professional C++er, but I know enough about the language that I can talk to one. I wouldn't trust myself to write a serious computation program that required numeric analysis to be accurate; but I know enough about numeric analysis to recognize such programs, and to seek out somebody who does. A little knowledge can often give you the wisdom to recognize problems that require a specialist.
The Scylla is profound depth with no knowledge outside your specialization; the Charybdis is being "a mile wide and an inch deep". -- BetsyHanesPerry
A marketing friend told me that over time he was coming to know less and less about more and more, until eventually he would know nothing about everything; while his grad student friend was learning more and more about less and less, until he would know everything about nothing. -- AlistairCockburn
It seems that in organizational patterns the unit of survival is the organization, not the individual. JimCoplien is right concerning what is best for an organization. What is best for each of us in our careers is another issue. It is striking to me, that there does not seem to be anything about the content of Jim's C++ idioms book and his organizational patterns work which would lead an uninformed reader to believe that they shared the same author. They both reveal deep understanding which could have been core grokking by two separate specialists.
The goal of an organization seems important, along with its lifetime. If the organization is a project team that will be disbanded quickly after development, then recruitment of deeply specialized team members seems optimal. If economics precludes this optimal recruitment strategy or there is a longer commitment involved, adaptability may be a more important value than specialization. Does a beehive have more evolutionary advantage than a set of adaptable humans? No. But we don't ask much of beehives. If the level of commitment is not too long, deep specialization seems optimal for project work. It would be interesting to see where this breaks down. I think it should.
All in all, I think that depth is optimal for organizations, but I don't think that it is optimal for humans. Supra-human structures depend on us as a substrate of variability. -- MichaelFeathers
See SpecialistsAndXp
Perhaps this shows a need for both "tall, thin men" and "short, fat men" in software. I think that we need to look again at how people are being used. I want a BrandX database tuner specialist when my queries are running long, but I don't want them to write C++ code for me because it's too peripheral to their interests.
But it's a matter also of prevailing conditions of certainty or uncertainty (everything is). In a project where the technology base is fairly well understood, or where there are already some reliable local experts, I want generalists. Where there are parts less well understood, we need some specialists. This goes with the appropriateness of waterfall verses Iterative development. The less you know the territory, the shorter your steps should be.
This goes to volatility, which seems to be the other prevalent software development concern. If the technology base is rapidly changing, you need technologists to help deal with that. You need them to spot trends that you can capitalize on with abstractions to create some false stability. When volatility is low, generalists are the only need. When volatility is high, you need some specialists also.
But I don't know what the specialist/generalist mix should be on any particular project. It depends on the project.
The dung beetle is a specialist. It has a special lifestyle and diet that would be poorly suited to life in northern Indiana in winter.
But look at cockroaches and termites and ants and flies. Where do they *not* live? They're generalists, and they've adapted to many different environments. If they stay in one place "too long", they adapt uniquely to that place and its environment. They had the basic generalist preparedness, physically, and then developed "domain expertise".
I'm not sure that's a bad thing altogether.
Tim
There is not any species of insect that can live in all the places that people can. (Except for lice. :-) ) They are all specialized. "Cockroaches" is not a single species, but rather many species. Humans are much more adaptable than any species of insect. That is the point of the title of this page. And it is a fact.
The question is whether it is a fact we should emphasize. I think there are many more problems in software because people say "I can just pick up a book and learn it" than there are from depending too much on experts. When you *do* have a problem with an expert, it is usually because he thinks that his expertise in one field will carry over into others. This is not a problem with expertise, but rather with hubris. -- RalphJohnson
There is specialization and then there is over-specialization. Two thoughts on the badness of over-specialization:
Humans must also have the maturity to determine what they want to do and not let idealists force them into doing other things just to avoid being labeled an insect.
I see this discussion drifting away from the central subject, which I gather to be a question of the viability of specialization for individual humans. The question prompted by that Heinlein quote is not, "Is specialization best for organizations?" but "Is specialization best for individual health?" Perhaps organizations benefit from specialization -I think they do - but am I a better human being as a result of my strong specialization? -- BrentNewhall
First you have to ask what do you want to do? Do you want to take out the trash, wash the windows, and clean toilets? Or do you want to be the sous chef, pastry chef, prep cook, manager? You should have experience at everything. But at some point you know what you like to do and may prefer to do that. -- Anonymous
I think the issue is deeper than that. I'm beginning to think that it's unhealthy to focus exclusively on one job, even if you like it and want to. -- BrentNewhall
Please don't try to save me. I can make my own decisions. I would also submit that doing programming is the job, switching from one area to another doesn't make any difference.
See also HumbleInsect
EditText of this page
(last edited November 5, 2014)
or FindPage with title or text search