Official Certified Double Blind Peer Reviewed Published Study

Slang for an intense and carefully-documented and reviewed study to "prove" that a given technology is "better" or that a given statement is true.

This "slang" is used exclusively by TopMind, and almost always used as a derogatory reference to the standards for evidence that science has set for itself.

That's an inaccurate statement. I usually use the term in reference to situations where somebody wants super-tight evidence in one situation but accept super-loose evidence in another, such as ArgumentByElegance. In other words, inconsistency or hypocrisy in evidence level requests.

It would be really nice if we had rigorous studies about contentious paradigms, techniques, tools, BUT, we usually don't. I'm just asking that the same standards and criticisms be applied to both sides. Just because I don't have an OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy for my position does not dismiss your side from the same burden. Example:

Person A: "The sky on Venus is pink."
Person B: "No it's not; it's green."
Person A: "Do you have an OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy that shows it's green?"
Person B: "No, I don't."
Person A: "I knew it! See, it's pink."

It should be obvious what's wrong with the last statement.

(Disclaimer: Some of the above has been added to since the replies below.)

-- top

[FacePalm]

Please take something for that reoccurring headache of yours. I have no problem with "high standards" for science. This is one reason I support RaceTheDamnedCar.

[...Whilst spouting random idle speculation like, "I'm confident a document processor built with a RDBMS would score equal or higher on most representative numeric maintenance metrics thrown at it." Can you say "hypocrisy"? Good, I knew you could. How about that TableOrientedProgramming GoldenHammer HobbyHorse of yours? Show us sound metrics in favour of it. Maybe it's time for you to RaceTheDamnedCar, eh?]

Roughly 99% of this wiki lacks OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy to back statements or suggestions of tool superiority. Why limit fussing about that fact to just me? If you are going to crucify me for not having OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy, then to be even-handed, you'd have to nail everybody up there. You'll probably have to make the cross out of reinforced metal.

[You're the only participant who makes such prolonged and consistent efforts to criticise others' favoured techniques, paradigms, approaches, tools, languages, and so forth -- and typically with so little evidence or constructive basis. Therefore, it seems only fair to make you rigorously defend your HobbyHorses to the same degree that you demand of others.]

Also note that I believe WetWare is a large component of software maintainability, and each individual has different WetWare. A universal OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy may suggest fits for a "generic" person, but says little about individuals. If a given shop tends to hire a certain personality type, which is common in my experience, then a general-population study could be of limited use. The tools that work best for visual thinkers may not for linguistical thinkers, and vice versa. -t

[Really? Please provide an OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy with supporting evidence.]

FacePalm

[Uh huh. Furthermore, if you "believe WetWare is a large component of software maintainability, and each individual has different WetWare", why do you criticise individuals who express the nature of their WetWare by demonstrating preferences for, say, ObjectOriented programming or LispLanguage? Doesn't that seem a tad, y'know, hypocritical?]

I usually don't. It's when they imply there is some objective metric or universal truth of superiority that the fireworks starts. It usually comes in the form of, "If you were smart like me and you read all the wonderful obscure books I have, you'd just know X was better. You'd just feel it in your bones. No need for numeric metrics."

Once again, topper, you argue in bad faith. In talking about tools, languages, and techniques you use PersonalChoiceElevatedToMoralImperative. You choose to completely ignore the work of dedicated professionals whose whole careers have revolved around gathering the statistical data to support their positions. You pretend that the entire field of science involving engineering process as related to software development does not exist. Instead you continually throw this bullshit "wetware" argument at us as if that has some merit, despite having no support. And you wonder why it is that people are getting tired of you on this board. Hmm. Can't imagine why.

Sorry, but for the most part they do not use "statistical data". The contentious works are usually ArgumentByElegance. And I didn't say that the field doesn't exist, only that it's very immature. I didn't create topics such as DisciplineEnvy; others have agreed that certain kinds of studies are generally neglected either due to cost and/or due to being against the culture of research institutions. If you have numerical metrics (GoodMetricsProduceNumbers) that cover a sufficient number of useful aspects that show my pet tools are full of it, please present it. Stab me with numbers, not platitudes and insults. -- top

Why do we need to go through this argument again, top? As has already been endlessly pointed out before, Deming, Fagan, Crosby, the QFD folk, and lots of others have published their findings -- including statistical case studies to no end. You are simply ignoring the inconvenient truth that the data are already gathered and have been published, including the interpreted findings. Since you continue to keep your head buried deeply in the sand and speak from this ass-high position, you once again earn a TenSeven from the Wiki community. Oh, well.

If I remember correctly, those are only available for a fee. There is no use discussing those unless all readers and writers of wiki somehow have access. Readers and I have no way to verify their alleged greatness. Also, many don't appear tied that closely to source code issues. (Somewhere around here was a list and links I vaguely remember.) -t

See, this is exactly what I'm talkin' about. There are white papers covering this topic all over the internets if you care to look for them, but you can't even be bothered to do that. Why should we be bothered to discuss the matter when you choose to remain ignorant? Waste of time.

{I suspect you were aware of that conclusion before you engaged TopMind. So, perhaps you should seriously ask yourself that question, rather than TopMind.}

If you argue point X, it's YOUR job to find evidence of X. "The truth is out there somewhere on the intertubes" is not a valid debating practice. I'm not going out to find your saucer-driving unicorns. If you are too impatient to gather evidence for your own damned case, that is NOT my fault and shouldn't be dumped into my lap. Be an adult. Vacuum your own house or hire a bloody maid. - t

[While I agree with your last point (and I wish you would actually follow it when you make claims), the one who is arguing point X does not need to make evidence Y available. It is sufficient to give you instructions on how to find it. Any barriers to actually doing that are your problem.]

At least link to it, give a few paragraphs of overview, describe how it's relevant to topics here, maybe extract a sample or two, etc. "Just trust me" is insufficient to both me and readers. You at least owe a decent intro. -- top

[That's one way to give instructions, but it's not the only one. I have no intention of limiting myself to sources that allow links just so you can be lazy.]

Me be lazy? Mr. "I have solid proof that you are flat wrong and it's.......out there somewhere and it's your job to go on a David Carradine trek to find it." Try that on the jury; and then be careful not to drop the soap in the slammer. I don't know what kind of alien debate/discussion rules you are following, but please send that book back to Uranus with no return address.

[Yes, you be lazy. You proposed, as a minimum requirement, that the most work you have to do to get to the source was click a link. That's just you being lazy, especially since you don't follow that for your own claims. BTW, this isn't a court of law and they have different standards of evidence, so that analogy goes by the wayside.]

I don't know what you are talking about. Your whole approach here is just bizarre. You are strange. It's not a court of law, but you don't even come semi-close to the rules of courts and formal debates, or even informal debates. You are so far off from the ideal that it is just plain silly. Generally if you make a claim about X, then you are obligated to bring the evidence for X. "Here's google, you go make my case for me" is stupid and rude. -t

This page is devolving into Yet Another incidence of topper whining that nobody will bring him all the data that has been quoted for the last couple decades in CS magazines, published papers, presentations, all over the internets, and even on this board.

A previous contributor was correct; we've hammered on him sufficiently that any further discussion would be pretty pointless, eh? So, top once again earns his bad sef a TenSeven.

I believe your claim of ubiquity is highly exaggerated, like much of your claims in general. Select one such alleged study that you feel is the closest to CodeChangeImpactAnalysis and shows OOP or thin tables or FP or sets over bags (mixed bags/sets actually) being better, and follow GoodMetricsProduceNumbers, and I will spend up to $40 to purchase the publication (I'd need permission from wife for more) and examine it myself and produce a general description and some samples etc.; all the things that you lazy asses SHOULD have already done if using it to back your nebulas claims and call me every name in the book. -t

[I think you mean "nebulous". Here's one on sets vs. bags: http://portal.acm.org/citation.cfm?id=1142351.1142363&coll=ACM&dl=ACM&CFID=108084325&CFTOKEN=70445506 I'm sure your wife will be happy you've spent as little as $99 on something so vital for your personal and professional development as a year's access to the ACM digital library.]

Or read http://www.almaden.ibm.com/cs/people/jayram/papers/JKV-containment-pods06.pdf for $0.
[Sssshh!!! <whispered_aside>I'm hoping he'll want to subscribe to the ACM digital library.</whispered_aside>]

[Note that the paper doesn't quite give the experimental-result numeric metrics you're looking for, but it doesn't need to because it goes one better than that -- it gives meta-numeric results. It proves that the time required to determine whether the result of one query (of a certain kind, called a "conjunctive query") will be a subset of the result of another (this is the "query containment problem", which is important for query optimisation) can be calculated algebraically (and determined algorithmically) for set-based systems but not for bag-based systems.]

I'm not considering machine speed/efficiency. I'm more interested in programmer productivity. I forgot to include that as a restriction above, and apologize (although have mentioned it multiple times in other related topics). I've already agreed that the field of "Computer Science" has done well in terms of machine speed/efficiency, and have puzzled over why it has struggled to carry over to people efficiency. -t

[Ah, you mean things like http://portal.acm.org/citation.cfm?id=1181775.1181777&coll=ACM&dl=ACM&CFID=108119944&CFTOKEN=46196628 By the way, the paper I cited above puts a big nail in bag's coffin, in terms of automated query optimisation, regardless of programmer productivity.]

As discussed in our related topic, one can choose to set-ify specific schemas or tables as needed to get set-based optimization. But I won't focus on machine efficiency here. {This comment applies to another ACM link. There appears to be mistaken identity here.}
[FacePalm]

[ComputerScience is the field that deals with machine speed/efficiency, or more accurately computational and algorithmic speed/efficiency. People efficiency is the subject of SoftwareEngineering, HCI and the like. Studying it is expensive and difficult, and sometimes subject to curious constraints. For example, the university where I work recently introduced a stringent approval process for studies involving people or other creatures, citing potential ethical issues as a motivation. The result is that to do any new research -- regardless whether you're an academic or a student -- you must complete a ten page application and get it approved by a university Ethics Committee. It's a pain to fill out, and speaking as a member of an Ethics Committee, it's a bloody pain to approve. However, if your research involves no human subjects, it's possible to get away with filling out only the first two pages of the application and approval is almost guaranteed. As a result, myself and the other three academics in my office have refused to conduct or supervise any research involving human subjects. So, ComputerScience projects are fine, but we're not doing or supervising any SoftwareEngineering research that involves human subjects. I suspect our experience is nowhere near unique.]

I realize SE studies may be harder, but it's also likely where better applicability lies because the low-hanging-fruit CS studies are mostly done. But you seem to be agreeing with my assessment that human efficiency (developer productivity) is largely ignored by academics, regardless of the reason.

I'm curious, why couldn't something like the study suggested in WhyNoChangeShootout be done? It's not as if you are locking the participants in a room. That wouldn't reflect the real world anyhow because developers can draw on just about any resource they want. Just ask them to code to the spec.

--top

[Largely ignored? Hell no. A number of my colleagues have built an entire career out of studying human productivity issues. You seem to be making a concerted effort to ignore SoftwareEngineering publications, perhaps because their existence does not fit your beliefs. I even cited one such study, above.]

Well, then list them instead of "machine" studies.
[I listed one for you. Go to http://acm.org/dl and search for more.]
That's their damned portal. If you know about all these grand SE studies, then pick a good one out. Are you afraid to pick one? It looks so.
[FacePalm... That is a good one.]
You are making zilch sense, and causing yourself skin problems. Why should I have to sift thousands of studies to find one that somewhat matches the criteria I outlined? You are the one who made the claim such already exists. Just finger the fucker instead of yap yap yap. Jeez! That's like asking me to send out search parties of all forests to verify bigfoot doesn't exist just because you claimed he does. Fuck that to hell! Stop trying to spin it into my burden using childish games. It's your burden, Mr. Rube Goldberg.
[Why should you have to "sift through thousands of studies"? Because I am not going to sift through any studies for you. I gave you a link to a sound study that employs numerical metrics about programmer productivity. I gave you a link to a site where you can find thousands of peer-reviewed papers on research into programmer productivity. Anything beyond that is up to you, because it is your burden. I'm not your research assistant.]
That "task context" study probably says nothing significant about paradigms or language choice. I reject that it's my burden for reasons already given. This is over.
[So you're not even going to make enough effort to read it to determine whether or not it says something significant about areas that interest you? Fine, but you lose the right to complain that programmer productivity is, to use your words, "largely ignored".]
Like I already said, I will only purchase it if you can give me reasonable assurances that it fits the stated criteria. So far it looks more like a project/task management tool with some hook-ins to the Eclipse code editor. That's not a topic that I am currently interested in, at least not as a C2 debate. I don't understand why you referenced that particular one. It seems almost randomly-selected. I am not seeing what you are trying to get at with it. What does it allegedly demonstrate about some of the heated debates we have about evidence and academics? -t
[Indeed, I did choose it at random -- as a clear illustration that there is academic research into programmer productivity that produces numerical data. I don't recommend you purchase that particular paper over any others, but I do recommend you obtain a subscription to ACM's Digital Library so that you can read the literature, in areas that interest you, for yourself.]
Yes, everybody probably ideally should, and floss and eat green leafy vegetables every day also. That aside, it's still not my job to search for your side's evidence during debates. -t

[As for WhyNoChangeShootout, it contains its own criticisms. You might wish to read them. If you have a particular hankering for some research and it hasn't been done, I suggest you do it yourself.]

Sorry, but I don't find the criticism very convincing. With nearly infinite variations in styles, one will have to accept something as a representative. Get 3 coding teams for each style to reduce the chance of such complaints. ANY study of paradigm/technique A against paradigm/technique B is going to require picking a representative of A and B. That's a silly excuse because it would otherwise forbid any study comparison of A and B.
[Then feel free to conduct the study yourself. A local university will probably be delighted to have students to do the work and have you supervise it, as long as you're willing to share publication credit with one or more academics.]
So now you admit that something similar doesn't exist already?
[I don't know whether it does or not. Do your own literature review. I'm not going to do it for you.]
YOU claimed there was already sufficient research done. If you can't back that up, then go ahead and be barker instead of a biter. You are full of hot air.
[Huh? Where did I claim there was "already sufficient research done" in that or any other area? I claim there is a lot of existing research into human factors, including programmer productivity, which you seem insistent on ignoring. However, there should be a lot more research in every area of Computing.]

[Re "low-hanging-fruit CS studies are mostly done", that's pure foolishness unless you have some magical insight into what studies will (or will not) be done in future.]

I didn't dismiss that some CS breakthrough work will come along, but it is harder to find such and SE is an open prairie of opportunity in comparison.
[How do you know it's "harder to find such"?]
PostSeventiesSoftwareEngineeringIdeaSlump
[That page is dueling rants, not evidence.]
Regardless, SE gets the shaft at the expense of CS because academics are not being daring enough.
[No, you are not daring enough to do your own research -- starting with a literature review -- on a subject that interests you.]
I don't believe such actually exists (or at least is too sparse and narrow), and I'm not going on a unicorn hunt to verify your claim that unicorns exist. You expect the jury to go to the library to research your claim, which is silly. Perhaps it would be "nice" or "convenient" if they did, but you cannot insist on it. Your sense of burden is off base here.
[Your attitude speaks for itself.]
Projection, rudeboy.
[Your attitude, it seems, is being particularly talkative.]
I've been accused of far worse things.
Further, I believe benefits ultimately come down to individual WetWare. There is no magic universal formula or GoldenHammer. I believe the best productivity would come from mind-specific tools and mind-specific conventions and the ability to find and hire like-minded individuals. Edward Yourdon's study of SmallTalk productivity suggest this is the case. Early users of SmallTalk scored fairly high in his productivity metrics. However, when SmallTalk became fashionable, many people flooded into it, and productivity flat-lined to match a "typical" language. One tentativeness explanation is that people who pick a language/paradigm/technique/tool because they like it are more productive than those who select it because it's in style. Tool favoritism would make in interesting study, but I don't have the resources to pull it off. Studies with just "code counts", such as CodeChangeImpactAnalysis, may not show any significant objective differences. While this may demonstrate that there are no (objective) golden hammers, and quiet some zealots, it won't deal with WetWare issues, and that's why I'm not that interested in them. WetWare studies, which would hit far closer to my interest and claims, are far more expensive, and beyond my means. (A thorough code-change-impact study would probably also be resource-intensive, per WhyNoChangeShootout.) -t
[You will find "Wetware studies", like the one I cited above, in the literature. Whether they answer your questions or not is for you to find out.]
"Like"? I think not. Your arrow is really a boomerang. Your sample studies are a horrible selection as far as the kind of topics we usually argue over.
["Kind of topics???" Is there a theme to our arguments? I hadn't really noticed, other than to note your dislike of object-orientation and truly-relational databases, and an inclination to disagree with anything that isn't promoting ExBase or some close cousin to it. These days, that's pretty much everything. I've posted links to two articles that deprecate bag-oriented databases, one here and one on the BagAtational page.]
Moved replies to BagAtationalDiscussion, under topic "Internal Versus External Bags".

CategoryRant

OctoberTen