Scientific Sins

In your arguments, be wary of these scientific sins. These sins are of the venal variety (rather than the mortal sins of plagiarism and fabrication), but I see them committed all the time. And I certainly have committed them myself, I must confess.

Note that these are not the same as the FallaciousArguments, though there is some overlap.

These sins only apply in the context of scientific claims. That obviously includes formal research and publication, but it can include some of what goes on here on WardsWiki. It also includes many claims put forth in marketing materials and industrial literature.

As the craft of programming and the discipline of SoftwareEngineering (and much of ComputerScience) involve many human factors (psychology, physiology), economics, and a whole lot of EmergentBehavior (in many ways, we're a soft science), drawing rock-solid conclusions by repeatable experiment or deduction from axioms is frequently difficult or impossible. This is one reason, I suspect, that snake-oil is far more frequently peddled in our discipline than in, say, chemical engineering.

Feel free to add your own.

Extracting principles from anecdotes. It is an all-too-common mistake, among non-scientists, to draw conclusions from small sample-sizes (including a sample size of one). That can be excused, as many non-scientists are not trained in statistics and the ScientificMethod. However, many scientists do this in informal argument, which is harder to excuse. Just because you think that language X is more expressive than language Y, and so does your pair and all the other programmers at your consulting firm, it doesn't make it universally true.

Extracting absolutes from probabilities. As parts of our discipline are soft, many conclusions that are reached through valid empirical research are statistical/predictive in nature, not absolute. (In addition, the experiments involved are difficult, expensive, and/or time-consuming to conduct or repeat). Despite this, it seems whenever a study gets published claiming that tool X made a certain group of programmers n% more productive than tool Y under certain specified conditions, advocates of X will invariably treat this as absolute proof that X is better than Y. (And if another study reaches a different conclusion, Y advocates will react the same - and then the two camps will debate whose study is "better"; though it's a safe bet that the debate which occurs will be shallow and superficial, rather than a solid critique of either study's methodology). Many other narrow scientific claims are similarly exaggerated.

Confusing conjecture with conclusions. Conclusions, here, refers to things which the scientific community has established to be true, depending on the hardness of the science. It may be axiomatic truth, proven from other axioms. It may be a repeatable experimental result that is shown to be true by the weight of numerous trials. It might be a statistical observation which, although not universally true - see the previous sin - is still scientifically significant. In all cases, the conclusion follows from research - research which must be documented both in methodology and results, so that other scientists can confirm the experiment. Conjecture, on the other hand, is little more than opinion. Informed opinion in many cases, but it's a belief that hasn't been confirmed by experiment or formal proof. Conjecture itself is not bad - many conjectures ultimately turn into theorems (e.g., FermatsLastTheorem, which until recently really was Fermat's Last Conjecture) - and many others are disproven. What is bad is when a) someone asserts that a conjecture is scientific fact, simply because it is conjecture offered by an expert, or conversely when b) someone asserts that something which is a demonstrated conclusion is just some conehead's opinion.

Dismissing probabilities as worthless. The opposite of Extracting Absolutes from Probabilities is the tendency of practitioners of "hard" sciences to dismiss the soft sciences (and the experimental results thereof) as worthless, just because the conclusions they reach are statistical in nature rather than absolute laws of the universe. While science cannot "prove" that GUIs are more "productive" than CLIs, for instance, science can show that X% of a random sample of users are more productive at tasks A, B, and C when given GUI G over CLI L. Yet whenever such a study appears, defenders of CLIs will immediately dismiss it. If the conclusions are narrowly drawn (which they should be for a study like this), then it's claimed that the study doesn't apply - even if numerous such studies, all supporting the same general conclusion, appear. If the conclusions are widely drawn, the study is flawed (in the latter case, the CLI defender has a legitimate gripe, as the study author might be Extracting Absolutes From Probabilities, or at minimum has questionable methodology). Fields of study such as economics and psychology are most certainly not worthless - given that the experimental results of these sciences have been successfully used to solve numerous problems.

A related sin occurs when someone (often not a practicing scientist) asserts that because a science is soft (or has soft elements - especially when HumanFactors are involved), one cannot really any draw firm, empirical conclusions about the subject matter. Hence, the opinions of laypersons (especially ones with a superficial knowledge of the subject matter) should be given equal weight to the writings of practicing scientists; even if the scientist follows the ScientificMethod to the letter in reaching his conclusions, the result is by necessity "slippery" enough that it cannot withstand any challenge. No real evidence is offered in support of such challenges - just anecdotes and opinion - but it is asserted that even the existence of a differing opinion is sufficient to invalidate any scientific finding (or at least reduce it to the level of informed conjecture; as opposed to the loftier status of empirical evidence). References to the literature are often derided as ArgumentFromAuthority, as if someone was quoting a TV star or other (lay) celebrity on a scientific matter. The ultimate conclusion of this assertion, of course, is the proposition that a majority agreement (if not consensus) among all interested parties - including opinionated non-scientists - is required before any claim in the discipline can be considered authoritative. Practicing and reputable scientists, of course, know full well the scope of their research. But just as it is sinful for a scientist to offer conclusions unsupported by the data; it is also sinful for others to offer the decidedly un-scientific proposition that no conclusions can possibly be generated at all.

Citing deprecated evidence. Many folks in the humanities and other areas (and a few scientists who ought to know better) like to quote scientists and scientific studies to bolster their own arguments. One problem that frequently occurs, especially among non-scientists, is that rather than inquiring of knowledgeable scientists as to the prevailing wisdom of the field and using that, many instead choose to quote scientists who a) were the ones they were exposed to in undergraduate "core curriculum" science classes; and/or b) tend to support their own agenda. In many cases, the scientists that are cited are ones whose work has been deprecated, obsoleted, or even considered by modern practitioners of the discipline to be complete and utter bullshit. Examples abound in PoMo circles, where SigmundFreud, KarlMarx, and other leading lights of the 19th century are cited as vanguards of modern scientific thinking, in support of some proposition. Modern psychologists regard Freud (and modern economists regard Marx) much the same way that modern astronomers might regard Ptolemy or modern physicists might regard Plato - historically interesting, but of no evidentiary/empirical value whatsoever. Sometimes, this is done by scientists who ought to know better; when this occurs, the scientist has stopped practicing science and is instead practicing PseudoScience.