Evidence Totem Pole

The idea that some evidence categories are stronger than others.

A possible, but not mutually-agreed-upon ordering is (strongest first):

Arguments about the order

What is the reasoning for this ordering? I tend to disagree strongly with some of the placements, but would rather hear the rationale than just to change it for my preferences.

It's the ordering of a Platonist.

{I thought it this was obvious. I would be interested to see an alternative ranking.}

I demand EmpiricalEvidence that this ranking is correct.

I don't think it has to be correct more than agreed-upon between 2+ parties with issues to hash out. There is no use arguing about the existence of a chair if both parties agree that it is there.

Why should there be a hierarchy? I would suggest that each of the above is a necessary component and the items form a self-reinforcing set. It is inappropriate to pull one item and declare it superior to the others with the implication that it can stand on its own.

It's only a rough guide. You are welcome to present an alternative ranking or scoring system.

Perhaps AdVerecundiam should be split into individual authorities and collective statistical surveys (non-filtered). "Bob says that X is better" is weaker than "4 out of 5 software engineers out of 1000 surveyed say X is better". I would place the second above "philosophical reasoning" because it is possible for something to be better without knowing why. Of course, such surveys cannot filter out the BandWagonSyndrome and other peer-reinforcement effects.

``I would expect that the statistical reasoning of `4 out of 5...` fits closely into the `thorough example` end of thing anyway.``

{The quantity versus the type of evidence are generally orthogonal. We can have one anecdote or a thousand, but we could also have one actual experiment/simulation of a "representative" example, or hundreds of representative examples. However, our current list is one-dimensional such that it doesn't represent both axises (axi?) very well (type versus quantity).}

A Ladder metaphore might be more useful. I don't think a real totem pole implies rank.

Man, stop kicking dirt into my real-life metaphores :-)

The above makes it sound like AdVerecundiam is "fallacious" (FallaciousArgument). It is weak evidence, not outright fallacious.

In the context of rhetoric, it's fallacious. That doesn't mean that appeals to authority are wrong; just that it means -- like you said -- weak evidence.

Re: "referring to a published scientific paper that contains the above (especially the first 3) is not AdVerecundiam."

I'm not that comfortable with "science==educated-votes". "Peer review" is merely a shortcut to more formal processes that are not otherwise practical.

It is not AdVerecundiam unless the person referring to a paper is saying: "Look! This author is right because he's an authority!". I find it far more natural to assume the person is saying: "This paper already presents a reasonable argument and evidence towards this conclusion, so I have no reason to repeat it in full here." And if you read carefully you'll notice the 'that contains the above (especially the first 3)' IS indicating that the paper being referenced happens to contain a proper argument and evidence.

I'm very much of the opinion that emotion cannot be placed into this list because emotion is often a very accurate guide - think of emotion as responses that are hard-coded in WetWare, responses that have been evolved over time. So long as the stimulus isn't too new, the instinctive response is quite likely to be extremely sensible. -- BenAveling

Studies referenced in the book "Supercrunchers" would suggest otherwise. It appears that human subject experts tend to glom onto a few "pet factors" and over-magnify them when making summary judgments.

What sort of decisions were they examining?

Fields in which there were "experts" involved in making decisions. Typical examples include sports scouts, financial, wine futures, and law (such as predicting court outcomes). And they also did studies with regard to human versus machine prediction abilities on "lab games". The "few-factor-glom" pattern seems consistent. It appears an embedded feature of human nature. (Related: NarrowStaffSelectionFactors)

Lawyers aren't "experts" at predicting court outcomes - they don't receive training in that field. Sports doesn't have measurable 'experts' (no tests to become a sports scout) so any claim of expertise is suspect. You certainly won't find human "experts" at lab games. And finance is one of those strange places where observations and making decisions strongly affects the outcome (making the domain highly subject to uncertainty principles) and where you can't point to any [Formal proof, Deterministic experiment, Statistical experiment, Anecdotal Experiment, Simulation/Modelling, Conjecture, or CommonSense] that does any better - the argument that "human subject experts tend to glom onto a few "pet factors" and over-magnify" seems like talking out of your ass unless you can point to the correct factors and the right magnification.

I'd be more interested in a study regarding human subject experts in chess or go - i.e. people who are measurably "experts" - making 'summary judgements' based on their chosen 'pet factors' and comparing these results to intelligent but untrained persons attempting to track statistical properties, formal reasoning, and conjecture. I'd bet significant money that the 'experts' with their 'summary judgements' would make strong moves and win almost every time, thus supporting BenAveling's claim.

Arguments about the WikiName

A totem pole is a carving that tells a story or family history. It may have other uses too. The animal at the bottom may or may not be more important then the animal on the top (as it is upon what the others stand. Without it, the totem is smaller.)

Why not use EvidenceOrgChart?.

Too confusing IMO. People will think it about org charts. Maybe EvidenceRanking??

http://www.dictionary.com

Main Entry: totem pole

1 : a pole or pillar carved and painted with a series of totemic symbols representing family lineage and often mythical or historical incidents and erected before the houses of Indian tribes of the northwest coast of No. America

2 : an order of rank : HIERARCHY

Main Entry: crucifix

: a representation of Christ on the cross

Main Entry: cathedra

: a bishop's official throne

Why a TotemPole? Does it have to be a religious structure? How would we feel about an EvidenceCathedral? or EvidenceCrucifix?

The assumption is that the totem pole represents a ranking or hierarchy of some sort with the most important beings on top. Crucifixes are not like this in design.

OTOH, there is plenty of cross-related imagery. WilliamJenningsBryan? 's "cross of gold" comes to mind...

I'm familiar with the usage ("low man on the totem pole"), but why use a religious reference at all when we're talking about evidence? Why not use ranking or hierarchy if that's what we mean?

Framing the concept in secular terms might suggest we could build such a hierarchy objectively.

{Further, I don't know of a decent secular visual/conceptual metaphor.}

Ranking or rank. It doesn't suggest objectivity.

Are you a Native American ("Indian") by chance?

1/32nd Siksika. Are you?

{So you are only 1/32 offended? That ain't bad.}

I think we could dispense with the totem pole if we concentrated on a specific application. Given the lack of context, the totem pole seems, symbolically, quite appropriate.

Does the "possible ordering" given below make any sense in my daily life? Can I apply this when I'm searching for a pair of socks in the morning. Or when I want to know what time it is? The fact is that many of us can go a whole day achieving personal goals and conquering small tasks without this ordering or something similar.

I believe it originated in discussions about how to demonstrate that one paradigm is "better" than another.

Heated argument centered around human competence at making guesses

Can't you disagree without being so rude? I'm tired of dealing with people who use such an aggressive style on this wiki. Go argue with yourself. My participation in this spot has ended.

When you misrepresent evidence or studies as supporting a point it does not support, you deserve a kick in the teeth. Can't you present an argument without resorting to sophistry?

I did not intentionally distort any info for the purpose of manipulation. If you have an issue with something, please state it without conspiratorial brushes. KeepCriticismNarrow.

I consider making statements in ignorance a greater offense to reason than intentional conspiracy. Sorry for giving you the benefit of the doubt.

Thanks for ruining what *could* have been in interesting discussion by using AspergersSyndrome-like communications "techniques".

Hello, poT. I'm kettle. Maybe poT could have responded to content rather than throwing an anal-expulsive tantrum about the 'rudeness' of an appropriate and well-deserved jab against his lack of professional ethics when it comes to presenting arguments and putting forth the appropriate degree of effort to ensure his references support the claims he's making. Maybe, then, we'd have had an interesting conversation. But, instead, it seems poT wishes to focus on slights against him... perhaps poT finds throwing mud very 'interesting', because it's certainly what he spends most of his time focusing on. Does poT ruin what *could* have been interesting discussions intentionally (like an intelligent man), or through ignorance (like an emotional boar)? Who knows. But do be aware that, just as a person has no 'right' to incompetence the moment he presents or sells himself as a professional, a person has no 'right' to ignorance the moment he presents himself in an argument for purpose of making a point.

I have no idea what you mean by "lack of professional ethics", but I don't want to know anymore. Your style so far sounds too similar to Lars who is offended by very hard-to-predict statements; and he's worn out his welcome with me. He lives in a funny little world with funny little morals. I can't stand having discussions with that......person. Have a nice day.

Funny, your style so far sounds too similar to Top's, who persistently presents flawed and misleading arguments then smugly says it isn't 'intentional' on his part as though he's incapable of learning to do better. He's really worn out his welcome with me. He lives in an irrational little world where EverythingIsRelative, including truth and evidence, all so he can justify in his willful ignorance whichever unsubstantiated opinions strike his fancy at the time. He IS, however, utterly predictable in his tendency to favor whining about minor slights (be they deserved or not) over focusing on useful conversation, and in his tendency to go for the last word no matter how unprofitable the conversation. I can't stand having discussions with that......person. Have a nice day.

[You two: Get a room.] Actually there'd be three people - probably two against one, which wouldn't be fair - so he'd have to forfeit. Top became the bottom.

Maybe a kickboxing ring... Debate Boxing, akin to http://en.wikipedia.org/wiki/Chess_boxing. I'd probably be able to stand TopMind's behavior if a judge were docking points for every fallacy and I could hit him between sessions.

[A couple of years ago, I designed a framework for hosting on-line Web-based structured debates. See InternetDebateProposal?. I really should implement it.]

Mutual hate, finally something we both agree on.

Utility of Experience

Experience can be useful evidence even if one cannot name all the individual cases or scenarios that produced experience. Neural-network-based systems, such as the brain, often have lossy learning by storing the fact that a certain pattern is/was common, such as by reinforcing weightings or links between axons, but not necessarily the details of each experience leading up to it. For example, if you create a training data set for an artificial neural net, there's no way to extract the "images" of each individual training card or instance. Yet, the net has "learned" to recognize the pattern(s) of the training set.

The more extreme experiences tend to be remembered, but not necessarily the mundane. For example, a bear may remember details of some fights it had with competitors, but not how it caught each and every fish it ever ate. Similarly, I still remember some of the specific mistakes I made in trigonometry because each red mark was a big mental "ouch!". However, I remember hardly anything specific about the problems that I got right. (It's possible the bear brain is very different from human brains, but bear with me for the illustration, pun intended.)

Experience judgment in humans can be useful; however, there are at least two problems. First, if the training set is not documented, there's no way for the reader to verify. Second is bias. Humans tend to be biased toward certain patterns and tendencies. We don't absorb all lessons equally, or at least weighted consistently across individual people. For example, some may remember the effort to fix a bad problem more than how loud their boss yelled at them, and visa verse. - t

For consideration:

Proof that visualizations, impressions, ideas and dreams involve all, or most, or some, or any of the sensations or capabilities we class as capabilities. To illustrate: I can not recall the color of, or in fact whether or not anyone I ever dreamed of as wearing their glasses, or for that matter, what color tee shirt, or sweater, or stocking cap they may have had on. What would prove I had? Would a photograph taken during that dream of the said object be evidence?

In today's newspaper a member of law enforcement in Greece was shown on the ground, surrounded by and engulfed in the flames from a gasoline projectile. The immediate impression I had, before opening the paper to see a photograph below the fold, was that he most certainly would have died from what appeared to be happening. The photo below the fold and the description of the event showed him escaping to safety and surviving the flames. In this case, first impressions did not prove to be evidence of a fatality, for subsequent facts disproved what seemed apparent.

Some evidence we amass is based on impressions of what we see, or imagine to be, in a moment, rather than what is in fact and evident, and demonstratable from what is or was.

Did the magician really pull a silver dollar from behind another one's ear? Or did any of the things appearing to have happened really happen? Is seeing believing? I would suggest that much of what we come to believe or espouse can never be proven fully and with logic. If it could be disproven, I suggest that the magic and livelihood of living would disappear.

Should FormalProof? be at the top of the pole? I'll grant that the two are rarely in conflict, but wouldn't broad-based experimentation trump formal proofs? If the formal proof says P and the experiments say not-P, then I'm convinced of not-P. Even formal proofs merely model reality.

No, correct formal proof is absolute and incontrovertible, and can be verified by automation. Empirical experimentation is always subject to error, no matter how small. Unfortunately, the degree to which formal proof is applicable to the real world varies. In mathematics and computer science, the relationship is one of equivalence. In physics and branches of engineering, it should be the same, but the number of variables involved (some of which may be unknown) complicate this.

SoftwareEngineering also has many variables. In fact, I'd say it's harder to objectively measure than engineering designs because engineering is heavily tied to physics, while SoftwareEngineering is largely tied to WetWare. It's only a little bit about computing machines and math.

SoftwareEngineering is a branch of engineering.

I'm not sure I agree. Somewhere there's a topic or two on that. Engineering is much more heavily guided by the laws of physics, while software engineering often revolves around WetWare, which is a very soft science so far.

Mathematics constrains software engineering in pretty much the same ways that physics constrains, say, electrical engineering. There are also plenty of "WetWare" issues for the other engineering branches.

Other than user interfaces, such as (real) control panels, what are common examples? And while I agree that math constrains software design, it's not a very narrow constrainer. It still leaves a hell of a lot wide open.

The usual ones you point out. You still have to be able to communicate your designs to co-workers. You still have to have domain knowledge. You still have to find solutions to poorly specified problems.

That's true, but those appear to be smaller issues in physical engineering. Writing software is primarily about communicating with humans. The machine is secondary.

The only one where the issues are any smaller is in communicating designs. The methods used to communicate are largely set, so there's not much need to argue about which ones are better. The others are, if anything, larger issues. The cost of being wrong about them is larger for the other engineering disciplines.

That's a big one. Making code grokkable and team-maintainable (often related issues) is the bulk of the effort. Well, understanding customer requirements is a big problem too in my experience, but that's typically not something we heavily debate about on this wiki.

See Also: EvidenceVersusProof, EvidenceEras, GoodMetricsProduceNumbers, OfficialCertifiedDoubleBlindPeerReviewedPublishedStudy, AnecdoteImpasse, AreWeaklyBackedOpinionsAcceptable, ProxyFactor

JulyZeroEight

CategoryEvidence

EditText of this page (last edited December 23, 2014) or FindPage with title or text search