Collaborative Decision Making Software

PhilipGreenspun asked, on his blog:

Consider a university teacher. She wants to hold a class in which the students work together to solve a problem. To make the example concrete, suppose it is an economics class in which the students have to come up with a $1 trillion stimulus plan for the U.S. Or suppose it is an engineering class in which the students have to come up with a design for an exercise machine that can function as a first-person shooter videogame controller (my personal favorite solution to America’s obesity problem).

During the class, the best student ideas will need to be highlighted and expanded upon. The unworkable ideas will need to be pushed into the background so that they don’t clutter the discussion. If you ask Jane Professor how many students she can accommodate in an “active problem-solving seminar”, she would probably say no more than 15. Why so few? Jane is going to have to be the one to direct the discussion, suppress truly bad ideas (like my personal economic recovery plan!), highlight the best ideas, and summarize some of the results.

The students should enjoy a course like this because their efforts might in fact be productive. Perhaps the engineering course is sponsored by a company that can produce the product if the design is good enough. The students should also learn more from this course than from a standard “here’s some knowledge handed down from on high” because they have to be active thinkers. We’re so enthusiastic about this method of learning that we tell Jane Professor we’re giving her 100 students intead of 15. What does she do? Go back to the traditional lecture and homework format. She can’t moderate a discussion among 100 students. Even if she were to run a standard Web-based discussion forum system for the 100 students it would be too much work to moderate. The loudmouths and first answers would dominate, not the best answers.

Two questions then, for the assembled readers…

At a minimum, the software would need to provide distributed features to allow some ideas to be flagged as more important. Perhaps the authors of those ideas would thereby earn some “karma points” (slashdot style) that would enable them to moderate up or down other ideas.

One big area where I think traditional discussion forum software falls short is that there is a strong presumption in a discussion forum software that the most recently posted content is more important than older stuff. If the challenge for the semester is a complete solution to a design problem, every part of the solution is equally important, whether that solution was developed early in the semester or later. Also, we need something that lets a group of people collaboratively develop a hierarchy.

Wikipedia seems like an example of a collaboratively developed solution to the problem of “how do you write an encyclopedia”. There isn’t very much identification of content with authors, however, and it becomes tough to know who should get credit for various pieces of the solution (in a university course, you need to make sure that each student is doing something!).

Perhaps you should get fifteen students together for a semester and to figure out how to make collaborative decision making work with 100 students...

I see a different problem with the large-format class. It’s not so much that it’s difficult to determine who was responsible for a good idea. It’s that it’s difficult — if not impossible — to tell who among 100 people has learned anything.

Among 15 people, even those who never hit on anything in the brainstorming sessions might learn what sorts of ideas work, or even how to work with a group on a collaborative project like this (and like one they might find in the real world). To put it stronger, it’s possible to help people learn in a small group, where it’s impossible to respond to 100 people at once.

I think it’s related to another point that distinguishes this from Wikipedia. Those who write Wikipedia articles are already experts in various fields, or they’re good writers who can help massage the output of the experts into something more easily digestible. The point is, most people writing Wikipedia entries are not doing it to learn how to write Wikipedia entries. They’re applying an existing skill.

--JohnArmstrong?

What about a dual level forum: intra-group unmoderated by the instructor and inter-group moderated by the instructor and seen by the whole class. The intra group forums would have to be initiated by the instructor maybe with a question or sub-assignment and the group would have their discussion around the specific issue and when a consensus or impasse is reached they would submit to the main class wide forum.

This allows study groups to informally work together but with some overall direction from the instructor. It would also allow the instructor to look in on the individual group discussions to see if all its’ members were contributing more or less equally. The rest of the class would get to see and discuss each group’s progress at natural intervals without being exposed to all the details.

--demetri

1. teacher has to be involved in discussing proposals not ideas. a person/group should present a proposal for global discussion

2. there should be software support for forming groups. groups should be formed for

3. common tools for all groups:

4. specific tools

teacher has to bootstrap it, of course. and oversee. etc

--de

One possibly useful piece of software is Google Moderator: http://moderator.appspot.com/

Inside Google, we use this all the time for deciding what questions are most important in question/answer sessions.

--rif

My personal experience with software and the development of ideas is that contemporary software based forums significantly slow the flow of ideas. I spent several years in a technical graduate program comprised of both traditional and distance learning (including forums, video taped lectures, and classroom collaboration). Without fail, I found the most efficient method for developing ideas and communicating anything of substance is a small group with real-time dynamics.

Consider for a moment a small group in a room. The members of the group can quickly exchange ideas and build upon them. The size of the group constrains the number of ideas to a small enough number of abstractions that most in the group can quickly follow the discussion. If you introduce something like an online chat forum, you dramatically slow the pace of comprehension as everyone in the conversation is sits endlessly waiting for the next thought to emerge on the screen. If you add more than ten or fifteen participants, the flow of new ideas grows exponentially beyond the point of any single participants ability to conceive of everything being said. The net result is that the development of ideas is diluted by too many minds and a grossly inefficient delivery mechanism.

My suggestion is that to be efficient and useful, any software that moderates the development of ideas should provide near real-time communication enforce small group dynamics. Thus, if you have a group of 100 students, the software would divide them into six or seven cohorts that are each their own group. Each group should be provided with the equivalent of a video conference or physical meeting place and a mechanism for selecting and recording their ideas. The software might then provide a mechanism for selecting the best ideas from each group.

--Scott

Reminds me of an experiment: http://gowers.wordpress.com/2009/01/27/is-massively-collaborative-mathematics-possible/

--Ben

Lots of this functionality is already available, not specifically for students but as general support for engineering teams in companies. You could use something like reddit (open source at http://code.reddit.com/ ) for voting on and discussing various proposals; these sites weigh a combination of vote totals and recency when ranking items, so a well-liked submission would stay on top longer.

Regarding your concern about identifying content with authors, I don’t think this is something you can just tack on to a model of effective collaboration like Wikipedia. If you could objectively grade individual contributions to a group work effort, we’d have a lot fewer court cases about pay discrimination. If you’re going to make classes a lot more like real work, you might have to grade them like real work. The best way I can imagine is to have the instructor meet periodically with each student and assess that student’s progress in becoming a more valuable contributor to the class effort. The downside, which probably makes this approach infeasible in a typical university, is that grading will be highly subjective, with different students having divergent goals and no guarantee that your class can be an effective prerequisite for anything.

Really, if you believe it is possible to filter ideas in this way, what you want to do is start a consulting company (deja vu?). You bring entry-level people in as students, maybe even hold open RFP sessions online. Company team leaders prototype the leading proposals for each client, and meet with the client and each other to compare approaches and mind-meld. After that, a team is selected to execute the project. Evaluation is based on effective performance at each phase, and might even be done in a distributed fashion by letting each team leader or client vote on what people/teams to fund using actual money.

This isn’t much different from what typical management consulting companies do right now, but if you could shave a couple of years off of college and replace them with low-level work at this company, you’d probably be able to get a year or two of extra productivity out of each person as compared with the college + junior consultant career track.

--ShimonRura?

One solution to this I’ve considered in the context of improving public debates is an explicit way to structure arguments by requiring that any given assertion links to either other assertions or primary evidence. Think of it as sequent calculus for bridge building.

In a classroom setting, the top level solutions to the problem would be brainstormed, then people would be free to analyse the assertions ‘This would work’ or ‘This wouldn’t work’. After some time the most promising solutions would be selected by the teacher, and students would focus on a more detailed analysis i.e. higher quality argument of those.

By making the dependencies between assertions explicit it would be easy to trace the reasoning of an argument even when the various steps are written by different people.

For example, if the problem was ‘Get a payload into space for less than $1000/kg’ then the top level ideas might be 

  1) A Cheap rocket
  2) A fast high-altitude aircraft
  3) Space Elevator
And they could be analysed with people saying

“1) Isn’t possible because that would require a 100x decrease in launch costs compared with NASA [cite source]. My personal belief is that a 100x improvement in efficiency isn’t possible. [baseless]”

And people could further argue that there are prior examples in the car industry, or that the cost of fuel couldn’t drop 100x…

Using a pretty GUI it would be easy for everyone to see which suggestions are popular because there is much more evidence for success than failure, and which are popular because there are strong arguments on both sides.

--PhilipGreenspun

How about a real-time version of StackOverflow.com? I.e. basically the features of StackOverflow.com with real-time updates and some usability enhancements necessary due to the real-time feature.

It would work like this:

Just an idea.

--Florian

This is an interesting problem, and I’m curious to see what readers propose. However, I wonder if this flies in the face of Parkinson’s Coefficient of Inefficiency http://en.wikipedia.org/wiki/Coefficient…), and perhaps the class should be just kept to 15 students.

--Charless

You throw around 100 as a reasonable number just past 15, but in reality, there could be a very big difference between 50 and 100. In particular, if relationships count for a lot, you have four times as many with 100 as with 50.

If each one of the 100 students has something unique to contribute, you’ll probably have problems. But if there are, say, ten novel ideas at a time, the 20 students who have the most-popular idea can just say "+1" or something, and the lone student with idea 10 can still get a hearing.

Dunbar’s number

Keep in mind that Dunbar’s number applies to people who live in contact with each other in meatspace. If it’s tougher to keep track of relationships when they’re mediated by a text-only interface, then the effective Dunbar number in this case could be quite small — in particular, if it’s 15, you’re screwed.

Of course then you should just come up with a way to divide and conquer: splitting up into subgroups is pretty standard in workshop situations. Split up into groups of 15, have each group churn out ideas, then have each group list both their best and their craziest ideas, and try to merge the lists together at the end to find a couple best ideas. The software could help by making this asynchronous: at any given point, each subgroup could publish its best/craziest ideas so far, or ask (factual) questions of everyone in the group, or post any interesting facts or approaches they think might be useful to the whole group.

(Demetri touches on this above, as does Scott — though the number of ideas only grows linearly with the number of participants, the number of assimilations of ideas grows quadratically (O-n-squared), but since there are n people, the time to accomplish this grows linearly in n (which is still undesirable).)

A Wiki gives the communication and persistence capabilities this requires, but you’d also need some human rules for how to go about it: e.g. each subgroup has to decide (vote or consensus) which of its ideas to promote, each subgroup gets x representatives to the group-as-a-whole decision, etc. (Much of Wikipedia consists of rules for humans, not rules for machines.)

We should take this discussion to Ward’s Wiki: http://www.c2.com/cgi/wiki?CollaborativeDecisionMakingSoftware

--GeorgePaci

EditText of this page (last edited February 5, 2009) or FindPage with title or text search