Open Source Project Organization

moved here from DistributedSoftwareDevelopment

The GNU and Perl 5 efforts managed to pull this off, too. I think they used CodeOwnership (one guy owns the code) instead of heavy duty software configuration management tools (though I know the Perl 5 effort is using a Web-based interface to PureDDTS.)

Both projects also have some fairly rigid and specific standards, e.g., for makefile conventions and source code indentation.

-- PaulChisholm

Where did you get the above info about Perl5? I've been on the Perl5 porters mailing list for a few years now and the above doesn't reflect what the Perl5 porters have been saying and doing for the past couple years. How old is the above information?

They don't use a web-based interface to PureDDTS. Once upon a time LarryWall used a DDTS database for bug tracking, but that was awhile ago, and there wasn't a web interface for it (I got a chance to speak with Larry about this a little bit at the 1997 Usenix LISA conference in Chicago).

Truth be told, they haven't been using too much of anything as a separate tracking tool. About a year ago, Nat Torkington looked into something using MySQL and a freeware tracking tool called PTS or XPTS, and a few other freeware tools (I had some involvement in that as well -- albeit very limited).

In May of 1997 I managed to use some of my contacts in the SCM world to get the folks at Perforce Software (see http://www.perforce.com/) to donate the use of their SCM tool (PerforceVersionControl) to the Perl5 porters (as many licenses as they needed at the time, about a dozen or so). Perforce (a.k.a. P4) is most assuredly a heavy-duty configuration management tool, though its not pricey and its not heavy-handed with process (but it is heavy duty in functionality, just not as heavy duty as ClearCase ;-).

Perforce also has some good support for problem tracking built into it: the notions of "jobs" and "change-lists" (see the P4 web pages for more details). They use this facility for bug-tracking, though they are trying to find a nice WWW interface for submitting and following-up problem-reports (JitterBug was one possibility bandied about at an earlier time).

For now, Perforce seems to be filling most of those needs quite well, though we will probably regularly revisit the idea of putting a web-interface on things and/or integrating with a database like MySQL. But Perforce has been making enhancements to the "jobs" feature so we may just stick with that if it keeps improving (or maybe require only a thin layer to interface with it from the WWW and email).

And the Perl5 porters dont really practice CodeOwnership, at least not in the way its originally described (one could argue it is a distant cousin in some respects). They have something called the PatchPumpkin? and a role called a PumpkinHolder?. For a given release, a lone Perl5-porter is designated as the PumpkinHolder?. They "own" the PatchPumpkin?, and are responsible for applying all the patches and changes to go into the next release.

Numerous individuals have permission to check files in and out of Perforce; but for the branch (codeline) being used for a given release, the PumpkinHolder? is the one with final approval for what goes into the codeline. The role of PumpkinHolder? rotates among individuals from release to release. So rather than CodeOwnership, one might call this ReleaseCodeOwnership?.

For individual Perl modules that are distributed along with the core, CodeOwnership usually is practiced, but there are many exceptions (especially for patches and fixes).

Oft-times issues will be bounced around on the perl-porters mailing list, or on the core developers mailing list (those who actually have checkout permissions). If things get very heated or deadlocked, then often it will be up to LarryWall to make a ruling, but far more often than not, Larry will sit back and let things work themselves out.

Having the mailing list serve as a VirtualForum? for frequent and informal dialogue, group decision-making, and team-coordination, is one of the single biggest keys to the success of this particular distributed development effort.

-- BradAppleton

What can we learn from Linux?

Independent thinking, is right on the mark.
Modular extensible design, I can't speak to but I suspect exists.
Excellent tools, seems almost wholly absent. The distributed toolset is least-common-denominator FTP and email.

Perhaps I am seeing what I want to see, but it appears that a gelled community trumps tools. I suggest that item 3 should be Unity of Purpose among a dispersed team.

-- MarkDelaney

Item 3 is possibly obsolete. The cutting-edge GitVersionControl now used allows safe distributed development. Submission of patches to a mailing list for incorporation to the trunk is pretty much the standard practice for large OpenSource projects.

In Linux' case (and most of the successful shareware systems), the Unity of Purpose is provided by a smart person who controls the main repository. Is this a requirement for distributed development?

-- SteveFreeman

Linux is very modular; the times I've tried my hand at kernel hacking, I never had any system crashes in totally unexpected ways. If I last tweaked memory allocation, the system would crash in malloc. If I buggered with scheduling, the system died at a context switch. (Not to say that every version crashed and burned, but I learned that I'm not a kernel hacker.)

I think whether or not it's possible to have multiple maintainers for the main repository isn't the issue. I think the real issue is that it's non-parallelable work, and if you try to do it in parallel, you'll break it. I say this having known some people who could manage that type of thing, but they do it by using mutual exclusion. The only benefit you get from that is an increase in your TruckNumber.

However, I'd argue that you don't even get this; if the changes between patches is so great that only the main repository janitor knows all of the code, you'll have problems anyway. You really need to have several people who know most of the code, and in the case of a Truck, one of these should be able to fill in almost as easily as a co-maintainer, and there's a lot less headache until then, especially if management decreed the maintainers could share that work, rather than the other way around.

-- EdGrimm

While there are several OpenSource projects that have produced some very valuable stuff, there are reasons to believe that those successes may not have much bearing on commercial software development:

Open-source projects are generally staffed by unpaid volunteers.
Open-source projects rarely have financial constraints
Open-source projects rarely have schedules or deadlines to meet.
Open-source developers are generally not required to fulfill any requirements other than the ones they dream up themselves.
Most successful open-source projects are centered around one person who has vision and passion for the project. (How many open-source projects that have survived the loss of their creators?)
A successful project will always have a different organization from an unsuccessful project (lots of people working on it vs. nobody working on it), so it seems that degree of success influences the organization, rather than the other way around.
The "successful" open-source projects constitute a very, very small fraction of the total number of open-source projects.

I'm not knocking open-source; I'm just questioning whether the lessons learned about good open-source project organization can help improve closed-source software development.

-- KrisJohnson

The most *successful* OpenSource projects are mostly staffed by developers by companies who use the software, those companies do have requirements to fulfil, financial constraints and deadlines. Many open-source projects do commit to deadlines (e.g. Ubuntu and Gnome with their 6-monthly releases).

So can we conclude that closed-source projects have the following properties?

Are staffed by paid workers.
Have financial constraints.
Have schedules or deadlines to meet.
Fulfill requirements established by uncreative managers and customers.
Succeed by having no central leader and perfectly distributes workload across the team; most closed-source software survives their respective creators.
Driven by organization, which brings success.
Have greater success ratios. Successful closed-source projects constitute a large portion of projects ever started.

There is a bit of facetious tone, but I'm sure you, as the reader, know where this leads.