Combining Open Source And Xp
Combining Open Source And XpThe idea is to develop a site similar to http://www.SourceForge.org or http://www.CoSource.com where, however, where the XP practices provide controlling of the OpenSource development approach. The problem for a potential OpenSource customer is that if they want to pay money to have some product developed by OpenSource developers then they also want a guarantee that the product will be completed on time and within budget and to the quality they require. However, money should not become the sole motivating factor, this risks turning an OpenSource project into a ClosedSource project.
CoSource? does this but doesn't have any means for the customer to check whether progress is being made (although they define a third party to judge when the project is completed). And this is where XP comes in: through UserStories, UnitTests and ContinuousIntegration the customer can always check progress. Plus after each 2-4 week iteration they can terminate the project and only pay for the work done to that point.
XP would not be enforced, can't anyway, but the intention is to offer tools which allow the customer and developers to a) communicate and share ideas, b) allow the customer to see whether progress is being made and c) both sides to check the quality. Tools should not be forced upon projects, however, customers would have the right to define which tools should be used for a project (after all they sponsor the projects). However, to a certain extent, a project should be given time to find it's own tools of choice.
Benefits for customers:
Refactoring: supported through tools such as JRefactory and a tool which produced statistics on how much needs refactoring (pipe dream?).
Unit and Acceptance Tests: xUnit and perhaps limited automatic generation from user stories - a new project role: User-Story-to-Acceptance-Test Translator!
UserStories: Wiki like tool for UserStory entry/tracking.
Continuous Integration: CVS + automatic UnitTest execution, i.e. check in fails if UnitTests don't run at 100%. Each project would also have two branches (at least) one release quality and one development. The only person who would be allowed to move code from development to release would be the head developer. Only code that executes UnitTests to 100% can be checked into the development branch, but all developers can check in to that branch. This comes from OS, where a single developer is responsible for maintaining the main codebase.
Iteration Planning: Something like Wiki to provide allow developers to enter their engineering tasks, and from there an integrated calendar application to handle the planning.
Onsite Customer: replaced by allowing the customer CVS access and access to user stories and acceptance tests.
BTW is it just me or are there a very limited number of tools supporting the XP development approach??? This gives me an idea: who would be interested in financially supporting or being a developer as part of an OS project to develop some XP tools?
Finally how can this all make money? That is, how do the developers get paid and when & how does the customer pay.
Charging for User Stories: I wonder if there is a way customers could be charged per User Story. As you suggested, someone would need to work with the customer to write the stories. I guess the focus would be more on growing existing software, than on making new software. Teams of developers could bid on individual User Stories, or on collections of them. This would provide an approximation of the Planning Game, since the developers would set the price for each story, then let the customer decide whether to implement it, break it into separate stories, etc.
But: I think charging the customer for User Stories will limit the number of user stories and therefore the description of the problem domain. A better strategy might be that customers only pay for those user stories that were implemented. This would imply that the customer would not pay at the end of the project, but after each iteration, which is probably a good thing.
Basically each user story becomes a mini-project with it's own developers, time schedule and price.
Payment for test success: Another strategy, although idealistic, would be to assume that all requirements can be recorded as automated tests. Given that assumption, the customers would write User Stories and tests. No money would change hands until a group of developers deliver software that passes all the tests. The first group to pass the tests wins. In this scenario, the customer would set the price. If the price was too high, nobody would work on it. So, if the feature was important for the customer, or urgently needed by the customer, they would naturally offer more money. I think www.CoSource?.com works something like this, except without the tests.
Payment based on rating: Also, it seems like some kind of code quality control would have to be enforced. Without this, a developer might put in unmaintainable code, just to get the next paycheck. Maybe some kind of developer rating system would help. Something like the SlashDot moderation system. If somebody makes bad code, their rating goes down, and customers would be able to see these ratings before committing to work.
Sure - my basic idea was to have a group of developers work on a project, after the project is completed, each developer rates all others based on their contributions and developers are paid accordingly. Of course, this has the problem that developers can "gang up" and award each other good ratings and thereby clean up. Therefore, you would need some sort of controlling for checking how many postings and check-ins each developer did. Or lines-of-code they wrote, or lines-of-doco ... etc.
Of course there are always problems:
Also, it seems like some kind of code quality control would have to be enforced. Without this, a developer might put in unmaintainable code, just to get the next paycheck. Maybe some kind of developer rating system would help. Something like the SlashDot moderation system. If somebody makes bad code, their rating goes down, and customers would be able to see these ratings before committing to work.
Refactoring could be another problem. If developers become too focused on delivering the next small chunk, then they might not want to make the investment to improve the internal code quality.
Interesting ideas!
Open source (I thought) usually doesn't have a paying customer? Much of the discussion above is about SponsoredOpenSource, i.e. where a customer IS paying, rather than discussing Xp with OpenSource per se.
Normally yes: an OpenSource project does not have a sponsor, but the intention is to give developers the possibility of earning money (admittedly not as much as a full time job) working their own hours and giving them a freedom of choice on what they work.
Also the intention is not to railroad traditional OpenSource, i.e. if developers have a choice of working on a project that pays and one that doesn't and both interest them equally, then unless the developer is an idealist, the traditional form suffers. I don't know how this could be prevented. -- GerritRiessen
There's also an assumption on this page that Open Source code is done collaboratively by individuals not in the same geographic location. That is a bit of an obstacle to Xp.
True, but my view of XP (at the risk of being flamed) is more of a set of best practices which (and here is comes the problem if you leave some out) support each other in very subtle ways. So the intention is provide tools which allow these tools to applied so that the customer is guaranteed quality and can see progress. -- GerritRiessen
Xp Friendly Open Source Projects?
It's difficult to see some of the most important Xp practices working in an environment where people can't pair program and the customer is not on site. Maybe it could work.
Pair programming can supported using SkyPe (although not open source) or something like kibitz. Of course it's difficult, but one should be able to provide tools for doing VirtualPairProgramming. Customer can be virtually on site by regularly checking the mailing lists, and UserStories - assumption being that a tool used to maintain UserStories, and their associated tasks, online. They would be able to checkout all the sources and UnitTests to ensure that progress is being made.
In the end, I don't know how many of the XP practices will be supported, but the key is to provide controlling mechanisms to the traditional OpenSource process. One way I see this happening is by using parts of XP, I can't predict what will happen but I think it could work :-) -- GerritRiessen
Two suggested projects which might be suitable for Open Source with Xp are:
NovelCompilerArchitecture If anyone is interested some slightly novel compiler architecture ideas are sketched in RethinkingCompilerDesign. Here elements of the TestFramework? would reappear as sample code delivered to the customer and sample code in the IDE. Heck - compilers even compile themselves so that's part of the test suite too. Of course we'd add torture tests too. I have some very specific ideas as to what OnceAndOnlyOnce means in a compiler - and it's a bit more extreme than most compilers are. The need for an on-site customer is possibly slightly reduced in writing a compiler, since anyone doing so is a programmer and should have insight into what 'customers' expect :), i.e. less contact with the real end-customer is required.
Am I subverting things by suggesting that parts of the test code become part of the deliverable? Not at all. OnceAndOnlyOnce rules!
-- JamesCrook
This SourceCast? tool looks like it might be useful for this kind of development: http://www.collab.net/products/sourcecast/
The ProgramFactory? looks like it uses a similar concept as well: http://www.ProgramFactory.com
Or along the lines of the SourceForge interface with extensions, but definitely a web-based tool.
TheMozillaProject has ContinuousIntegration on a grand heterogeneous scale. They have a MozillaTinderbox with Linux, Mac, Win32, SunOS all building continuously over and over. -- StevenNewton
See also http://www.advogato.org/article/202.html by AndrewMcKinlay
Moved here from OpenSourceExtremeProgramming
Wouldn't it be great to have an Open Sourced project made with ExtremeProgramming? After some thought this is the main problem:
Other than that, it might work very nicely! Assuming CVS is used UnitTests could be easy to get - cvs checkout UnitTests - and all code submissions would need accompanying UnitTests. Instead of pair programming we could have many-eyed peer review by using a mailing list for patch/code submissions (at least from new contributors). This would also help with reducing redundant code. With many people reviewing most of the code, chances are most redundancies would be caught. What does anyone else think? -- NathanHeagy
- It seems, in some ways, similar to ExtremeHacking.
NathanHeagy and I discussed this a bit more elsewhere, we came up with the idea of a CVS and Wiki combination where random people could drop by and change the code as long as the changes improve the 'score' of the predefined UnitTests.
Seems to me that the project 'leader' would be the person who defines the UnitTests and at least starts the basic code bits.
This idea also seems similar to some of FridemarPache's enhanced Wiki thoughts. I've always liked those ideas, but could never come up with a workable way to implement them. An OpenSourceWiki? sounds like it might decrease the 'threshhold' enough to get random people to change code. I think I would do it at least.
Does anyone have thoughts or ideas about this? I have thought of the security problem, not sure what to do about it though.
Yes, open source development has a relatively high barrier to entry. You usually need to join voluminous mailing list, understand a culture, agree to the group's coding standards, and work in concert with others. This is difficult. But perhaps it is a good thing.
A high barrier to entry? Only on very large projects. Most open source projects have only a handful of coders and a fairly small codebase. Quite often you can grok the design easily enough to add the minor feature you need to add without disturbing much of the rest of the code. However, this is just my experience.
Check out http://www.smalltalksource.org - it's a wiki for open source development.
Given an open project of sufficient size, the probability of someone deliberately contributing an obscure bug (or HeisenBug) clever enough to thwart existing UnitTests approaches one, and is followed by the probability that the bug also introduces a security flaw. Human review of untrusted patches, while also not perfect, is still necessary. -- CharlesMiller
Note that many projects use the XP principles outlined above. Patches are sent to a mailing list by contributors. The maintainer reviews the patches and commits; for large patches the list members will chime in with feedback before the patch is committed. Continuous integration (or as close as you can get) is achieved in this same way. Mozilla has even created tools (MozillaTinderbox) for continuously building and testing the product (including fingering the last person who committed in case of a test breakage). Successful projects are the epitome of the 'simplest thing', failed projects are those that overdesign in the early stages.
The more I think about it, the more I think that OpenSourceExtremeProgramming will have slightly different rules than ExtremeProgramming. The idea is still that you have to take all the best practices of open source programming and do them all the time, the result is different in that the best practices from open source are slightly different (but overlapping) than conventional programming:
Isn't OpenSource already ExtremeProgramming but in a different setting ? A lot of the practices and characteristics of OpenSource are reflected in XP and vice versa. Here I'm thinking of testing, feedback from users, pair programming -- where IMHO OpenSource practices long distance pair programming, high level of communication, high discipline, refactoring, collective code ownership ....
However, XP works best with a customer, a single group of developers all at the same location, and a tight schedule. OpenSource is best suited to a distribute group of developers, no direct customer, and no definite time schedule.
Having advocated the combination of the two, or rather the notion of CorporateOpenSource, I find myself moving away from that notion to one of never the twain shall meet! I.e. One can see parallels, but the combination would bring too much commonality that a mixture would not bring any improvements.
Perhaps, wearing my corporate hat, an interesting topic would be the development of a CorporateOpenSource methodology that borrows from OpenSource but allows a multinational corporation, and there are plenty of those, to get full benefit of its developer resources. I.e. Corporation X wants to develop product Y. What does it do? It assigns a project team in location Z and gives them a deadline of XX days. Everybody on that team works 60 hour weeks to get the thing done, and then it only works in one language! Instead it could have assigned project teams in locations A,B,C perhaps ensuring that timezone distribution was such that no overlap occurred in development time of the individual teams. The corporation could then enjoy a development time of 24 hours per day and developers could enjoy a healthy working week of 40 hours. Plus the end product would be available in three different languages - product and manuals.
Flea was wrought as eXtremely as possible with only one team member:
http://flea.sourceforge.net-- PhlIp
The OpenSource projects I've found most interesting are those that were originally developed by corporate IT departments and then OpenSourced for all the expected benefits - more review, more testing, longer life. Or they are projects that were developed by people working in corporate IT departments collaborating together for the same reasons.
(For example, much of the Apache JakartaProject, TeaServlet?...)
The idea is that these are frameworks and toolkits that were developed in the course of developing applications or subsystems, rather than being complete packaged applications or subsystems themselves. That is, someone was getting paid to create a (ClosedSource) application or subsystem, and in the process found themselves building a framework or toolkit that they realized could just as well be spun off into its own package for the sake of reuse.
This suggests a different view than was discussed above. Most of the discussion here has been assuming that the goal of the OpenSource project will be to build an application, which some customer will pay for. Since ExtremeProgramming is about building applications, it then is tempting to wonder how OpenSource applications can be built with ExtremeProgramming practices.
But the alternative view is that it makes sense to keep application development ClosedSource, using all the ExtremeProgramming practices as usual, while any frameworks or toolkits created in support of application development could be released as OpenSource.
So the attitude would be: Hey, in the course of building this application we created some packages that don't depend on the particular business problem we're solving for our customer, and which are probably really similar to what other people have written and are going to write over and over again. Since we don't like the idea of duplicated effort, let's release these packages as OpenSource (or let's look around to see if someone else has already OpenSourced something similar that we could use).
Further, if this attitude really does enhance our productivity (by reducing duplicated effort, providing external testing and review of code, etc.), then we should look into how our ExtremeProgramming practices can maximize this kind of reuse (as I describe briefly in ConsideringOpenSource). Specifically, RefactorMercilessly can be applied, reducing the dependence of our framework and toolkit code on our core business solution code, which will:
(If they refactored but didn't branch, it would cause problems for the other teams; the only alternative to branching would be to not RefactorMercilessly, which would mean that embracing OpenSource would interfere with ExtremeProgramming instead of enhancing it. The beauty of OpenSource is that each team can use the OpenSource package as long as it is helpful, and then take the code in whatever direction they want if their needs change. So this point about being able to refactor into the OpenSource package is really the key advantage that using an OpenSource package has over using a ClosedSource package with the same functionality.)
This approach does have some implications for how the OpenSource projects are maintained, but nothing really beyond how OpenSource projects are already being maintained. For example, there must be UnitTests - but many of the ApacheJakarta? projects are already using JUnit. There must also be CodingStandards, ContinuousIntegration, and FrequentReleases, but those are nothing new for OpenSource projects. UserStories and OnSiteCustomer are irrelevant because they are handled by the application developers outside the scope of the OpenSource projects. PairProgramming comes for free since all the developers are working in ExtremeProgramming teams already.
Efforts to refactor ApacheAxis raised some interesting issues of CombiningOpenSourceAndXp. After identifying the subsystems, we started by separating out a configuration subsystem. Some of the committers understood YouArentGonnaNeedIt, but one or two argued for keeping complex but currently unused features for the sake of future flexibility. However, refactoring was helped by the Axis build including both unit and functional tests.
The distributed nature of open source communities makes it more important, IMHO, to document the current design so that newcomers can more easily find out why the software is structured the way it is.
EditText of this page
(last edited June 28, 2008)
or FindPage with title or text search