Vcaps Project

The dates in this article are mostly pointless and unanchored like "now" and "years ago". It would be nice if this was changed so the article makes sense in the future as well.

The Vehicle Cost and Profit System (VCAPS) is being built at Ford Motor Company. The project has been going for about 5 years now [Specific dates, anyone? Ed.]. Up until recently the system was being built using typical WaterFall methodologies. It was fortunate that SmallTalk and GemStone were being used. It was also fortuitous that they were hiring experienced people. About one and a half years ago the project began to change in methodology. An underground effort to incorporate ExtremeProgramming practices was started.

The culture of the existing group had many ideas that were contrary to ExtremeProgramming. In particular the prevailing attitude was

It was surprising that the project had been running for 3 plus years and no one had seen fit to even create a single automated test. This was the first thing that needed to be changed. It started out slowly with one subsystem coded by creating the UnitTests first. This included the TestingFramework and some much needed test data. This small kernel blossomed into about 40% coverage of functionality over the course of about a year. Due to CodeOwnership the system was largely compartmentalized. UnitTests infected different areas like a virus spread by the programmers as they moved from task to task.

At year-end, management noticed a 30% reduction in bugs reported. At this point, management decided that UnitTests would be required. However, this was not set in stone. In one instance, a developer complained that a deadline would be missed if testing were required. In this case, special dispensation was granted. After releasing the code, about 32 developer hours were spent tracking down a problem that cost untold customer delays. It was discovered that the errant development team had based their work on obsolete versions. For our own sake, the UnitTests were undertaken after the problem was found. Adapting existing UnitTests to the new code took one hour.

Also at this time, management began support for ContinuousIntegration and CollectiveCodeOwnership. This was not necessarily altruistic on their part. It was in many ways a fall out of the turn over in personnel that occurred at about the same time. A separate workstation was allocated to the team to be used for SingleReleasePoint such that ContinuousIntegration and CollectiveCodeOwnership would work in this environment.

Now it was about a year into the underground movement and management had noticed a 40% drop in reported bugs. At this point we won support to start the system over and use even more ExtremeProgramming practices. It was obvious to the experienced team members that the cost of starting over would be less than trying to fix years of traditional methodology. Especially given a multi-gigabyte GemStone database with not one line of resident code.

There was significant resistance from management to such an extreme position. However, what we found was that all of the SeasonedSmalltalkers had been arguing separately for some time to rewrite. It was time to unite.

It was at this point then that a new project was created. The significant reduction in production bugs allowed us leverage to select a few more of the ExtremeProgramming practices. DeveloperOnlyXp was agreed upon minus PairProgramming, one of the more vocal developers refused to even try it. Our approved practices included:

At about 3 months we added PairProgramming. This was subtle and we learned a great deal. This was perhaps the most difficult aspect of ExtremeProgramming to incorporate. Programmers had to be shown that it not only created higher quality code, but also was faster (see FullStaffRedundancyWhileProgramming). We learned to always pair a believer with a non-believer; two non-believers will not actually pair together. The non-believers will not only remain non-believers, but will become adamant about it as well. We also discovered that learning to pair program was easier when the believer was more experienced in the technique. We realized the implication of this was that a new project would require at least one experienced XP person.

[This spawned the proto-pattern PairNonBelieversWithBelievers.]

In 6 months we were about 80% functionality complete. Four years of development were duplicated in only half a year. We are reminded of the proverb "everything you do today you can redo tomorrow in 15 minutes and it will be better."

We knew we were on the right track when a new requirement had to be applied to both the old system and the new system. This new requirement took 3 days to build into the new system (including the UnitTests) but required over 30 days to implement in the old system without tests.

Contacts: DonWells, TomKubit, JeanineDeGuzman, KevinBradtke

Even though we did not get dedicated customer time we were able to use CustomerShadowing. -- TomKubit

On the Phase 2 team, (using XP) there are 6 developers and 1 customer Project leader. On the Phase 1 system, (production support team, using traditional JangIt practices) there are 4 developers (frequently working OT) and 2 customer leaders. There are varying degrees of customer involvement, which is a difficulty we face.

The right team can accomplish goals beyond one's wildest dreams! These teams are also known as a high performance LearningOrganization; they elicit effective project management practices through team ownership of the plan, and they demand software development projects to be managed in two segments: a Project Deliverable Schedule (CommitmentSchedule) and an Engineering Schedule (EngineeringTask). The two are NOT the same; most project managers fail to let the engineering team work to the beat of the engineering plan, instead they assign work and force teams to use a project plan focused only on Customer Deliverables.

In fact, we all know this erroneous practice is contradictory to good Object thinking, let alone reusable frameworks, testing (UnitTest), RefactorMercilessly, ExtremeProgramming, and so on. Thus, we achieved our successes when we used ExtremeScheduleNegotiation. -- JeanineDeGuzman

Another important ExtremeProgramming practice we just started recently is MovingPeopleAround. There was agreement to do this from the beginning, but managers will be managers. -- DonWells

Another difference between ChryslerComprehensiveCompensation and us is the way we test. We actually have 3 types of tests. We have UnitTests, AcceptanceTests, and CustomerTests. The data in the VCAPS system has a very distinctive life cycle. Our AcceptanceTests run a small representative sample of data through the entire life cycle exercising the system as a whole end to end. The thing is that it runs in a minute or two. So we always run both UnitTests and AcceptanceTests together as a single test suite. CustomerTests are script driven and require a person to perform. These tests are conducted on a large database copied from production. These are also our parallel tests to be sure we match the old system. The CustomerTests are performed because of the history of this project, which until recently was driven by traditional programming methods. They don't trust us. We hope to change this. -- DonWells

Some other differences between C3 and us are that we still believe in ExtremeProgrammingCodeReviews. We also practice ExtremeProgrammingDesignReviews and have incorporated RefactoringIteration into our process.

We also deal with ExtremeRoles differently. We do not have a TheCoach for example. Instead, we see many functions or roles as hats on a hat rack for anyone to wear, rather than people pigeonholed into assigned roles. -- DonWells

It's wonderful to see you guys trying the XP practices. It's a particularly interesting experiment because it's [mostly?] developer-only [if I have that right].

Your comparisons with what you believe C3 to be doing may not be quite on: it appears that we've not been communicating perfectly what we're up to. For example, we don't really assign roles, or much of anything else. On the other hand, we use hats mostly as mock awards for things like digging too deeply into one's bag of tricks. ;->You are referring to the beanie with the propeller on top?

Please keep telling us your experiences and we'll try to relate them more accurately to what's really going on on C3.

I just realized we have said nothing about our problem domain.

What we are doing here is determining the production revenue, expense, net income and profit of a Ford vehicle. All the vehicles and all the models and options. The data is given to us once per quarter and once again per year. About 5 complete cycles per year. We also have to report to the government for AALA and NAFTA regulations. As input we receive the manufacturing bill of materials for all the vehicles. This includes data on every item in the car right down to sand for casting. We also receive plant level fixed costs and expenses. We receive information about detailed labor hours as well. A large volume of data from a variety of sources. We put it all together and voila everything you could possibly want to know about manufacturing an Escort... dollar wise.

Now the real fun begins. How does that compare with what it cost 3 months ago? Keeping in mind that if the bill of material structure changed but still calls out the same parts that is not a real difference. Also keep in mind that we also need to know what caused that difference, was it a design change, manufacturing change, etc.

What you get is a very good environment to apply ExtremeProgramming. Basically we are bean counting. The Problem is that the way we count the beans is very important and is dependent on exactly how the analysts are counting them...this time. Each go around will have a new set of rules and government regulations. The reports this system produces are not used for their own sake per se. They are used to make complex decisions about car and truck production, purchase orders, acquisitions and divestments.

Choosing a SystemMetaphor was difficult for us. We had data from a variety of sources that had no commonality it seemed. A breakthrough was when we discovered that all the data could be uniformly represented as if it were a BillOfMaterial?. So we chose BillOfMaterial? as our SystemMetaphor. In the original system it was often very difficult to find the right data. In the new system it was obvious where it would be. From the cost of operating capital all the way down to the cost of a single screw, our new model has the same metaphor. After we had gotten down the road with this new metaphor we measured that this uniformity was going to save us 95% in database size. We also now had one and only one way of accessing costs no mater if it was for a wiper blade or finding the average cost of a light truck. We went from processing times averaging about 15 minutes to coming back immediately.

The VcapsProject has been killed because yet another system will replace the VCAPS system we were building to replace the original VEPS system. In the end our customers were very happy with our work and still would like to see the Extreme version put into production. But from a purely political stand point that will not be possible.

We found it to be easy to mothball our work. We documented all the design decisions we had made, created an overview of how the system works, and documented how the ExtremeProgramming process worked for us. We also had a pair of developers do a final refactoring to make it ready for the deep freeze.

VCAPS started out as every programmer's nightmare but in the end it was a fun project and all of us learned a great deal about ExtremeProgramming by participating.

Was it killed for external reasons, or because the project wasn't delivering what the customer needed? Why will another system be needed to replace it?

It was killed for external reasons. It was deemed that the systems which feed it needed to be replaced, thus making our project obsolete before it was finished.

The way you word it it might seem a case of "the operation was a success, but the patient died". Did the results meet the business needs of the customer? If not, why not?

It was in fact a fabulous success from the customer's point of view. In fact they found a pot of their own money to finish the project even if it was destined to be replaced again in the future. They really wanted our software and it was heart breaking not to deliver it when we were almost done.

Is this a success story for XP, or a failure?

This is a success story for XP. The goal of XP is to have customers happy with the software, managers happy with the cost, and developers happy to be on the project. We had all 3.

VCAPS died with its boot on.

I think this goes to prove that most things you do are just part of a BiggerPicture?. -- PeterForeman

The VCAPS project is now in maintenance mode. There is one new thing to be added, but then it is going to be replaced. The remaining two team members told me the automated tests we wrote are still very useful for weeding out integration problems when modifying code or adding new code. But, they are not creating any new tests because they believe the AcceptanceTests we created are not very extendable and it will waste too much time to increase the coverage. We had used the UnitTest framework to run our AcceptanceTests instead of creating a domain specific interface to help create the tests. That is, we hand coded the acceptance tests and that made it laborious to maintain them.

Even so, everyone on the project who had used the test suite is now TestInfected, recommending automatic testing to the new projects they move to. -- DonWells

The VCAPS system was supposed to be replaced but wasn't. It is still in production after 10 years and working just fine.

So what's the verdict on the end of the project? Was it premature or not?

This project was undertaken at a car manufacturer that doesn't like to be named since a couple of its vehicles flipped over. There are huge amounts of waste at this company in the IT area. You often hear the phrase "we are not a software company" and yet when you look at the size of the (numerous) buildings dedicated to software you can't help but think "yes, you do way more software than a software company." Anyway, there seems to be some confusion because of this. VCAPS replaced VEPS. F@ST will replace VCAPS someday. We worked on VCAPS version 1 and version 2. Both versions were very successful. Version 1 was in production. Version 2 was not in production.

We consider version 1 a success for two reasons. First and foremost we were able to reduce the bugs from unbearable to "what bugs?" When I arrived there were a dozen people working long hours to keep the thing going. Now there is one person. Second, the system is still running just fine doing its job a decade later.

We consider version 2 a success because we built it while maintaining version 1 with out needing more money. It showed what was possible with good people and a good process. Many of the managers learned important lessons and the developers went on to be good managers who understand the importance of a good team with the right process. The unnamed auto company even now is putting F@ST into production due to the contributions of a VCAPS team member.

While it is true that the second version of VCAPS didn't go into production it is also true that the first version was made better by XP practices. We increased the quality of the system so much that we were able to keep up with any request the customers made on us. That was something new at this unnamed company.

Eventually VCAPS will be replaced, but meanwhile the quality and reliability made possible by XP is still there. --DonWells

CategoryStory CategoryAdoptingXp CategoryProject