Metrics For Language Success
Metrics For Language SuccessFrom SmalltalkIsaFailure two important metrics seem to be emerging. Is it possible to use the power of Wiki to refine the definition of these and other metrics, not just rant or ramble? Probably not with me around. But let's see.
If I may I am just going to turn the order of this page around to see if helps. -- JasonAyers
If we are unclear on what we are measuring then our measurement will show us little.
What are the inputs to a project success or failure? In no order:
Do you think we could come up with metrics to test this? It would be very interesting to see which has the greater impact over a wide range of projects. Organizations invest huge amounts of money into building software and they spend that money based on received wisdom as to the impact of these factors.
I wonder what the result would be? I have no clue myself. I couldn't imagine how such a study would come out. I am sure such a study would be a big surprise to most of us, and contribute so much to the progress of software development. Maybe a Wiki reader who is much wiser than most of us could shed some light on the likely result?
Once we have gained some understanding as to the impact of these criteria then we could start creating measurements that might be composite measures of real value. Here are some first shots at those:
Design starts - The number of software development projects that start in a particular month or year that use or plan to use the language. This is a measure of popularity that interests vendors, since tools are usually purchased at the beginning of a project.
Project completions - the number of software development projects that complete in a particular month or year that have really used the language. This is also a measure of popularity, though one weighed by a useful measure of success. The emphasis is on completion of the project, not just the design.
These two are very helpful but they suggested two or three others to me as well. Let's get straight to the most pejorative:
Design tar pits - the number of software projects that are currently using (or at least maintaining code in) the language in a particular month or year, projects that would already have been completed or cancelled (depending on how extreme they are) if another language had been chosen.
I think this is giving choice of programming language too much importance in the probability a project has for successful completion. Programming languages (can) help, tools (can) help, methodologies (can) help, but in the end I think a well-functioning team is far more important than any of the above and such a team can deliver a robust and maintainable product in even an inimical language like Fortran IV (to avoid getting dragged into language wars) [What you meant to say is: this is not really a useful metric because ... the rest I agree with. In other words, strictly interpreted 'Design tar pit' would give a very low proportion of 'Design in Progress' and would in any case be much harder to measure (estimate). I was simplifying to make a point.]
This is obviously a subset of the easier to define and measure:
Design in progress - exercise for the reader
Note that the latter metric may well be the most significant from a vendor's point of view, if you can design the language, its environment, its relationship to evolving de facto standards and above all its upgrade licensing policy skillfully enough.
Average Production Years per Design Start - This could be graphed against any of the project success influencers. For the sake of the example let us say we are comparing different languages here. The metric is calculated as: Take all the Projects that make it to Project Completion and count the number of years that each of those systems has been in production in that language, then divide that total number by the number of Design Starts. If I am a consumer of a software language, I typically will consume a considerable amount of effort to get a Design into production to start with and lesser costs in subsequent years. The value to me, as a consumer, is to have a language that continues to meet my business (or whatever) needs throughout the years without forcing me to rewrite to another language (this value is both in terms of the one's software and the skills one's developers have acquired). So this metric counts the number of years a system has been successfully in production as a representation of its value to its user.
Given that younger languages are at a disadvantage with this metric, it is still interesting to ask which languages give better results here. Especially given Gartner is now saying 70% of all Java projects so far have failed (Gartner: Java and .Net - both initially a Disaster - 18.11.2002 um 10:05 Uhr - MUNICH (COMPUTERWOCHE)) and that .NET is predicted to have the same failure rate at least initially. -- JasonAyers
That says to me it's more about software projects generally than the choice of language. Unless Gartner have some more figures that say other languages have different failure rates
It is interesting that Gartner obviously thinks that it is the language/software delivery platform that make the difference here rather than any of the other criteria. They actually say that 40% of all projects fail. So that make Java considerably worse than the average.
I believe that ObjectShare did a survey of its customers in '98 and found around a 85% success rate on project starts but I don't have the data and anyway the criteria might well be different. I always thought we should go back and do that survey again, although the trouble is spotting all the projects that died a death.
-- JasonAyers
Money
An easier metric of success: whether the language and the platform (library, IDE, runtime etc.) made money for the entity that invested in it. Xerox didn't look like it benefited a lot from SmallTalk or any advanced research developed in its labs. Sun has been in the red for years now, in spite all the technical glamour about Java and J2EE, and investors in Sun lost money big time. On the opposite end of the spectrum Microsoft (Visual Basic, Visual C++) is doing quite well, Borland (TurboPascal, DelphiLanguage) is in the black; maybe there are other examples. Probably C should be considered the greatest success because most of the software that is run today was built on it.
One might say that this metric is irrelevant but the point is we shouldn't be talking about technical success without economical success anymore. A project that failed because of economic reasons should be considered implicitly failed because of technical reasons.
A more valuable Money criteria might be "have the companies using the technology made money out of using the technology" and the inverse of that, "how much have companies lost from using the technology both through failed projects or consequences of using it in production systems". Both are impossible to get statistics on, but I would think are more interesting metrics of success.
-- JasonAyers
For me the question of whether a language is a success is whether is is useful to me. As a hobbyist that means one thing (solving my problems), as a student another (passing exams), and as someone with bills to pay another again (getting paid employment). I suspect that this last role is the one most often meant here when people say that xxx is a failure. I'm not going to get a job programming Forth very easily, but it's been fantastically useful for teaching me a different way to look at problem solving as a contrast with the mainstream. Smalltalk is probably in the same class.
EditText of this page
(last edited November 19, 2013)
or FindPage with title or text search