The Rad Bottleneck

RapidApplicationDevelopment (RAD) tools for the typical CrudScreen parts of applications have always had the same problem: They made the first 80% easier, but the last 20% a bear (EightyTwentyRule). To get the fine-tuning a customer wants, either you spend hours and hours trying to add or force a feature that the tool was not designed for, or you have to bust into generated code (if it makes any) and learn their confusing conventions and frameworks in order to figure out how to tweak it. The result often takes longer than what would have happened if a more lower-level tool was used in the first place.

Nobody has found a way around the 80/20 problem after years and years of attempts. Either there is a solution that nobody has discovered, or there is some fundamental law of the universe that prevents it, such as the shear complexity of the interaction of all possible CRUD options together in one grand framework.

Except for frameworks that deliver 80% of the solution, with wall-to-wall DeveloperTests...

80% usually doesn't cut it, unless it's a small shop with illegal labor that accepts a compromise because they can throw extra labor at the deficiencies.

There used to be a very similar problem with HTML/Web site generation and WYSIWYG editing tools. But once editing tools starting treating the HTML like XML (that is, as a validated syntax tree), they could support "round-tripping" (editing the HTML outside the tool and then bringing it back in without losing the information).

I wonder if the solution there is applicable to RAD tools. I.e., if the RAD tools could be made to not only generate code, but properly parse, incorporate, and use code created and/or edited outside the tool, instead of seing themselves as the only piece of the puzzle.

-- MattChaput

Some say the best way to design HTML documents for the long-term is still hand-coding. I have not made up my mind on that issue yet. Anyhow, HTML is not a TuringComplete language. I wonder if the issues change if TuringComplete is added?

: Define HTML document. I prefer to hand code nearly everything; however, HTML is not sufficiently expressive for me to OnceAndOnlyOnce to the extent I like. This is simplified by using other languages in addition to HTML, and in my opinion would still could as a hand-coded site (assuming a text-ish editor is used for the other languages as well). Given this, it is surely better for the long term, at least until we have some practical WhatYouSeeIsWhatYouGet methods for representing and working with code. --WilliamUnderwood

I find it difficult, for me, to see the day that I will not still have to hand code at least some portions. To date, I find the most power and speed in hand coding. Automation is good, but there are those delicate areas where the good old human brain is hard to replace, and I see that as a good thing rather then bad. Also, I find that hand coded work stands out from the rest, as all starts to look and react the same without a little customization, creativeness, and other extra innovations thrown in (CookieCutter? is boring).
HelpersInsteadOfWrappers is one approach I've found that is a pretty good compromise. One has automation "toolettes" for common idioms, but does not have to "buy in" to a framework that may have limits over control. It's lots of small abstractions instead of one master abstraction. HelpersInsteadOfWrappers is slower than typical RAD tools, but also more tune-able. Maybe some kind of code generation using helpers is a way to achieve RAD. CodeGenerationIsaDesignSmell by some accounts, but perhaps should not be entirely dismissed if used right. One must still know the conventions used in order to modify the code, however. True RAD may thus require heavy familiarity with a the tool or conventions used. RAD as a shortcut to learning a tool/system/conventions is perhaps not realistic.

The new VisualClassEditor? for JavaSwing available for EclipseIde 3.0 does full round trip. Two windows, one graphical, one the usual code editor. You can edit in both and the change will be reflected in the other. Dragging in a Swing panel will create methods and addition code, moving panels will "refeactor" the code immediately. Adding methods with panels and e.g. adding layout constraints will update the graphical part.

Problem: not yet quite stable, some structures cannot be parsed fully and will be rendered as "?".

But I bet, that this has aleady been done in SmalltalkLanguage long ago. Therefore my question is: Is TheRadBottleneck real or a consequence of the tools not keeping pace with the languages (in this case java taking years to reach the tools maturity level of smalltalk)?

-- GunnarZarncke

I disagree that all RAD tools have TheRadBottleneck problem. Two-way tools (AKA round-trip engineering) are out there, where the visual tools and code editor (text based views) are kept in sync. Custom-generated boilerplate code always has the 80/20 problem, but Delphi, Smalltalk, and DotNet developers enjoy a level of productivity that other developers would find compelling enough to learn the new tools if they realized it. People who have wasted hours waiting for their C++ apps to rebuild, and people who subscribe to the convenient view that TheRadBottleneck is the reason they are using one crappy tool (Visual C++) instead of another crappy tool (Visual Basic) are forgetting that there is more to the world than C++ or Visual Basic. Open your mind and try PythonLanguage, SmalltalkLanguage, and DelphiLanguage.

I also think that the more dynamic (e.g., Python, Smalltalk, LISP) and less static (C, C++, Delphi) your language, the less time from code edit to runtime. This can really contribute to getting making that "last bit" easier. Yes, you still need to author hand-written code; RAD tools don't free you from that, but they do free you from a huge amount of drudgery. All of that drudgery you've saved makes your application better or gets you done in less time. There is not always a "down side" to this particular "up".

-- WarrenPostma

I don't see a significant difference between VB and Delphi. Delphi uses a more consistent language, which can reduce the number of small irritating problems a bit, but that alone does not separate it. Plus, VB has some dynamic and scriptish abilities that Delphi lacked.

I agree that using scriptish languages to make LittleLanguages that custom fit the domain/project/company-style is a very good RAD technique. However, that is usually not what most think of when "RAD tools" are mentioned. Plus, it often takes a few tries to perfect the LittleLanguage in my experience because one has to learn the ins and outs of the environment and where to give and take. To get KISS right, you have to know what to skip also. --top

Key handling is one of the toughest things about CrudScreen frameworks. Some keys are auto-generated some are compound keys, and each with potentially different characteristics such that some parts are user-editable but not others, or editable during add but not update, or not shown (such as auto-gen), etc. etc. I've been trying to find simple patterns, but so far its elusive. Surrogate keys can greatly simplify things, but making a framework assume that only those kinds of keys exist is limiting when one has little control over existing databases. Questions to ask when building such a framework:

Is the key compound?
If a compound key, does each part have different characteristics (below)?
Is the key (or portion) auto-generated?
Is the key visible to the user?
Is the key read-only?
- Read-only during update, but editable during add.
- Some parts of the key have different editable statuses per stage.
- Visibility (hidden status) may also be divisible per key component and stage.
How does the user change a primary key, or must they delete first and then rekey?
Will there be a "Rename" option if they want to change all or part of a primary key?

Here's an example compound key handling matrix per edit-screen stage/mode:

  Key-Part..|.Read.|.Add.|.Update.|.(Comment)
  ------------------------------------------
  sub-key-1.|...R..|..R..|..R.....|.Generated thru a different process/screen  
  sub-key-2.|...R..|..E..|..R.....|
  sub-key-3.|...R..|..E..|..H.....|
  Etc...

  H = Hidden
  R = Read-only
  E = Editable

(Dots to prevent TabMunging)

If one can have or assume a single key, such as a surrogate key, for all records, it can greatly simplify things (LiberatingConstraint). Someone told me that RubyOnRails's (alleged) simplicity is partly based on this assumption; but in the real world, one often cannot pick the database design. The DBA may also decide a surrogate key would impact performance. (R-on-R can handle compound keys, but loses its simplicity.)

--top

Managing "conditional" fields can also get tricky. Some fields may be relevant in some circumstances but not in others. For example, when filling out a medical form, the screen present different options/questions depending on the subject's gender. It's sometimes easier if you are allowed to have two different screens for each gender, but the customer may not want separate screens.

A "tester case" for such screens is to fill in invalid info related to Situation A, then change the check-box or options to create Situation B, press Save and see if the now "hidden" invalid fields are flagged for problems.

Generally you want the hidden items to "come back" if you (user) change your mind before hitting Save, such as changing the gender back. However, we usually don't want the hidden info to be part of validation, and often don't want it getting into the database either. Thus, we need a "soft persistence" treatment. -t

Re: "Nobody has found a way around the 80/20 problem after years and years of attempts."

Or maybe the problem is that the tools do not really cover the 80%, most I know do not really implement a FunctionallyCompleteCrudGenerator, they only give (a fake) impression of covering the 80%, but they actually do not even deal with a much smaller percentage of what CRUD really is (most can not even deal with the IrreducibleComplexityEntityModel)

I suspect that such a tool would be so complicated as to make hand-built code look competitive in comparison. The potential interface complexity required to handle all the "worse-case" situations could be pretty ugly. It's not just about feature counts, but how they interact and how issues or features can be hidden when they need to be hidden.

When I look at trying to make generic frameworks for CRUD, WebContentManagementSystem?s, and eCommerce (among other things), conceptually the features are pretty simple to comprehend by themselves such that the general problem appears simple to the uninitiated. No one feature is rocket-science in any way by itself (well, maybe taxes and coupons and discounts, but let's set those aside for now). But to get all those features to work together "properly" in all combinations of their presence (such as "on" or "off") is a combinatorial mess in terms of both design and testing. Often there are two or more feature sets that when switched on together create potential conflicts or a user interface that is too complex for normal users to grok without a complete design overhaul or a new "special" layer to handle relatively uncommon feature combos. Some of these we can anticipate at the design stage and some of them we can't. For those we can anticipate, often we have to make sacrifices, such as forbidding certain combinations, which will alienate or anger some customers as they grow or change. -t

By "we", do you really mean "I"? In other words, are you commenting on a general truth or your own limitations?

Am I being baited here? If you can build a Grand Generic RAD machine without trade-offs, you can be a billionaire. Go for it!

Not at all. I think it's a reasonable question. Try replacing "we" with "I" in the above and see -- honestly -- if it makes sense. If it does, it is quite possible that you are commenting only on yourself. This is no bad thing, not at all -- such reflections can lead to self-improvement. As for me, I did build CRUD/RAD systems (among other things) and my career has been very successful as a result.

I often mean "the industry". I will try to be more mindful of my use of "we" in the future.

Good, but my question still holds. Try replacing "the industry" with "I" or "me".

Not today. Note that I've also built semi-generic CRUD tools, but they tend to only fit the "personality" of a given shop's style and needs. General CRUD-RAD is still elusive, especially when dealing with legacy databases with historical baggage.

See RapidApplicationDevelopment, FunctionallyCompleteCrudGenerator, IrreducibleComplexityEntityModel, CrudPatterns, SufficientlyFlexibleAppsResembleInterpreters