Data Dictionary Based Rad

I've always been fascinated by the idea of using some form of DataDictionary (DD) for RapidApplicationDevelopment. This may include some form of EntityRelationshipDiagram-like info in table form, which is essentially a DD about table relationships. Perhaps "attribute-driven RAD" may be a better name for it.

In the ideal case, you'd only have to fill in DD tables (perhaps with the help of a mousey IDE) and most of the CrudScreen portion of the app would more or less write itself. I've experimented with the idea and used tools that kind of use the idea. The problem is that there are a lot of potential gotcha's, such as the EightyTwentyRule.

[more to come...]

--top

Possible Features:

One rarely has to textually type a database column name or table. (Even in a code edit window, an insert or auto-complete list would appear when requested.)
Based on some form of SetTheory or DeltaIsolation instead of copy-and-paste for CRUD objects that use slightly different DD specifications.
Covers common CrudScreen features: Add, change, delete, query (QueryByExample), and reporting.
EventDrivenProgramming used to imperatively override or alter behavior when needed.
Forms are by default grid-based rather than "draw-a-form". This is comparable to a modernized and smarter ExBase "EDIT" command (which cleverly used the same column spec format as the BROWSE command).
Data portion can be implemented in most RDBMS.
OpenSource

Who Makes It?

What's presented here is an interesting idea. I hope Top implements it so we can evaluate its merits.

Make me do all the work? :-)
You don't expect me to do your work, do you? ;-)
What about that AI project of yours? Teach it to learn. Maybe when a good open-source editable web grid widget comes along, I'll whip up a demo.
A good open-source editable web grid widget is an interesting idea. I hope Top implements it so we can evaluate its merits.
I gotta make tools in order to make tools in order to make a dream tool? Shit! Offshore outsourcing is starting to look good. Know any starving genius coders in Timbuktu? Screw principles. Microsoft does.
"I gotta make tools in order to make tools in order to make a dream tool?" Hell, yes. It's called "software development". You might want to look into it.
Yeah, but I shouldn't have to reinvent wheels such as data grid widgets. These should almost be commodities about now. I want to focus on data dictionaries, not on building scroll-bars. Dammit, IwantaPony! (I have looked into the TK kit a bit.)
Give the software industry another fifty to a hundred years and you won't have to reinvent every wheel. It took the automotive industry that long to semi-standardise and off-the-shelf a few parts, like fasteners, bearings, some transmissions, tires, spark plugs, fuel injectors, and a bunch of electrical bits. Of course, most of the rest is custom-designed. Unfortunately, the software industry is still at the horse-and-buggy stage, and we've barely sorted out how to attach the horse. For now, expect to design and manufacture some wheels. A lot of wheels, maybe...
- 50 years? By then I'll either be worm food or a brain-scanned emulation ("topbot"). Plus, I don't expect GUI's as we know them to still be the favored interface by then. What will replace it, I don't know (direct interface implants?), but history suggests it will change.

Sounds a bit like NakedObjects.

Objects in general are difficult to search, sort, query, etc. in a way that I expect meta-data to be. One is generally stuck viewing them in the way the original coded forced on you.

Reflection can overcome this to a large degree, and/or you can use the RelationalModel to manage collections of instances.

In my opinion, reflection is an instance of GreencoddsTenthRuleOfProgramming. Don't tack on a half-ass database after the fact, start with a database to begin with so that the design is cleaner and more standard. And not app-language-specific.

I have not found a good interface convention for specifying sub-sets and super-sets of columns without copying. If we keep copying, then we end up with the kinds of messes that MS-Access tends to result in.

Some columns will be applicable for some situations but not others. We might have the same column but define it different in one context versus another. What is the primary key used for reference in such a case? Maybe have a two-column primary key with an optional "alternate" number. But we do want to use native RDBMS column names most of the time. In other words, we don't want a secondary set of reference keys that are the same as RDBMS keys in most cases.

For example, using letters as stand-ins for column names, suppose we have this situation:

Report Q: A, B, D, F, J
Report R: A, D, E, F-alt, J

They both use similar columns, but not the exact same set. In addition, report R has a different definition for "F" (F-alt) than report Q. Suppose, for instance, that report R uses a compact date format whereas report Q uses a full format. If we use the actual source data table column name as part of the primary data dictionary (DD) key, then such "alt" creations are tricky because it violates uniqueness without gimmicks.

A screen example of an "alt" would be one screen that has tighter validation than another. For example, a supervisor screen may be able to put in special characters or codes that a rank-and-file employee cannot. In some cases we could use EventDrivenProgramming ("on-validate" event) to provide such behavior imperatively, but I'm not sure if that will cover every situation well. I haven't tested enough "generic" data dictionaries to say if imperative addition/subtraction of DD-specified features is sufficient.

Or we may want to specify an alternative display order, but without having to update the second list every time a new column is added to the primary list. If auto-referencing is not possible, then at least have some way to warn the developer if an alternative list needs updating. Thus, some kind of set comparing operation, along with non-pairs to ignore, needs to be addressed. Relational triggers of some kind may come in handy for this along with an easy way to have the set difference moved into the second list to avoid manual typing of column names.

For example, for the input screen "A" we want one field ordering sequence. However, for report "B" we want a different ordering even though the entity is the same (same data source table). One approach is to have 2 columns in our field dictionary, such as "sort_A" and "sort_B". However, this limits us to 2 sorts and is generally considered poor design. An alternative is create a second table with only ordering info.

table: field_dict // regular field dictionary ----------------- entity_Ref fld_id fld_type fld_order // default ordering (type "float") ...etc...

table: alt_sort --------------- entity_Ref // may not be necessary, per pending primary key issue sort_id // for example "B" for report B. fld_id fld_order

There's something ugly about this approach. It's kind of a violation of OnceAndOnlyOnce such that we have to remember to update 2 dictionaries (one primary and one dedicated mini-purpose dictionary). A trigger mechanism can be added such that anytime the primary dictionary is updated, the designer is warned about the related info in the "alt_sort" table. We could also add auto-replication with any new column placed at the end of the sort sequence. But, auto-replication is a poor substitute for clean OnceAndOnlyOnce. I'm stumped for how to fix this.

I've faced similar situations for field titles. I used to have a "short" and "long" title (and also a "help" text which may be a paragraph or so), but even this is sometimes not sufficient. Perhaps something similar to the "customAttributes" table in NonOopGuiMethodologies can be used for "extra" info.

I've been developing database-driven business applications day in and day out since the mid-1980s, yet aside from recognising words and phrases like "columns", "RDBMS", and "primary key", this section is incomprehensible to me. What, for example, is a "primary list" and an "alternative list"? What are "non-pairs"?

Is there a way this section might be explained more clearly?

You are right. I've started to add examples.

Have you compared -- in terms of complexity, maintainability, flexibility, etc. -- your table-driven approach above (which appears to be fine-grainted meta-modelling of the entire UI structure) with the equivalent written as, say, hard-coded SQL or even SQL in tables?

There would be some overlap with NonOopGuiMethodologies, but in general this topic focuses on RAD tools, not open-ended GUI's. The emphasis is on meta-data about data to drive most of the app, not fancy GUI's. It may have a rather bland GUI in fact with little room for shuffling stuff around. (However, a way to "escape" to naked GUI mode should ideally be provided in case we want to get tweaky.)

I asked because your mention of having reports that differ only in sort order, or column selection, etc., strike me as something that can be effectively written in minimal code -- whether SQL (in the case of sort order), or in a combination of native code and SQL in the case of column selections -- but typically involve a massive metadata schema to model in tables. I've been down this route, and it ends in vast schema complexity that can easily be vanished into a few lines of code. One reason programming languages exist is because implementing the machine-states they represent is easier to define in a high-level language than manually twiddling the underlying data structures. Or is it your intent that DataDictionaryBasedRad would be front-ended by some high level language, and the metadata schema would strictly be an internal structure?

I agree a SELECT list is pretty compact. However, they are not very easy to shift around and add to as text. Some kind of drag-able order list(s) is easier for that. But you are correct in that internal representation does not have to dictate user interface (IDE). However, do note that the issue of OnceAndOnlyOnce still remains even with a text SELECT list. If we have 5 SELECT lists and add a new field, somehow most or all 5 lists usually have to be updated. --top
The problem with SELECT owes to the fact that column lists are hard coded in most database query languages, like SqlLanguage and TutorialDee. This appears to derive from the RelationalAlgebra's definition of the PROJECT operator, in which column lists are static. This would be less of a problem if column lists could be expressed as first-class values -- thus allowing a column list to be defined once and used many times, or defined dynamically, perhaps with the bonus feature of supporting algebraic operations (UNION, MINUS, etc.) to be applied to column list values in order to generate new column lists. Under ExtendedSetTheory, PROJECT is a binary operator that accepts two extended sets, one is the set being projected and the other describes the "column" list (effectively -- this is more analogy than accuracy, but the end result is the same.) This provides the flexibility, re-use and maintainability you seek. In the RelationalModel, it would imply a new kind of PROJECT that accepts two relations -- one is the relation being projected and the other describes the column list. In practical terms, this would require -- as a starting point -- a new query language with a clean and simple syntax for defining anonymous temporary relations. Of course, this approach presumes that the consumer of a query result (such as a form, process, or report) can adapt to whatever column list is being used.
- Or implement TopsQueryLanguage, in which column "lists" come actually or virtually from a "column table" such that one can perform relational operations on column lists just like any other table.
- Indeed. That is an interesting idea. I hope Top implements it so we can evaluate its merits. It's easy: download the RelProject, add a SMEQL grammar and other relevant bits, rinse, repeat... Have you written a formal grammar for SMEQL?
- I'll consider it, but one dream project at a time.

But I am not sure what you mean by the hard-coded part. Whether the DD's are filled in with IDE's or hand-coded INSERTS is a personal choice. An open system would allow both approaches. That's an advantage of using off-the-shelf RDBMS (although I'd prefer DynamicRelational if appointed God of IT).