Greencodds Tenth Rule Of Programming

A database-oriented twist on GreenspunsTenthRuleOfProgramming in which, "Every sufficiently complex application/language/tool will either have to use a database or reinvent one the hard way." I don't know who originally said it, so I combined DrCodd's name with Greenspuns'. Ironically, even Lisp, the original "greenspun" language, reinvents one by having an internal complex data structure (nested list in byte-code form) that takes on database-like characteristics. -- top

What the heck is a "nested list in byte-code form"? I'm an experienced Lisp programmer, and I don't know what you're talking about.

The giant internal tree(s) that Lisp code compiles into. See HiddenDatabaseSyndrome.

Oh shit... Forget I asked.

I'm sure I'm going to regret commenting, but...kinda like the giant internal trees that every major database compile into? -- JonathanTang

It's an implementation detail, not a software application organization issue. The implementation could be gerbils on treadmills. If Oracle replaced the index engines with superfast gerbils on treadmills, nobody would (should?) know the difference. At the root level everything is just a list of 1's and 0's. But that does not mean that humans should work at that level. -- Top

So is the "giant internal tree(s)" that Lisp compiles into. Key word here is "compiles into"; anything being compiled into is an implementation detail.

No. nested lists is Lisp's native *interface*. Everything is conceptually nested lists. The implementation is all the RAM pointers that are never exposed.
The Lisp engine is essentially a half-ass hierarchical database of sorts, so it's already Green-Codding in its own way. I happen to prefer tables over nested lists in most cases as the "atomic structure", though. It's a WetWare preference issue. -t

These are perhaps syntax trees? Related perhaps to the notion that they model parsing Rules? I pray 't' please write a parser using a database please please please. Then explain to everyone here how extend the language using the language... Please.... Please Please... --- I am now sure that this is Joke page

I don't see what a parser has to do with the topic, but actually I've considered it multiple times for cases where I don't have time for a perfect parser, but a quick way to make something say 98% accurate with warnings for rough spots. A probabilistic pattern-based table-driven parser of sorts. It would beat hand translations from different app languages and data formats. --top [please don't remove this again]
It's looks like parsers were brought up as because of a guess as to what you meant by "giant internal trees". And it's quite clear from this response that you still haven't tried to implement a parser using a database, so I'm still not sure this should stay. [And, I'll remove whatever you post that is clearly nonsense. For example, talking about how you don't see what parsers have to do with a topic when parsers aren't being discussed at that point].
If somebody did implement a parser using mostly a database, would it change anything?

You could quibble with the interface that Lisp presents to manipulate its code. There've been various attempts to improve it, the most recent being DylanLanguage and GooLanguage. But the fact remains that a large number of programmers find it convenient, enough that Lisp is the only language where code-that-writes-code is an inherent part of the programming culture. It's much more convenient than any table-based code representation I've seen. -- JonathanTang

Are you confident enough in those claims to take the ChallengeSixVersusFpDiscussion challenge? I suspect the frequency of utility is domain-specific at this point.

Top, it may be the case that ChallengeSixVersusFpDiscussion is a reasonable challenge; maybe, maybe not, but it seems to me that it is asking people for a lot of work, whereas mostly people drop by c2 to talk, not to work. That's just a btw.

It would not be "a lot of work" if FP greatly simplified development, as some have suggested. I picked that example because of complaints about toy examples not being thorough enough.

Maybe, maybe not. FP might greatly simplify, but not simplify enough to make your challenge trivial to implement, for instance. I wasn't participating in the FP debate with you, so that's just a passing observation. -- Doug
Then select another biz-oriented example that demonstrates the power of Lisp. I am tired of "lab toy" examples. The real world is not like programming contest examples or textbook examples. They might be fun, but they are not representative. It is kind of like an NBA slam-dunk contest: fun to watch, but does not necessarily select out the best actual players.

More to the point, would you please stop assuming you know anything whatsoever about how Lisp implements anything at all. Seriously, you have completely the wrong idea, clearly, from everything I've seen you write on the topic over a very long period of time, but you seem very fixated on the notion that you do have the right idea. You don't, really, and I'm not being insulting, just objective.

I honestly don't think some of you understand what "objective" really means.

Just don't keep assuming you do know, please? It doesn't help your arguments. Clearly, everything is implemented as bits, but beyond that, no, no, really, no. I have implemented Lisp and other languages, that sort of thing is one of my specialities, and it pains me to see huge inaccuracy on the topic. Nor is it necessary to your favorite topics, anyway. -- DougMerritt

Yes or no, internally Lisp has a "complex data structure"? And, to the Lisp programmer, that internal structure more or less resembles nested lists, IOW trees of lists? Correct or not in a general sense? Also, some of the comments above that appear to be mine are probably not mine. At least I don't remember writing them. In fact, I suspect some stuff has been removed. Something is out of whack up there. The wiki history seems to be missing the older version that I remember. Either that, my brain is playing tricks on me. I remember introducing challenge #6 over much stronger claims than "programmers find Lisp convenient". A claim like that is relative enough to not trigger my Prove-it finger. Thus, either what I responded to was removed or I misinterpreted what is there. --top

But what doesn't have a complex data structure internally? You seem to think there's something different about things like Lisp, but to the extent that you say things that are correct about Lisp sometimes, you make them incorrect by thinking they are unique to Lisp. At the general level you're describing, a compiler or interpreter for e.g. PL/SQL does precisely the same thing that you're saying Lisp does. Until that becomes crystal clear to you, you are misunderstanding Lisp quite fundamentally.
I'm not insisting you become a compiler expert, I'm just pleading with you to recognize that you're not, thus far, and therefore to stop making strong assumptions on the topic. -- Doug
- What "strong assumption" did I make? If Lisp does not have an internal data structure, then I have encountered something very foreign.
- Yep. But do you get my point that PL/SQL does the same thing that you're saying Lisp does?
- I mentioned Lisp because the Greencodd rule is borrowed from Lisp lore, not because it is unique in that regard.
  - Couldn't it be said that Lisp allows one to manipulate internal structures more than most languages? The difference between a "loop" and a data structure are more blurred in Lisp, while in say PL/SQL or C they are hard-wired concepts that the language user cannot do much with.
- By itself that's fine. And I agree with you that something like GreencoddsTenthRuleOfProgramming exists; people are forever going off and reinventing half-assed databases instead of just using a perfectly good off the shelf DB. So I agree with your central point, there. HOWEVER, where I take exception is that you '''immediately went off into the weeds, right in your first paragraph, when you said "Ironically, even Lisp, the original "greenspun" language, reinvents one by having an internal complex data structure (nested list in byte-code form) that takes on database-like characteristics." -- no, that's not true, not even a little bit, and that's the only point I'm arguing here.
- So despite the fact that you are springboarding from Greenspun's Lisp rule, nonetheless, you really, really don't want to talk about Lisp the way you do above, because you're incorrect about that -- about Lisp, not about GreencoddsTenthRuleOfProgramming.
- Saying inaccurate things about Lisp weakens your point, whereas if you picked a nice clear true example, you would strengthen your point. The most obvious example is the way that people are always trying to use spreadsheets as databases, developing more and more complex macros, etc. -- Doug
- I suspect if we continue this, it would turn into a LaynesLaw debate about "database". However, to improve Wiki harmony, I am willing to remove references to Lisp and related material from this topic. Before I do, does anybody have any qualms about such an action? I propose moving it to LispAsaDatabase?. --top

As a side issue, I suppose I should ask if you believe in any of the mainstream orthodox definitions of "database"? Hopefully there are definitely things that are databases, and also things that definitely are not databases? Persistence hopefully figures into the definition? And although most e.g. business applications should use a database, other kinds of apps need not, right? Like, oh, say the calculator app that comes with Windows, which doesn't keep a history -- it doesn't need a database, so long as it doesn't keep a history, right? -- Doug

Re: "Persistence hopefully figures into the definition?" - There is talk of "RAM-only" databases being prepared to improve speed. Disk is merely a convenience, not a prerequisite to be called a "database". Ideally a database should abstract away storage implementation mechanisms from the database user. This was a key goal of DrCodd. I would generally call a database a "general tool to manage large quantities and/or volumes of attributes". By "general" I don't necessarily mean it can be used for everything, but rather it is not geared toward any single application but rather an attempt to provide a tool to handle common attribute-handling needs of multiple or different applications. If you can think of a tool that fits this bill but would not be considered a "database", then perhaps this working definition needs amendments.

It's potentially insufficient in several ways:
- It's not necessarily (certainly not inherently) about large quantities/volumes of data; databases are important for small quantities/volumes as well. This isn't a nitpick, the widely-used Microsoft Access database system is used exclusively for small quantities/volumes. (Although admittedly it's been said that it's not a "real" database, but it's just an example. Oracle has been used for small needs.)
- In general people inherently mean "persistent data" when they say database, but I don't see an inherent contradiction with using a qualifier such as "RAM-based non-persistent database", I would merely object if the qualifier were skipped when describing such a thing, given the usual assumption. A "non-persistent database" is an unusual kind of database, not a strong exemplar of the class.
- And most importantly of all, databases should offer transactions with ACID guarantees; it's common for people to say "that's not a real database" if such are not offered (e.g. MySQL some time back). What good is data that's not trustworthy?
- Oh, and in this day and age, "relational" is assumed if no other qualifier is used. A non-relational OO database is indeed a database, but it leads to communicational problems to skip the qualifier.
- If something lacked all of the above attributes, what is to be gained by calling it a "database"?
- I don't see why taking away ACID and disk significantly changes the fundamental nature of the thing in question. The developer writing queries may not even know or care if ACID or disk will be used. It may be removed or added down the road. Early DB's didn't have ACID when the term was coined, I would note. A related question is whether or not a hierarchical file system is a (hierarchical) database. I consider that borderline since they don't really deal with "attributes" as we usually think of them. File systems are more like "blob" management systems instead of attribute management systems. One cannot do a whole lot with the content in the "cells" (files) of a file system with the file system manager, at least not without external tools such as Grep or a specific application (I am viewing the OS and file system as separable things here). A database generally has some kind of query or navigation tool that looks at the content of the database. File systems tend to be ignorant of the content itself beyond retrieving and storing it.

Candidate defining properties of a database (and the "DBMS" that supports it) (feel free to edit):

shared access to data (from multiple applications)
concurrent access to data (from multiple actors)
ACID transactions (valid even if no concurrency)
persistence
large quantities of data
flexible/ad-hoc query

Not defining properties of DB[MS]s in general (though required for RDBMSs):

relational (OO/network and XML/hierarchical databases are still databases, whatever the technical merits)
logical-physical separation (again, a QoI issue)
constraint enforcement (PKs, FKs etc.)
(and more)

Can we agree on something like "any 4 of the 6 makes it a DB[MS]"?

Some test cases:

Oracle, SQL Server etc. -- definitely DBMSs
MySQL, Access -- DBMSs, but not as good, and not RDBMSs
WinFS, Reiser4 -- DBMSs
Excel -- almost a DBMS (concurrent, persistent, medium amounts of data, fairly flexible query)
- I've seen people misuse Excel and make DBMS out of it because it was the only thing they knew.
Word -- not a DBMS (persistent, medium amounts of data, but that's all)
Calculator -- not a DBMS (possibly persistent, small amount of data)
Lisp -- ?

Obviously, the more you implement without using a DBMS, the nearer to GreencoddsTenthRuleOfProgramming you are getting.

There is already a list of "database features" under DatabaseDefinition.

This is overblown, rather like Greenspun's rule. I program 'little databases' to go with 'little languages'. The former look nothing like Oracle just as the latter look nothing like LISP. Extending definitions till they cover the entire universe makes the definitions rather pointless. So perhaps the question is: "what is NOT a database by this definition?"

Are there similarities that can be factored to a single tool? Do you reinvent them because there are no satisfactory existing implementations, or because the needs are too different per app?

Mainly because I can, but also because I rather like minimalist designs that run really fast, and gluing a giant general purpose persistence engine onto a small app is silly.

Do you think a DB or DB-like tool can be both minimalist and somewhat general purpose? Plus, don't you risk a DiscontinuitySpike when you have to scale something to a RDBMS of some kind? Further, I disagree that DB engines have to be big and bulky. I agree that existing tools often don't offer much choices, but easy tables were common in the 80's before OO hype pelted such tools with bloated interfaces and bloated engines. The Oracle model is not the only model. Further, the DB engine is now smaller than many of the related engines with languages such as Java and Dot.net. The bloat argument does not hold much weight. (Note that the SqLite engine in C is available for embedded apps.) Related: SimplifyingRdbms and MinimalTable.

An app is stepping into DB Land when it has lots of repeating "instances" of something and DB-ish operations (CollectionOrientedVerbs) tend to be needed on these intances. These actions include the likes of search, sort, cross-reference, summarize, concurrency management, etc.

An example of GreenspunsTenthRuleOfProgramming may be found in TooMuchGuiCode. GUI's can largely be reduced to attributes, and managing those attributes via code (set/get's) can be tedious, verbose, difficult, and not easily sharable or reusable across languages. For example, it would be nice to be able to query the GUI attributes for certain kinds of debugging. However, code-centric techniques often don't make such very easy. You have to guess what is going on based on static code. (Related: AdvantagesOfExposingRunTimeEngine.) --top

Re: "My ObjectBrowser has query-like capabilities, so I don't need a stinkin' database."

Dude, you HAVE a database now, it's just not a relational one, and uses RAM instead of disk.

Combine with ReinventingTheDatabaseInApplication? Except, sometimes it is language features, not just applications. See DoubleCheckedLockingIsBroken.