The Prevayler
The PrevaylerAn OpenSource PrevalenceLayer. http://www.prevayler.org
Transparent persistence for JavaLanguage business objects. Orders of magnitude faster and simpler than a traditional database. Write plain java classes (albeit in accordance with a CommandPattern and a few constraints): no pre or post-processing required; no weird proprietary VirtualMachine required; no inheritance from base-class required. Clear documentation and demo included.
-3251 times faster than MySQL. -9983 times faster than ORACLE.For certain operations
How is this possible?
See also http://www.advogato.org/article/398.html
Oh oh, bench-mark claims
The performance scalability test is included in the Prevayler2 distribution. It is frequently run and scrutinized by skeptics. You can do the same, unless you are just slandering. --KlausWuestefeld
[ PrevalenceIsAcid ]
Apples and oranges. What MySql and OracleDatabases do (across a network connection, I might add) is different that what ThePrevayler does. Although some applications that use the former could instead use the latter, and might do their jobs faster as a result, saying ThePrevayler is "faster" than MySql is like saying a telephone is faster than a typewriter. They simply don't do the same thing.
Well, whether they overlap or not is perhaps debatable. But the primary reason it is fast is because it is language-specific. Most RDBMS are not. Data tends to outlive languages, by the way. There are a lot of other issues to consider. For example, for read-only operations MySql tends to be faster than Oracle. This is because Oracle is tuned for a mix of reads and writes. Thus, a benchmark that favors reads will give a potentially misleading score. Further, if you later need some of the features that RDBMS typical provide, you may have to end up implementing such yourself and then no longer win speed races unless you spend years using all the optimization tricks that database companies spend a lot of effort on. It is often said that human maintanence is more costly than machines.
KentBeck, RobMee, HumbertoSoares? and KlausWuestefeld paired up for a weekend in december 2002, on the island of Florianopolis, Brazil, and implemented Florypa, a minimal prevalence layer for Smalltalk (VisualWorks) based on Prevayler.
Is there a write-up anywhere on prevayler that doesn't use this mock question and answer format? Asking yourself leading questions and then answering them is a really annoying way of presenting anything.
No.
Aren't you simply dropping all code from DBMs and stuffing it all on the application?
No. I have seen prevalent systems that were thousands of lines of code. Most DBMs are hundreds of thousands of lines of code. --KlausWuestefeld
Citations please? Many open source relational databases are signficantly less than hundreds of thousands of lines of code.
First things that come to my mind are referential integrity, schema evolution, etc.
The JavaVirtualMachine takes care of referential integrity. Don't worry, it will not let you put a Pineapple object in a Person variable. As for schema evolution, does your DBM write your data migration scripts for you? --KlausWuestefeld
Type checking is not the same as referential integrity. The JVM will not enforce a NOT NULL constraint, or UNIQUE, or any of a number of things that might be expressed as a CHECK constraint, of course.
Reinstating this question, using GreencoddsTenthRuleOfProgramming: "won't a sufficiently complicated database application written using PREVAYLER contain an ad hoc, informally-specified bug-ridden slow implementation of half of a DBMs?"
No. Your DBMs probably didn't allow that but with Prevayler, you are finally free to use other people's OO code. --KlausWuestefeld
That doesn't answer the question. You mean "yes, but you can borrow someone else code to help build that 'ad hoc, informally-specified bug-ridden slow implementation'. What's porridge got to do with anything here?
Are the best practices one finds in today's database theory all best forgotten?
(These questions are beginning to seem fabricated.)
No. But you can forget the "best practices" derived from the AccidentalDifficulty of paging data blocks from RAM to disk and back. --KlausWuestefeld
If Prevayler would evolve, wouldn't it turn out to be something like ObjectStore?.
No. That would be a step backwards. --KlausWuestefeld
While there is no "inheritance from base class required" there is a rather important architectural restraint involved. Applications utilizing prevayler are required to adhere to a simple CommandPattern. No great burden, and in fact a nice guideline, but the statement above could easily be misinterpreted.
The client code has to issue transactions to the business objects through the simple CommandPattern. The business objects themselves, where the real gain is, have no such restriction. -- KlausWuestefeld
Seems interesting, but it doesn't seem to provide multi-user support (i.e. locking / concurrency). I assume this is ignored by requiring commands to be executed sequentially.
Commands (transactions) are executed sequentially so you don't have to worry about concurrency between them. You do have to worry about concurrency between commands and queries, though. Prevayler does not have to specifically provide for that. Remember: all your business classes are regular Java classes. You can use all the Java language synchronization constructs normally. -- KlausWuestefeld
"Command/transactions are executed sequentially" - doesn't that make the system really hard to scale?
No. Every command takes only a few microsecond to run. This is partly because all data is stored in RAM.
There's no guarantee that a command will take a few microseconds. A command could take years to run.
It could. But it doesn't have to.
Remember that read-only operations can still run (depending on your synchronization solution).
What synchronization solution? How do you perform a read-only operation while a command is executing without a chance of getting partially modified data?
In Java, the most common solution is to use the "synchronized" keyword. Just solve it the way you usually solve concurrency issues in your code.
SnapshotPrevayler? synchronizes on itself for each command. If another thread reads from the PrevalentSystem? during that command there's no guarantee of data integrity. Are you suggesting that code called by the command has to grab locks before modifying data that might be read? If so, how can read-only operations be processed while a command is executing?
I don't really understand what you mean here, but: Yes, depending on the semantics of your business objects, you may need to use "synchronized". Just like in any other Java application the other threads can still access the remaining, non-locked, objects in the system.
Read-only operations (a.k.a "queries") don't need to use Commands, but instead access the system directly. Maybe that's where the confusion is?
AnticlimacticSimplicity... Hmm, I like that phrase.
I'm just trying it out. Impressive! Looks like it's perfect for prototyping! -- FalkBruegmann
Many people are already using it in production.
I think you will find http://pot.forgeahead.hu/ both faster and better for prototyping as it lets you use any POJO data structure you can come up with. -- Bjorn Blomqvist
You might also want to look at JavaSpaces. The Sun contributed version is an "all in memory" implementation of an AssociativeMemory like Linda. Of course the problem that we found writing enterprise applications was the prevalence hypothesis. IOW, right now, today, there IS NOT enough ram. If you are dealing with Gigabytes of data, you will eventually fall over with today's machines. However, for those not dealing with large datasets, I would definitely recommend something like Prevalyer or JavaSpaces or any other in memory log-based PersistenceMechanism. My feeling is that the correct solution (as always) will end up being a hybrid solution. After all, some things you want to keep on disk. I can think of no good reason to keep years of transaction history for some application FOO in memory. The only time you'd want to bring it into memory is when you are going to run some rare report. Another example is an archiving or versioning system - why keep binary images of files in memory? Just because it's data? No. Again, a hybrid approach is probably the best. -- RobertDiFalco
You could easily wrap your binary data using some scheme where the only time you bring the large chunks of data into memory only when they're needed. This requires a few extra lines of code, and I don't see how one could arrive at the conclusion that because of this one needs a full-fleged RDB implementation.
-- I agree that a hybrid approach wins just as it always has. However the important thing to note is that an RDMS is no longer a part of that hybrid.
What about history? How many years of history should I keep in RAM? Do I have to use different classes if I want to load some state from ram and other state from a persistent archive?
If you want (?). There would probably be well-factored, not-very-instrusive ways of accomplishing this, with aspects, etc. But it sounds more like you just don't like this whole idea. I can think of many (most) of applications I've worked on, in fact most of the "serious" enterprise apps, that would have no trouble fitting in RAM. Of course there are an infinite number of possible applications that wouldn't fit in RAM without the modifications I suggest. This is all stated up front, no one is saying this thing is supposed to be all things to all people (unlike, say, EJB or RDB marketing).
[Summarize] : Me: It doesn't do what rdB's do (platform independent, fast bulk queries, arbitrary associations, multi-access). Anonymous: Do you need to do those things? Me: Yes. Perhaps not all of the time, but most of the time. Exactly. You need a tank, which requires lots of maintenance, slows down progress, etc, but has features you need. Most people just need cars -- even for "enterprise" apps.
I think a simple form of object persistence is an excellent idea for saving session state. If it can be transactional too then that is a plus, but depends on the application working hard to externalize updates as transactions. A simpler scheme, such as checkpointing, would seem enough.--RichardHenderson.
I'm actually using this in a real world project and have had none of the problems RDB people are positing. The only thing I've missed about relational databases is cascade deletes, and I plan to move to something like RDF in the future to solve that problem. What they don't mention is all of the hoops you have to jump through to build and maintain and change a RDB, the extra overhead you incur running unit tests, and depending on your organization, the DBA bottleneck. The bottom line is if you need crash-proof persistence - which is all most applications really require - something like Prevayler is all you need. An in the future if you find you require a RDB, you can always refactor.
What if there are other resources, e.g., JMS channels, transactional email services, etc., that need to be involved in the transaction? Is ThePrevayler XA-compliant? --RandyStafford
Not today, so this would be a requirement that would lead you to use an XA-compliant PersistenceMechanism, possibly a RDB. Many of your questions might be answered faster by simply downloading the source and taking a look at what it does, there's not much of it to grok.
Actually, two-phase commit is unnecessary. --KlausWuestefeld
Actually it is very much not so. But make up your mind: you either contribute here to WardsWiki and allow your claims to be discussed and possibly debunked here, or stop making claims supported by external links only, as people here are unlikely to contribute to yet another wiki. By the way, have you studied TransactionalInformationSystems ? --CostinCozianu
Isn't ThePrevayler based on two phase commit? It writes each deterministic transaction to a log before it performs it. The problem is that it can only deal with deterministic transactions, and JMS, email, etc. aren't deterministic. -- EricHodges
Actually the TwoPhaseCommit is the algorithm for resolving commit commands for distributed transactions. What prevayler does is a basic recovery mechanism, and it sweeps concurrency control under the carpet by serializing all concurrent acesses to the global object model. Well, it works in situations, but you have to be very careful in analyzing whether it works for your particular problem. If you have a distributed transaction situation, the lack of support for TwoPhaseCommit is can be a very serious deficiency, but again, depending on the particular situation it may not be too big a problem. Because what TwoPhaseCommit does at the infrastructure layer, transparent to the developer, could be done at the application layer with application specific business logic. I guess that's what Klaus' article it's trying to say: that you can deal with the issues at the business logic level. However, depending on the situation that extra business logic you have to add can be extremeley simple or it can be extremely complex, and I'm affraid the examples he presented are not very representative of how and why one would be in a distributed transaction situation and how to handle it.
But TwoPhaseCommit is implemented just like ThePrevayler. Write a message to a log indicating what you're about to do, then do it.
Not at all, there are fundamental differences in implementations. Of course they share some similarities because everything that has to do with transactions has got to write some into some log. But other than that, the two algorithms are fundamentally different. And by the way the solve different problems. Prevayler solves the problem of persisting the effects of a transactions against a single system in case of a soft crash, and even that implementation has problems. The distributed transaction means that a transaction that spans multiple database managers have to be either committed by all or rollbacked by all the participants, and of course the effects persisted, in presence of both soft failures, hardware failures and communication failures. TwoPhaseCommit doesn't offer an absolute guarantee that it will meet those requirements , as it is provably impossible, but it comes very close, and in the unlikely event that it fails, manual intervention by the DBA with database specific tools can solve the remaining problems.
I've implemented TwoPhaseCommit the same way ThePrevayler does. What are the fundamental differences in implementations?
Like for example the ability to roll back, and the need to contact a central transaction coordinator and deal with communication errors. See TransactionalInformationSystems, for more details.
We're using different defintions of TwoPhaseCommit. The two phase commits I've implemented had no roll back and no transaction coordinator. They were used for disaster recovery. If the system crashed during a transaction it could resume the transaction when it started again. I see from google that my usage is in the minority. Nevermind.
"Actually, two-phase commit is unnecessary"
After having testing Prevayler (and Prevayler-like IMDB's) we came to this conclusion: It is useful for prototyping, but fails miserabely in terms of a lot of OODBMS/RBMS issues.
In fact, we concluded that when all the problems with Prevayler was solved, we had an object-oriented database.
I agree with you there. I use Prevayler and like it a lot but the web site is very offputting.
Well the benchmarks are a little silly too. There are a ton of things that Prevayler is not faster at than RDBMS systems. Not only transaction processing but there are many complex sorted tree queries that I'm pretty certain would be slower on Prevayler than a well designed RDBMS when operating on my multi-terrabyte datastore.
-9983 times faster than ORACLE. -- You should be specific about the operations.
Prevayler, and much of the above, seems to miss the fundamental point that traditional DBMSes provide reliable, language-neutral, integrity-maintaining storage of vital enterprise data in a manner that supports a variety of applications, not just those written in Java. Enterprise databases tend to long outlive most applications -- or even the popularity of the languages the applications are written in -- because they represent genuine business assets; they are the valuable knowledge of the business captured in an accessible form. They are worth money. In some cases, they're worth a lot of money. In most businesses, the applications that access the enterprise databases -- or even that just require persistent storage -- with few exceptions are nothing but expensive overhead. They cost money. In many cases, they cost a lot of money. They are merely the necessary (and, from an executive or stakeholder point of view, unpleasant) expense of presenting and updating databases in a user-friendly manner. The databases -- i.e., the all-important data -- will still be there, delivering business value long after Prevayler, Java, and any associated applications and their language-de-jour development environments have come, gone, and been written, re-written, and replaced numerous times to suit technical fashion and petty (again, from a stakeholder point of view) tweaks to requirements.
Prevayler and its ilk do nothing but increase overhead -- due to having to port serialized Java objects (or whatever) to a new environment -- when it comes time to replace the Java applications with something better, newer, or cooler. Far smarter would be use reduce the cost of using Java (or whatever) with traditional SQL (or truly relational) DBMSes, so that the expense of using a DBMS is minimised while ensuring that the database needn't be ported to a new environment when the Java application is (inevitably!) replaced. -- DaveVoorhis
Language neutral?!? I've yet to meet two databases that speak the same language.
I think you mean you've yet to meet two DBMSes that speak exactly the same language, and indeed most don't. By "language-neutral", I mean that DBMSes are not dependent on any particular client-side language. You may switch from COBOL to C to C++ to Java to C# to Ruby to the NextBigThing, as many enterprises have done, or use all of them simultaneously, against the same DBMS. Prevayler is dependent on Java.
{Prevayler is dependent on JVM, not on Java. You could use Prevayler with Clojure or Scala if you wish to do so. I'm not so certain that this 'language neutrality' is a real benefit in any case, especially when what it really means is "Use MetaProgramming!". Excluding (not insignificant) concerns about existing documentation, how much more difficult would it be, relative to crafting SQL, to craft a small Java program each time I wish to query a Prevayler database? Heck, ThePrevayler already follows a CommandPattern; I don't need to go nearly so far as expressing a whole program - just crafting a command will do. Move Prevayler up to a 64-bit address space and have it handle the paging issues, possibly add SoftwareTransactionalMemory to allow concurrent transactions, and ThePrevayler could probably meet the scalability requirements for enterprise use.}
Despite the fact that Prevayler might be shoehorned onto some of the obscure languages that run on top of the Java Virtual Machine, the Prevayler home page states that "Prevayler is an open source object persistence library for Java." For. Java. I'll take Mr. Wuestefeld's word for it (literally), even if it isn't strictly accurate. As for your notion that it might be appropriate to "craft a small Java program" each time you wish to query a ... database, that may indeed be achievable in theoretical sense, to be practically reasonable it would mean making Java available in every context where SQL is used today, and the subsystems that host it would need to exhibit the same behaviour as a DBMS. Until that happens, Prevayler remains dependent on Java, the Java Virtual Machine, whatever.
Fine for RDBMS. However, I referred to DBMSes in the original paragraph. The main benefit of a DBMS is its role of maintaining long term independence of the corporate application infrastructure. The physical independence of the DBMS means its asset value can be relatively easily and inexpensively maintained over the long term (decades, now, in many cases) without risk of being contaminated by the expensive fashion-driven application development churn.
{Prevayler won't be replacing the RDBMS. But I would not be surprised if the RDBMS is eventually 'endangered' by PersistentLanguages - especially as they grow into the CloudComputing era with composable security models, ACID transactions, and high levels of scalability. A language with an advanced LogicProgramming subset would be raring for a takeover. Performance and features are likely to both be advantaged under such a design due to fewer serialization steps and better type and object integration.}
{I honestly wonder how well Oracle's years of fine-tuning its B-tree layouts, page-locking mechanisms, and in-block record-shifting (which is remarkably expensive, but necessary to maintain ordering, and tends to fight making blocks very large) would hold up against new designs for GarbageCollection, OptimisticConcurrency, and shared-structure writes. I'm guessing Oracle would do better (at least initially) due to specialized block structure, good query optimizers, and sheer experience of thousands of DBAs hammering out and documenting performance tweaks for many years. But that'd be excluding serialization, and the advantages of type integration, potential code specialization, and JustInTimeCompilation...}
"Language neutrality" aside, it doesn't "fit" other languages nearly as closely as it fits Java, removing one of it's main advantages. Sure, it may "run" in some other languages, but merely running is not enough to dethrone RDBMS's.
{SQL RDBMS doesn't "fit" other languages nearly as closely as it fits SQL. Thus, by your own logic, it also lacks the "main advantage" that ThePrevayler loses. So, that simply puts ThePrevayler accessed from external languages on even ground with respect to SQL RDBMS accessed from external languages. It's also why we've got 'procedures' and 'triggers' and such being added to SQL standards - attempts to reclaim some of this advantage.}
{The paragraph to which you responds is about PersistentLanguages and competing with RDBMS on an enterprise scale. (My exact words were: "I would not be surprised if the RDBMS is eventually 'endangered' by PersistentLanguages.") For that, I think Oracle is (if not the best) one of the best comparisons.}
{That's nothing special. It isn't as though the normal storage format for an RDBMS is any less proprietary binary. Data is made available through a protocol (e.g. ODBC) by a shared service that outlives individual applications. Query and DML facilities implicitly provide the necessary features to back up and restore data, thus allowing the data to live longer than any individual language. Data can be transformed without loss. There is no reason to believe that a dedicated process or object in a PersistentLanguage would perform poorly with regards to sharing or maintaining long-lived data.}
{If ThePrevayler were the incumbent, you could ask the same question in reverse: "Java is the de-facto standard for sharing info between multiple apps, languages, and tools inside a given org. Do you really want to switch to SQL? Sure, it has those nice LogicProgramming features, but how is it going to scale to multi-organization data sharing? What if it is replaced by some GreatNewLanguage?? Those DataLog and MercuryLanguage guys look like mighty powerful competitors to this upstart SQL, it might be gone before you know it! Besides, all the IT workers are familiar with it, and it works pretty well despite some warts." You have been claiming that a design is technically flawed on the basis of historical coincidence. I can't agree with such logic. I do not need to show Prevayler competing well against historical coincidence unless I'm making make a non-technical argument (e.g. about market success, adoption, getting support integrated into CrystalReports).}
{I do agree that ThePrevayler has no chance to unseat the incumbent on the basis of historical coincidence. Technical merit, after all, rarely determines market success, and I don't favor ThePrevayler even in terms of technical merit. But I'm confident in my hypothesis that someone over the next three decades will invent a PersistentLanguage, designed for CloudComputing and DistributedSystems, with a LogicProgramming subset with sufficient technical merit and the marketing edge (in terms of scalability, performance, and security sufficient for multi-organization data sharing) to put the 'dedicated' RDBMS down like an old dog. My own efforts at a GreatNewLanguage? are promising on this front.}
Good for you. I'm working toward similar goals -- a distributed, persistent language for general-purpose programming (including, perhaps, what is currently called CloudComputing), but fully taking into account corporate political and cultural concerns. I hope to do this as PhD research, but if that is not possible (for purely bureaucratic reasons; I'll find out in two weeks) I'll do it anyway. Within a decade, it will eat the world and you and Top will be begging me for jobs. :-)
No, two computer programmers on this wiki. What are you?
{I'm interested in learning more about the bases for your language design, and how you aim to account for corporate, political, and cultural concerns. I have my own fairly extensive accounting for what I consider to be relevant facets of those social concerns (and military concerns, too), but it may be neat to see how you're thinking different (e.g. what you consider to be problems needing partial solution, and what you consider to be the partial solutions). Do you have any blog or reference for your ideas?}
No blog or reference yet. There is my three page research proposal, but it's only been distributed internally at work and it's more sales pitch than information. In two weeks, I'll find out what restrictions, if any, there will be on informally publishing ideas and work-in-progress. However, I forsee a significant basis of my work being ExtendedSetTheory, but I may find reasons (I'm doing research, not advocacy) for that to change.
{Then I guess I'll see in a couple weeks. Good luck!}
It should be pointed out that ODBC does not require the use of SQL. The client can send ANY query language as long as it's text. Thus, a new query language could use ODBC as long as it's represent-able as text and can live with the "table shape" of the result. -t
That's funny, everything I can find indicates that ODBC requires the use of SQL as part of its specification. One could certainly create something ODBC-like that didn't use SQL, but then one wouldn't need to represent anything as text, nor would they need to live with the "table shape" of the result. (Although both would be likely as the first is a language in and of itself, and the results are naturally table shaped).
[ODBC was designed for connecting to SQL DBMSes, but there's essentially nothing that fundamentally precludes using the ODBC API with other table-oriented or relational query languages.]
From http://msdn.microsoft.com/en-us/library/ms710935%28VS.85%29.aspx
"ODBC defines a standard SQL grammar. In addition to a standard call-level interface, ODBC defines a standard SQL grammar. This grammar is based on the Open Group SQL CAE specification. Differences between the two grammars are minor and primarily due to the differences between the SQL grammar required by embedded SQL (Open Group) and a CLI (ODBC). There are also some extensions to the grammar to expose commonly available language features not covered by the Open Group grammar.
Applications can submit statements using ODBC or DBMS-specific grammar. If a statement uses ODBC grammar that is different from DBMS-specific grammar, the driver converts it before sending it to the data source. However, such conversions are rare because most DBMSs already use standard SQL grammar."
So, it appears that SQL is a required part, although it's allowed to use a DB specific language in addition to the SQL.
It's still not clear if non-SQL can be sent. As a last resort "trick", I suppose it could send a string constant as part of a dummy expression that contains query-language-specific content. As long as the server knows how to extract that string, it can use it.
[Non-SQL can be sent. I've seen ODBC APIs & drivers used with non-SQL applications, though it's not conforming to the standard.]
Yes, the previous comment is true: You can send any language you'd like using ODBC and JDBC. Relational databases use this to support vendor-specific SQL syntax. But if your driver does not support SQL then most 3rd party tools will fail, making your "support" of ODBC/JDBC questionable. -- JeffGrigg]
Do you mean 3rd-party tools that assume the DB uses SQL?
Yes. Here's a list of ODBC tools: http://www.topshareware.com/guide/hot/odbc.htm Now, how many of these will work with an ODBC provider that uses its own custom language and does not support SQL?
It's not ODBC limiting them, it's that they use SQL to communicate. For example, they may have a QueryByExample engine that generates SQL under the hood and send that to a remote DB of your choice via ODBC. After all, SQL is the Lingua Franca of query languages (for good or bad), and thus lots of tools are built around that.
[An ODBC driver is required to accept a particular variant of SQL. If it doesn't, it isn't really an ODBC driver, regardless of how closely it adhears to the rest of the standard. This doesn't prevent an ODBC driver from accepting other languages.]
[To be strictly accurate, the ODBC standard requires that an ODBC driver accept ODBC SQL, but there is no technical requirement to do so. As I noted above, I have seen ODBC infrastructure used to connect to external services via non-SQL languages. These were not intended for use with third-party ODBC tools. It was done mainly to provide a common API for both the SQL-based data sources and the non-SQL-based data sources, and probably because ODBC driver source code already existed that could be relatively easily converted for similar purposes.]
Perhaps it depends on how one defines "accept".
EditText of this page
(last edited July 10, 2010)
or FindPage with title or text search