Business Rules Metabase

Rather than coding BusinessRules in a generic language Java, C++ etc.

Put them in a high level quasi-language and/or "rules" database explicitly for the purpose of storing and enforcing business rules.

I've often wondered what a system would look like with certain constructs coded in Java, C++, or whatever and a set of business rules coded in some high level or visual language. What are the responsibilities of the low level constructs versus the high level ones?



Progress still continues with rules databases and inference systems. It wasn't ready in the 80s, and isn't ready today, but, really, it comes down to implementations of such systems along with the optimizations they provide.


This advice is "MotherhoodAndApplePie?" kind of right.

But:


Interesting Questions

It's nice to ponder abstractions like this, but I think many of us have learned the hard way that you pay big-time when you start using languages where code can't easily be run through a debugger, to say nothing of typing and good IDE's. -- SteveConover

A rules-metabase does not preclude a potential debugger. I am not a fan of strong-typing so will leave that debate to the many other topics that take on that HolyWar. --top


I find that business rules are often intertwined too much to put into clean "slots". Plus, you have to recreate context somehow for each slot. A better approach IMO is to factor out what can be readily put in data tables, leaving the rest to traditional code. Some things fit well in tables, and some don't. See TableOrientedProgramming, CodeAvoidance.

For example, how would something like this be translated into a simple table?

 //(dotted to prevent TabMunging)
 .if feature A {
 ...if feature F, G, and H {
 ......if feature J {
 ... ......doSomething_1
 ......} else {
 ... ......doSomething_2
 ......}...
 ...} else {
 ......doSomething_3
 ...}
 .} else { // non-A
 ...if feature F {
 ......doSomething_4
 ...} elseif feature G {
 ......doSomething_5
 ......} elseif feature H {
 .........if feature L or M {
 ............doSomething_6
 .........} else {
 ............doSomething_7
 .........}
 ......} else {
 .........doSomething_8
 ......}
 ...}
 .}

PageAnchor: 423

This sort of thing transforms into flat set of logic-rules in a fairly straightforward manner:

 on match A,F,G,H,J then 1
 on match A,F,G,H,~J then 2
 on match A,~F then 3
 on match A,~G then 3
 on match A,~H then 3
 on match ~A,F then 4
 on match ~A,~F,G then 5
 on match ~A,~F,G,H,L then 6
 on match ~A,~F,G,H,M then 6
 on match ~A,~F,G,H,~L,~M then 7
 on match ~A,~F,G,~H then 8
 take no action more than once
 take all selected actions in ascending order by number

You'd need to attach each of these rules to some sort of 'operation' (unnamed above). The operation would then be performed, or at least computed (e.g. to an ordered list of numbers - a compiled 'program'), in a data-context containing information on the simple-propositions A,F,G,H,J,L,M. You'd let the business rules database figure out how to optimize it. This example is rather trivial since the consequences of the actions don't affect the decisions to take certain actions (though I'm assuming action 5 doesn't affect the state of H,L,M). A more AI-based ruleset that I would typically deal with would include rules saying when the robot needs to go back to the refueling station (based on point-of-no-return heuristics and a big margin for error) so it can both alert the operators and add it automatically to a mission plan. (Unfortunately, I'm not being paid to work on that at the moment). It isn't too rare that there exist places where the consequent of a 'decision'-rule or 'planning' rule can affect future decisions or plans. Handling these requires describing post-conditions and an approach to handling the FrameProblem.

Admittedly, the if/then/else style programming is likely a substandard description of the rules - it has already implemented the heuristics of the programmer and such. 'Disassembling' hand-optimized code like this is not the best way to go about implementing the business-rules; better would be to examine the original source - the requirements document - and figure out how to encode those rules directly, such that the whole of each rule, and potentially its intent, is clearer to later maintainers. A flexible BusinessRulesMetabase might then allow you to embed decision-strategies and heuristics and fuzzy issues of confidence and priority for the purpose of optimizing the runtime or result if the default strategies and heuristics prove to be inadequate. --AnonymousDonor

I actually reengineered, developed and support for couple of years some rules like this based on T-SQL stored procedures. Code was actually transitioned at least this way: BigIron (70s - machine code, 80s - assembler), Clipper (late 90s), T-SQL (middle of 2000s). Business users (accountants) and programmers prune rules for years(!) with no real success. After coding rules into table with structure like this, business users finally able to understood what was wrong all along and after that, rules table grown nearly twice in overall number of rules and we really easily introduced several new classes of rules.

Actual table (based on example):

  OrderID ActionID A F G H J L M
  ------------------------------
  1       1        1 1 1 1 1 ? ?
  2       2        1 1 1 1 0 ? ?
  3       3        1 0 ? ? ? ? ?
  4       3        1 ? 0 ? ? ? ?
  5       3        1 ? ? 0 ? ? ?
  6       4        0 1 ? ? ? ? ?
  7       5        0 0 1 ? ? ? ?
  8       6        0 0 1 1 ? 1 ?
  9       6        0 0 1 1 ? ? 1
  10      7        0 0 1 1 ? 0 0
  11      8        0 0 1 0 ? ? ?

--AnonymousDonor

(in response to the example): Agreed. These sorts of things seem like very brittle abstractions that hold up well in esoteric design meetings, and are utterly destroyed in any PairProgramming session over 10 minutes long. -- SteveConover

Regarding the example at PageAnchor 423, Because the example is fairly trivial, we can easily turn those particular conditions into a list of AND conditions. However, in practice one needs more complicated Boolean expressions or more complicated expressions in general. But this would simply lead to non-normalized (non-nested) IF statements:

  if (foo and bar or x or y) then do_A
  if (bar and x or y or z) then do_B
  if (x or y and z or q) then do_C
  etc...
  // letters not intended to match original example

Thus, your solution has not simplified nor removed the need for IF statements or something very similar to them. It merely reshuffled the IF expressions: a different factoring, not necessarily a better one. And I am skeptical of the notion that if one simply thought about the problem hard enough, they could remove most cases of messy business rules. Business rules are just plain messy sometimes (EssentialComplexity). Some of the reasons for this are described under AreBusinessAppsBoring. The bottom line is that one is modeling human thought and politics, and not physics or natural laws. Humans are flawed and/or often have other priorities besides simplicity.

--top

If you consider describing a set of patterns and performing "pattern matching" in general to be "very similar" to a bunch of if-statements, that is your prerogative. But, from a language-design and algorithm-implementation and every reasonable and well-informed perspective, the control-flow management characteristics are very distinct. People writing a bunch of 'if/then/else' statements need to concern themselves with the ordering in which they provide the statements, need to know how one event affects future events rather than just writing that down someplace, etc. A group of business rules that apply in the context of known patterns/situations and a particular operation, provided in no particular or relevant order, is something that I, as an intelligent and well-educated individual and expert programmer, cannot consider to be "very similar" to IF statements.

And, despite what you inferred, I didn't imply that you could "remove most cases of messy business rules". Rather, I implied that both rules and all the messy exceptions to these rules would, could, and should be encoded in a straightforward manner: exceptions and specializations are also business rules. I also said that reverse-engineering the if/then/else solution is a poor means by which to create the business rules for the BusinessRulesMetabase because this mechanism makes the error of embedding any simplifications and assumptions and optimizations and heuristics the programmer used. -- AD

If ordering matters, then some mechanism needs to be in place to determine the order.

I won't disagree. Rules about ordering should be included.

IF statements indicate that by their position in the code text. You have not showed how you handle ordering. One approach is to have a priority or sequence number. If ordering does not matter, then its a wash either way.

I rapidly gain the impression that you are not very literate. {IsYourRudenessNecessary} I did, in fact, have a rule indicating for ordering in the above example, written in plain English. There are a wide variety of ways to indicate ordering rules, but I don't wish to turn this page into a discussion on Planning and Workflow logics. And regarding the "its a wash" comment: if ordering really doesn't matter, then being forced to express one should be considered an objective disadvantage - it is semantic noise, a LanguageSmell if it is forced on you by the language, and a DesignSmell if it is not; with it comes all the problems of PrematureOptimization.

The order-ness was not clear in your made-up language. Ordering matching the code sequence is more WYSIWYG. But, it is true that a metabase could be sorted by the priority value. Still, its easier to edit and change positions of blocks than data-entry of numbers (at least for my brain). If one representation is objectively better than another, then I ask that you define and demonstrate your metric(s). --top

The main advantages of business rules metabase over a vast array of if-then sequences is the centralization of business rules and the clearer expression of them, both of which aide with maintenance and tweaking - i.e. you don't need to run around finding and deciphering all sorts of 'if-then' statements distributed across the client and server application space in order to locate the rules, and you don't need to spend time figuring out how to translate rules into if-then sequences in the first place. When a business-rule changes, you change it in one place, and the BusinessRulesMetabase (as with DeclarativeMetaprogramming and LogicProgramming in general) is responsible for distributing the change and handling the ripples across the application space. However, even when you aren't expressing opinions inconsistent your own table-oriented policies, everything seems to be subjective to your mind: if you happen to like this extra labor of translating between if-statements and business rules, maybe that's best for you. I just happen to believe that it is better to let the computer do as much work for me as possible.

Without a specific domain example, it's probably difficult to investigate the options further. One nice things about IF statements is that they can use context to avoid repetition of similar info or logic. All items within a given block "inherit" the same condition. If one "node-ifies" the rules, that is, make them all independent expressions or rows, then it tends to require repeating the same info over and over. Sometimes this can be remedied by using sets such that nodes can inherit commonalities from common sets. However, sets can be difficult to study/navigate visually, unlike nested conditionals (and trees in general). The advantage of a rule meta-base, though, is that tools and reports can be added to help study and browse the set relationships, perhaps using something like GraphVis?, but with more interactivity. It turns sets into visual graphs (networks). --top

Re: "The main advantages of business rules metabase over a vast array of if-then sequences is the centralization of business rules...i.e. you don't need to run around finding and..."

I agree that database-like features facilitate a more dynamic grouping of rules such that different aspects can be "sorted together" or re-projected differently as needed. One is not stuck just with just the original author's view/grouping the way you get from text files. See SeparationAndGroupingAreArchaicConcepts for more of my opinions on this. However, our current development tools and conventions have to mature before this happens. Ideally one could view and edit such rules as ANY of the suggestions given here. It's merely a projection/view of information suited for a particular need. We wouldn't have to choose between trees (nested blocks) or tables or stick-and-node diagrams or Venn diagrams: we could view and/or edit it as ALL of these as we choose! That's the future I would bet (if I live long enough to see it). It's makes our WetWare happier to give us the view that best fits our needs or preferences, and thus more productive. --top


Business rules are often inherently complex and seemingly arbitrary. Simplified abstractions either don't exist or can't be relied on to last over time. Exceptions to the "rules" of our abstractions often build up higher and higher over time (AddingEpicycles) until it looks like it may be better to switch to a lower abstraction. (See LifeIsaBigMessyGraph.)

Realistically, a 'simple' table would be a bad decision. You need a collection of 'rules', not of 'strings' or anything else. SimplifyingRdbms - you need the ability to declare a type that contains whatever structure is appropriate to hold various rules and patterns.

It helps to separate what can be done (the actions, doSomething_1 to doSomething_8) from the circumstances under which it is appropriate to do it (the conditions, presence of combinations of features A and F to H, L and M). Inverting the representation can help to clarify: are the preconditions for doSomething_1, say, all present? (These are A, F, G, H and J, by the way) Then a simple table could represent them as:

   Action Precondition
 1  A
 1  F
 1  G
 1  H
 1  J
 2  A
 2  F
 2  G
 2  H
 2 !J
 3  A
 3 !F
 3 !G
 3 !H
 ......
There are more elegant and more sophisticated approaches, but this is pretty simple, as requested.

I don't find this a very useful format for editing rules. Plus, it will get messier if we have more complex boolean expressions. I don't know any easy way to represent such a pattern in tables, and it is a common pattern. Perhaps this is something better for trees than tables (AbstractSyntaxTree).

It is not primarily a format for editing rules, or for representing a pattern, though it can do both... whether it's useful depends on your perspective. If the system is set up to follow the rules thus represented, then simply altering the contents of the table changes the behaviour of the system, which could be either terribly useful or completely disastrous. The more complex the conditions you seek to represent, the more complex their representation, granted. This is not necessarily messier, however. The overall complexity of the system is almost certainly higher, but if the rules are managed by the user community, then the complexity can be partitioned more appropriately between users and developers.

The bottom line is that humans must maintain it. A tablized form of the above is harder to maintain in my opinion. Whether it is more or less "complex" is perhaps subjective or moot. I see little use in using tables to get rid of the programmer if the total labor of maintenance is going to be higher. The nested approach is easier to grok IMO because it uses context (physical proximity) to factor out some of the repetition. Everything under "feature A" does not have to state over and over that it is under feature A, unlike the table version. I don't know of any place that has successfully used tables to represent and edit non-trivial Boolean expressions, let alone nested logic, and I am one who usually pushes tables in general.

Perhaps we need to explore general ways to represent many-to-many tables so that they are easier to visual digest. Outline-browsers have worked well for trees, and now we need some comparable GUI conventions and techniques for many-to-many.

Humans or their computers must maintain either code or data. Replacing expensive programmers with cheaper users can make economic sense, especially if the turnaround time for updating the table is shorter than for amending the code. I'm not sure what you mean by a 'non-trivial Boolean expression', but I guess it would be one that could not be represented in a table-based system of arbitrary sophistication (rather like intelligence to the anti-AI; what's the equivalent of the TuringTest here?). If you can represent a combination of two conditions, you can represent a combination of an arbitrary number of conditions, so I guess that isn't your objection. If you feel that putting the actions, as opposed to the conditions, into tables is where the problem lies, then I would agree with you; in general it's best not to try.

I am suggesting that there would be no savings. I am not saying it could not be tablized, for anything can with enough tables and rows. The problem is that it will take much more labor to maintain such rules as tables than as code in most cases, even if the non-programmers are paid less than programmers. I have a pretty good feel for what confuses users, and I doubt many will get or enjoy the concept. Sure, there might be one out of a hundred who quickly grasp it, but that is the exception. The user would essentially be a programmer now. For one, there will probably be a couple of many-to-many tables. Many-to-many tables are tough to read and work with for most people, at least in their more raw form. Most GUI's "hide" many-to-many tables by projecting by one factor at a time. But, it results in having to hop around from screen to screen. Programming code is the more concise representing of such "rule trees" IMO, with a tree-browser in second place.

Think of it this way. Is the purpose of tablizing it to make it faster to work with for skilled people once they learn it, or to make it easier to learn than code? If the second, then even when they fully learn it, they will never be as efficient as a programmer would be doing the same thing in code. They would have to have sufficiently lower labor rates to justify the extra time, and programming rates are dropping due to globalization and liberal visas. If the first, then why do we usually use code? The concept of Boolean "decision tables" has been around at least since 1970. There was a book that came out about them around that time (this is from fading memories, so don't hold me to it).

Think of it this way... Over the last five years, our company has reduced the average development man-days required for a product launch from over 12 to under 4 (excluding acceptance testing). This has required nearly 500 development mandays to build up a library of over 600 BusinessRules supporting over 50 product launches so far. That is an average of 10 days per product launch, already less than the original 12, so the changes have paid for themselves without taking into account any benefit from improve turnaround times. These savings have been achieved almost exclusively by replacing business logic that was embedded in code with database accesses.

Now, we don't force (or even allow) our users to confront many-to-many tables and hideous GUIs. They get to play with a pretty little Smalltalk application that allows rulesets to be cloned and versioned, and individual rules to be altered or rebuilt from scratch. It typically takes less than a day to prepare a single product launch. Although an entirely new build from scratch would take considerably longer (and, frankly, would be a complete pain), it is something we have never done, because it has always been easier to begin with an existing version.

Over the years, I have often been asked to 'externalize' code; either to explain what it would do in different scenarios, or to increase code parameterization (usually removing parameters to a table). To do this, I have frequently produced tabular equivalents of procedural logic. In all that time, there has been not a single instance that I can recall where I was asked to produce an even vaguely procedural equivalent of something that was in tabular form (unless you count flowcharting decision trees). My users, at least, much prefer to see a few tables rather than pseudocode or natural language equivalents.

Despite the evident success of this approach, I am actually agnostic on the question of how BusinessRules should be held or represented. My argument would be that procedural, tabular and arboreal could be used interchangeably. I just happen not to have a mechanism to hand for automating the conversion between representations. And I just happen not to have user/GUI combination that can produce executable code or programmers who can access tree-structure diagrams. But I do happen to have users, GUIs and programmers who can all just about deal with tables. Viva la lingua franca!

Is it possible you have specially-trained power-users who have learned how to use such tables and re-use the technique for multiple customer deployments. I can see that happening in a way similar to SAS implementation experts mastering SAS configuration without having to write textual code. But I am mostly considering internal domain experts, not an IT company building applications for clients.

Some things work well as user-editable tables and some don't. Nested interweaving factors are often poor for such tables in my experience. But I otherwise agree with you that different things are probably better represented in different ways and it would be nice to have a way to automatically convert/translate between representations. That would make an interesting research project. The underlying (internal) form would probably be some kind of database, but what the user sees could code-like, tree-like, etc. depending on needs and preferences.

There is just not enough documentation, books, and tools out there to describe how to do it well for typical projects. I don't know how to do it well myself except for limited areas within the app. It is an area ripe for exploration and studies and public-inspectable demonstrations.

One compromise is to put expressions (Boolean or algebraic) into tables, or at least allow expressions when they are more effective than attributes alone. See PayrollExample. For a fancier system, perhaps there should be a tree editor for such expressions. That way they cannot be syntactically wrong. The sanctioned functions/operators would come from pull-down lists. Such expressions still might be stored as tables, but they would be represented as a tree(s) (when editing) or a textual expression when displayed.


Will this metabase have version control? So that I can see who made what changes to which rules? Will it be able to tell me what rules were in effect on a particular date in the past? How will I test new rules? How will I find out if certain combinations of rules cause infinite loops?

As far as versioning and logging would then be more like database transaction logging.

That wouldn't be good enough unless it were possible to take an individual rule, roll it back to its state at some particular point in time and carry on working. For this to work in the real world we'd need a combined database, multi-user source control system, programming language, development environment. Then we'd need to train up an entire generation of programmers capable of dealing with the complexity of a system that does everything and doesn't separate any concerns. Who would be the users of this system? Would we train business people to be adequate programmers? What are the benefits of this idea over doing things the old-fashioned way?

Maintaining business rules is a complex job. There is no magic bullet. But if code were some kind of presentation-neutral meta-data, then each could use the editors or views that they preferred (as long as version control was in place). See CodeAvoidance. But, I do think it would be relatively easy to adapt database transaction logging and tagging principles to code version control.

It's perfectly possible to use such an approach to improve SeparationOfConcerns without 'training up an entire generation of programmers'. I've seen it done in COBOL using DB2 as the source of BusinessRules. As it happens, the DB2 tables were maintained by the users using a Smalltalk application, but developers often preferred to hack the DB2 tables directly ('it's only data, after all'). The advantage for the users was that they could change the BusinessRules without any software changes (within limits, of course). The principal advantage for the developers was that they didn't have to sort out all the detailed BusinessRules before creating the code. The trick is, as always, not to overdo it. Maintaining a proper separation between actions and conditions is generally a good thing; whether there is additional benefit in pulling the conditions out of the code depends on the circumstances:

When the developers no longer need to 'sort out all the detailed BusinessRules before creating the code', the business analysts, who no longer have developers banging on their door/shoulder to get them to define the business rules, tend to think they too can proceed without defining the specific rules that apply to the business processes, and the result is a system that doesn't actually address the requirements of the business.

I don't suppose anyone has any evidence for this assertion, do they? In my experience, most business analysts will happily define business rules without being prompted by developers. Often they struggle to get people to decide what the rules actually should be, and find the "the developers need to know" stick a convenient weapon. But if they don't define the business rules in good time, their testing colleagues probably have a higher success rate in hurrying them along than developers do, especially when they're the same people or have the same boss!


One area that seems a nice fit for BusinessRulesMetabase is a "suspicious activity detector". It may detect suspicious numerical trends, suspicious transactions, and/or suspicious product combinations. Often these have to be adjusted by trial-and-error and having the programmer be around to keep tweaking the criteria is not practical. Instead, a smart business analyst can manage most of the criteria via some kind of "rule editor". The programmer may provide the building blocks, such as threshold-analysis using standard-deviation, etc., but how the combination of building blocks are applied and adjusting the threshold levels and weights is often best left to a business analyst. --top


Power-User Interface Specimens

Here are some samples of interface/interaction ideas that have been spotted in the wild:

Microsoft Outlook's in-box rule manager.

Using a "rules tree", one can use the wizard to set multiple rules to process and distribute in-coming email messages. Rough example:

It uses "all" and "any" in place of "and" and "or". One can also add "not" variations, eg: "Recipient has none of the following...". Note how it resembles an English-ified version of an AbstractSyntaxTree where the operators are usually based on SetTheory operations. There is also ranking involved in that rules "higher" in the list are executed before those lower in the list.

Product Category Table

 table: product_categories
 -------------------------
 product_ref     // foreign key to product table
 category_ref    // foreign key to categories table
 start_date
 end_date
 parameter      // value usage depends on context, such as a discount percentage
 parameter_type  // integer, double, date, text, etc.
 (Primary key: product_ref + category_ref)

table: categories ----------------- category_id category_title etc...

Although most of the actual activities are programmed by the programmer, the power-users can use categories to make fairly complex activities and conditions happen, such as making products automatically appear in a featured-product-of-the-week, timed-discounts, auto-send warning emails for certain activity on tagged items, etc. Many companies have programmers make explicit tables and screens for such features, but often they can just be handled using categories and a little extra programming that doesn't involve any new UI's being built.

Fancier versions can combine existing categories using some form of set-theory, perhaps similar to the Outlook tree.

ChallengeSixVersusFpDiscussion is also potentially an example.

ComputerProgrammingForEverybody describes the use of Lotus-123 macros by power-users; however, that may be too open-ended, resulting in the kind of spaghetti code that newbies tend to make.

http://decisions.com has graphical rule-builder tools.


See also: DataCentricThinking, PayrollExample, UserQueryInterfaces, CompilingVersusMetaDataAid, RuleBuilderInterface, QueryByExample


CategoryBusinessDomain


EditText of this page (last edited September 15, 2014) or FindPage with title or text search