Logging Toa Queue

One of the more common patterns that I've seen is that many projects have built a logging system where the destination of the log messages is a JMS queue. This provides several advantages:

The processing of the messages (logging to an archive or file) can be handled in a separate thread or process, leaving the logging thread free to do "real" work.
If the messaging system being used features guaranteed delivery the logging program is assured that every message will be appropriately logged, with no losses.
If JMS is used in a publish-subscribe way, then logged messages can be processed by many different "filters". For instance one filter may take messages of a very high priority and generate pager messages for systems administrators, while another filter might be simultaneously logging all messages to a log file.

This is more of a design idiom than an architectural pattern. Each of these benefits can be realized through other implementation choices -- for instance the first one can be accomplished in one program through a shared-memory queue and multiple threads, while the last one can be done in LogForJava through setting up multiple appenders. However, adding in a JMS element can make it somewhat easier to accomplish all of the above at once.

-- KyleBrown

LoggingToaQueue is a good example of the FireAndForget nature of MessagingSystems?. It can also be used for other tasks. As an example, the system I'm currently working on takes in requests, processes them, and returns results. For various reasons, it needs to persist the request and result for later querying. By having the persistence occur due to a message being sent, we take out the persistence cost from the request handler (and replace it with the cost of preparing and sending the message... somewhat lower, but still non-zero). -- RobertWatkins

Adding in a JMS element looks better in the resume for the next project, and I contend that is the only quality for LoggingToaQueue. Messaging is a heavyweight mechanism that should deal with serious business, while logging, even if important, should not be resolved by messaging. Otherwise LoggingToaQueue is hardly justified, especially when considering you have higher quality alternatives such as LogForJava. -- CostinCozianu

Costin, I think that you're giving an opinion without much forethought here. Sometimes logs are important. VERY important. In that case, the guaranteed delivery of a queue is well worth the weight. And I would debate that putting a message on a log in most messaging systems is really that heavyweight. I've seen this particular pattern implemented in multiple high-volume banking and financial applications where it is CRUCIAL that the application logs be as correct as can possibly be for auditing purposes. -- KyleBrown

I know, but in that case they are not quite logs like regular logs. They are integral part of the business model and the data model, they are to be treated like any other data transaction, and messaging is one possible solution. The regular meaning of logging is informative messages that might help a developer and/or administrator figure out something about the functioning of application, and usually they are orthogonal to business logic. -- CostinCozianu

No, there is no "regular" meaning of logging. Logging means different things to different people. The problem is that people will often try to apply one solution (usually writing to a file, or standard output) to all versions of the problem. Logging in general (even when it's critical to the functioning of the business) is too often treated as an afterthought, and not a central part of system design. What this pattern is saying is that when logging is important (for any reason) that sending log messages to a queue has a lot of benefits. -- KyleBrown

So, let's say then that we need to log some information, and that information might be CRITICAL. What makes you decide to use JMS instead of just inserting in a table ? -- CostinCozianu

The above three points give the reasons why logging to a queue might be a good idea. As to why use a queue rather than a table, well, primarily the information might need to be processed in several ways independently when something happens. For instance, it might be inserted into a table (as you suggest) but also recorded on a mainframe flat file, ftp'd to a remote location, or (possibly) acted upon by a human in some instances. And yes, I know that you could use database triggers to kick off some of these things -- but database triggers can't call into all sorts of programs (for instance communicating from mid-tier to mainframe) and can't do all of the kinds of processing you might need. -- KyleBrown

The question is not why to use a queue rather than a table. The question is why to use asynchronous messaging, distributed processing (also messaging), when the information we manipulate (logs) is tied to our business process and business requirements. More, the INSERT solution will be bound to a local transaction while to do transactions with queues you need a distributed transaction just to place the message in a queue (according to the JMS spec at least). The information is said to be critical, so the best a queue can do is guaranteed delivery (to another process) in case the transaction succeeds, while just inserting into a table is guaranteed persistence if the transaction succeeds.

Ok, let's accept that the business requirements ARE to send that information to a mainframe to be asynchronously processed there, although I'd be wary when business requirements are expressed in terms of technical implementations. Then we are really talking about a message queue by definition. Still not all logging activity is likely to have the same requirements. So why not use a logging framework to begin with (LogForJava or others) and in some cases one of the logging action to be taken (appenders in LogForJava terms) will send the log info to a JMS queue ? -- CostinCozianu

Personally, I think that's a great solution. In fact, that's what we've implemented in one of the cases where I saw this -- we built a custom appender for LogForJava to handle logging it to a queue and did all of our off-machine processing based on pub-sub notification from the queue. -- KyleBrown

CategoryLogging