Lisp Macro
Lisp MacroLisp macros are very different to C macros. They are a way to transform lisp code. Macros will be used in Lisp code. During a macroexpansion phase, the Lisp expression will be passed to the macro function. This macro function can do arbitrary computation at macroexpansion time. The result of this call has to be again Lisp code. This Lisp code then is what the interpreter or compiler sees and executes or compiles.
So a typical sequence in a toplevel READ-EVAL-PRINT loop looks like this:
1.) read a Lisp expression. Input comes from a stream and the result is a Lisp expression (interned Lisp data).
(read) -> you type "(incf x)" , Lisp reads a list with two symbols.
2.) macro expand this Lisp expression if it is a macro form. the result is a new Lisp expression (interned Lisp data)
(macroexpand '(incf x)) -> some new Lisp expression
3.) compile the Lisp code
4.) execute the compiled code
5.) print the results
Crucially, the macro language is Lisp itself, so the full power of the language is available. Contrast this with C macros, where a separate language (which doesn't know how C works) is used.
There are few other techniques that can do what Lisp macros do, and although most of the time you don't need them, there are occasions when you really need them, and there isn't much of an alternative.
Trivial example:
(defmacro eight () (+ 3 5))If I use this macro in an expression,
(eight)it is equivalent to using 8 itself, because the computation is already done at compile time. However, this is easily accomplished by the C preprocessor.
Now consider the similar macro:
(defmacro eight () '(+ 3 5))This expression is quoted, so now whenever I invoke
(eight)it will return
(+ 3 5)which is then evaluated (at run time) to yield 8. This is similar but crucially different from the first example, and this difference could not be achieved using regular functions. Think about this difference in the context of calling rand() rather than using the constant '3'.
Also the macro can return anything it wants, including fragments of programs, so it can be used to build up pieces of programs at compile time - it can generate code. It thus can be used to extend the language. This is where the real power of macros is, and it's also the most difficult part to explain.
Languages that do not support Lisp-style macros in this sense must resort to writing programs that generate programs which are run and whose results are then incorporated into yet another program, etc. I've done this many times in C, and it's a pain.
As a secondary benefit, Lisp macros can be used to do a variety of relatively less important things like add syntactic sugar, but this is trivial compared with their full power.
There are comments here about what you cannot do with them, such as that you cannot use them with apply in Common Lisp because they are not functions. That is an accident of history - Common Lisp handles functions a little strangely in general; in some dialects of Lisp, as long as the macro happens to return a function as a result, you can use the macro with apply, because apply ends up seeing a function.
The fact that you can't use macros with apply in Common Lisp and in most Scheme implementations confuses understanding of the issue, but isn't essential to the ultimate nature of macros. Many, many kinds of macros with a variety of strengths and weaknesses have been proposed and implemented, but if they don't involve doing computation at compile time, then they're not "Lisp macros".
The non-Lisp world (e.g. "macro assemblers", "C macros") usually mean something similar but slightly different with the word "macro": they still mean something that happens at "compile time" or the equivalent, but something that works by string rewriting. Interestingly, unrestricted string rewriting is Turing Complete: you can calculate any computable function by string rewriting. However, C macros, for instance, are crippled in several ways, and other more powerful macro systems like m4 and some assembly macro systems are pretty dangerous to work with precisely because they operate within the domain of strings, rather than on program representations, making Lisp macros a unique combination of power and relative safety and convenience. -- DougMerritt
LispLanguage includes an extremely powerful (TuringComplete) macro facility. Via macros LispLanguage allows the user to adapt the language to their problem domain and in its ultimate expression, turn LispLanguage into a DomainSpecificLanguage (or LittleLanguage) customized for their task.
This page is for discussing how and when to use macros and for examples of macro use, see DoWeWantLispMacros for whether or not they should be used.
Uses of Macros
Macros are/should-be used for these reasons:
## this is only a limited view on macros. See my comment on macros and CLOS later on this page.
Macro Systems
There are three main macro systems in use:
Since I know that I use macros for deep, meaningful stuff, here are "prototypes" for the ones I consider interesting:
(def-synonym-function synonym original) (def-pipe-function function-name (input-stream output-stream &rest other-args) &body body) (def-enumerated-value-domain control &rest data) (let-gensyms symbols &body body) This one is a very common idiom, except it's usually called something else. (struct-comparison-predicate struct-name reducer-name &rest fields) I need to rework the interface to this one. (successive-character-predicate-parser input (&body predicates) &body body) And even more so for this one.Three of these are convenience utilities; one is an "inconvenience utility" which I don't actually use anymore. One concisely defines a whole set of functions and symbols which relate to each other, in a manner which I very often want. The remaining two are nano-languages to simplify the writing of certain sorts of code. Most of these, I use quite often. I could expound the above at great length, but I expect it's not necessary to answer your question. -- DanielKnapp
Other common macros are:
(when-bind var condition &body body)bind var to the result of condition and evaluate body if the condition returned true
(with-collectors (collectors &body)COLLECTORS is a list of symbols, for each element of COLLECTORS a new function is introduced in BODY, these functions collect their arguments. WITH-COLLECTORS returns all of the collected arguments as multiple values. ie:
(with-collectors (a b) (dotimes (i 10) (if (oddp i) (a i) (b i)))) => (1 3 5 7 9) (0 2 4 6 8) ;; two valuesUsing macros for resource management:
If you have a scarce resource that you need to manage the lifecycle of explicitly instead of tying it into a GC finalization, then the approximate equivalent of the try/finally code on FinalizeInsteadOfProperDestructor is
(let ((r (allocator-for-my-scarce-resource))) (unwind-protect (progn (foo) (bar) (etc)) (deallocate-my-scarce-resource r)))(UNWIND-PROTECT means "evaluate the first argument. When you stop with the first argument, for whatever reason, also run the subsequent args" - if you like, "try" the first arg, and "finally" do the other args)
You can't depend on some client programmer always remembering to do this, so you make it easy for them. Write a macro, which would probably be called WITH-SCARCE-RESOURCE in this case, that wraps the unwind-protect so the user never has to remember
(defmacro with-scarce-resource ((name) &body body) `(let ((,name (allocator-for-my-scarce-resource))) (unwind-protect (progn ,@body) (deallocate-my-scarce-resource r))))so now we just have to do
(with-scarce-resource (r) (foo) (bar) (etc))
Can someone explain to me why Lisp macros are better than, say, the C preprocessor? (I don't know much about Lisp, and I suspect that Lisp macros are much more elegant; I just want a concrete explanation from someone who knows what he's talking about.) -- AdamSpitz
Sure. In C, the best you can do is textual substitution. This allows you do to things like
#define MAX(x,y) ( ((x) > (y)) ? (x) : (y) )taking great care to put in all those extra parentheses. That's about the level that I, personally, can "rewrite syntax" using the C preprocessor. Note that you do not have access to C at this point, only to this mini-language provided by the preprocessor. It allows you do do things like splicing tokens in, with our without the double quotes, etc. But that's about it.
In Lisp, your macro can perform any computation on its arguments whatsoever. It can decide which arguments to evaluate, and whether they should be evaluated zero times, once, or any other number of times. If these arguments have side effects, this could be important. This features allows you to add first-class syntactic elements to the language, i.e. elements which are on par with the built-ins of the language. (In fact, many of the built-ins are macros, e.g. SETF, WITH-OPEN-FILE, etc. The number of special forms, i.e. those who have special cases for the lisp interpreter/compiler, is actually very small in lisp, and many are obscure low-level things not used very much in day-to-day programming. The obscure ones include BLOCK, GO, PROGV, SETQ. The well known ones are LET, PROGN, IF, UNWIND-PROTECT etc.
What do typical, real life macros used by lispers look like? they may look like this:
(map-over-active-allocators (alloc :continue-on-error t) (log-message :info "~&Notifying allocator ~A" alloc) (my-remote-operation alloc))Which might be defined as follows:
(defmacro map-over-active-allocators ((var &key (continue-on-error t)) &body body) "Execute BODY over each active allocator in a context where VAR is bound to each one in succession. If BODY raises an error, the error is ignored and the form is attempted over the next allocator if CONTINUE-ON-ERROR is T, otherwise, the error is re-thrown and BODY may not be executed on all allocators" `(loop for allocator over (currently-active-allocators) do (let ((,var allocator)) (handler-case (progn ,@body) (error (err) (when ,continue-on-error (error err)))))))This code omits some issues of hygiene (e.g. continue-on-error is evaluated multiple times) for pedagogical purposes.
There - that help? :-) -- AlainPicard
Beautiful. ThankYou. :) -as
Also see CeePlusPlusTemplatesCommonLispMacrosComparison.
I've been thinking a lot about macros ever since I asked my question here last year. I think I understand them a lot better now, and part of my brain recognizes them as a really elegant idea - but I'm still having trouble coming up with situations in which I'd want to use them.
I should probably explain where I'm coming from. I'm a Smalltalk/Ruby guy, and I'm trying to imagine what Ruby-with-macros would look like. And I can do that - I can imagine a language that looks like Ruby, has Ruby's object system, and has Lisp-like macros underneath - but I'm having trouble coming up with examples where macros are really a big win. In a few cases, macros seem like they might streamline a bit of syntax (usually by allowing me to write a "naked" Symbol), but... I dunno. It seems like macros ought to be useful for more than that.
After thinking about them a bit, none of the examples on this page are particularly compelling - they all seem to me like workarounds for the ugliness of typing "lambda". But Smalltalk and Ruby both have a much more lightweight syntax for closures. I can write an equivalent to your with-scarce-resource macro using just functions and closures:
def with_scarce_resource(&body) {
resource = allocator_for_my_scarce_resource()
try {
body.call(resource)
} finally {
deallocate_my_scarce_resource(resource)
}
}
(That's not valid Ruby code; it's just the syntax that I'm imagining for this new language. Imagine that the try__finally function is the equivalent of your unwind-protect - it's a function that takes two closures, starts executing the first, and executes the second when the first stops for whatever reason.)
So now I can write:
with-scarce-resource { |r|
foo()
bar()
etc()
}
Point is, I didn't need macros to do what you did in the Lisp version, and you didn't need them either - you could have done everything with normal functions and lambdas, except that "lambda" is ugly to type, so you wrapped it in a macro. And it seems to me that the map-over-active-allocators example above is similar.
So my current theory is that lambdas are so fundamental that they deserve a special lightweight syntax, and that once you have that, you don't need macros so much anymore. But there must be other things that you use macros for. What are they?
-- AdamSpitz
I _highly_ recommend On Lisp by Paul Graham and Let Over Lambda by Doug Hoyte. However: As you rightly point out, usually macros are just shorthand for lambdas (or more interestingly, closures). This is in some ways due to the inherent expressive power of the lambda calculus. As a first response, macros can help clarity; I routinely write ease-of-definition macros for special types of functions, etc. However, there are places where macros really save time: creating context and compile-time calculation.
Functions suffer from the fact that they require lexically-scoped arguments to be supplied explicitly, and as we know today dynamic/global variables have many drawbacks. Context creation combines the two: Variables are not global and you don't have to supply them as arguments everywhere in your code.
Compile-time calculation can be used for everything from optimization to automatic code generation.
Examples:
In some of my physics analysis code, I use functions which keep track of their dependencies and only run if necessary, i.e. if function-b calls function-a somewhere, then whenever function-a is set to re-run function-b automatically knows to re-run as well. This is useful when you have long chains of computation and update the data source at some point; you just specify the most basic function(s) which need re-running and call the master function, only the parts which need to be re-run are executed. This is a blend of compile-time calculation and optimization.
Like aif:
(defmacro aif (test then &optional else)
`(let ((it ,test))
(if it
,then
,else)))
It violates referential transparency, but we gain in essence a new special term "it" which is bound to whatever the containing aif set it to. The fact that language has pronouns is a big hint that often times it's better to express ideas within a context than to insist that everything be understandable via substitution.
Example:
(aif (member x some-list)
it
(print "x is not in some-list"))
And you can do this for any conditional construct you can think of, e.g. awhen, acond, etc.
Another use for anaphoric macros: Ever wanted to write a recursive anonymous function but instead were forced to add a labels to your code? Try alambda:
(defmacro alambda (lambda-list &body body)
`(labels ((self ,lambda-list
,@body))
#'self))
If you make heavy use of functions then you probably are prone to using recursion, so alambda may find quite a few good uses.
Let Over Lambda has even more examples (these are from On Lisp), but I won't bog down the page with them.
Gary Hollis
There is a language which is almost ruby + macros, namely IokeLanguage. A lot of IokeLanguage's most useful features are written using macros: see http://ioke.org/dok/index.html for examples. I know this doesn't answer the deep question AdamSpitz is asking ... except maybe in one way, which is that macros let you choose between evaluating their arguments and not (in IokeLanguage anyway). See also RobRix's comment on how this works in IoLanguage, below. -- JasonGrossman
Cool. Thanks. Lemme think about these for a while. :) I'm sure I'll be back with more questions soon. In the meantime, if anybody's got any more, I'd love to see them. The more examples I see, the more likely I am to be able to convince myself that macros are worth having. :) -- AdamSpitz
I started learning scheme, and it seems that macros are good especially to define new special forms, where you can control the order of evaluation of your arguments. For example, you can create a curried-lambda macro, which is just like lambda, but returns a curried function instead. Or if you want some syntax to ease the working with lazy-evaluated data, or something like that. -- AmirLivne
After some more thinking, I think I've convinced myself that I don't want a language without macros - but I also have this feeling that every time I need to use a macro, it's a failure of the language.
In reading about Lisp macros, I've seen several people give the advice: "Don't use macros unless you need to - if something can be done just as easily as an ordinary function, it should be." I'm guessing that that's because macros are harder to understand than ordinary code - they force you step outside the "rules" of the language, to think at the meta level. It's easier to reason about something when the rules don't keep changing under you. (Would that be a good guess? Do you agree with this policy of avoiding macros except when necessary? Why?)
So what I want, I think, is a language that has macros - but where you would almost never need to use them. In fact, every time I DO use them, I want an alarm bell to go off in my head saying, "Is there something I could add to the language to make this macro unnecessary?" I'm not about to start adding features to the language all willy-nilly, but if there's a simple rule that I can add to the language to drastically reduce macro usage, I think that's worthwhile. The obvious example is the "hide a lambda" kind of macro seen above - if Lisp had a more lightweight syntax for lambdas (like Smalltalk and Ruby do), all of those macros would just go away, and I think that the language would be better off for it.
I don't quite believe everything that I'm saying. :) Part of me wants to say, "Well, if macros solve everything, then why replace them with mechanisms that solve less-than-everything?" But I still have this feeling that macros are more complex than ordinary code, and that the less I need to use them, the better.
I'm not experienced enough with Lisp to have any idea whether I know what I'm talking about, though. :) So what do you think?
-- AdamSpitz
Adam, one of the reasons for which it is recommended to use functions when you can, instead of macros, is because functions are a fundamental type, whereas macros "disappear" after read-time. Thus you can not MAPCAR over a macro, pass it as an argument, etc. Typically, you use functions to solve your problem, and you use macros to express your problem. -- AlainPicard
That makes sense, and it's one of the reasons that I'd like to lessen the need for macros. More on that below. -- AdamSpitz
After thinking about them a bit, none of the examples on this page are particularly compelling - they all seem to me like workarounds for the ugliness of typing "lambda"
To put it differently: "none of these programming languages are particularly compelling - they all seem to me like workarounds for the ugliness of assembler" -- MarcoBaringer
<grin> I know, I know. :) But are you sure that macros are the right workaround? -- as
One compelling use of macros in lisp can be found in the CLSQL library. What they did there was to embed a pseudo-SQL directly into the language, which compiles to calls to the underlying database (it supports several different SQL servers, including Postgresql). Some example code:
(select [email] :from [users] :where [= ["lower(name)"] (string-downcase customer)])This eliminates the problem of having to represent SQL as strings, which can lead to SQL-injection vulnerabilities. I think this solution is far more elegant than string-based parameterized queries in other languages, and it's way better than using an ORM framework.
One time, I had to write DB code for a customer that was using a database with an absolutely horrible proprietary query language, which was based on XML. The names of fields had to be looked up to get a corresponding field number, which was what you had to use in the query. Ditto for the table. You could write a whole page of code for one query.
They provided a framework in Ruby, but it was nothing more than a direct transliteration of the XML API. Using Lisp macros, it only took a day to boil this system down into a near-SQL macro language similar to CLSQL's macro language, which prevented the project from accruing technical debt.
Macros can save you from far more than just the word "lambda."
After thinking about them a bit, none of the examples on this page are particularly compelling - they all seem to me like workarounds for the ugliness of typing "lambda"
Well, yes. But then again, the CommonLispObjectSystem is largely a set of macros and functions, and it sure saves a lot of typing of lambda. Don't underestimate the power of saving typing, i.e., of a powerful notation. -- AnonymousDonor
## Okay, CLOS is an interesting topic in the light of macros.
So, why does CLOS use macros and how does it use macros.
First, CLOS is *not* a set of macros and functions.
CLOS is defined in terms of CLOS. At the base, CLOS is written in CLOS itself. Especially if you have a MOP-based implementation. So, the lowest level of CLOS is generic functions, methods and classes. On top of that, a functional level is defined. And again on top of that, macros are defined.
An example: Class definition.
At the lowest level you create an object of type standard-class:
(make-instance 'standard-class :name 'foo)
The functional level is represented by:
(ensure-class 'foo)
The macro-level then is:
(defclass foo () ())
So, why does CLOS have this three level architecture and why does it use macros at the top level? Well, the implementation of CLOS in CLOS enables the software developer to change and extend CLOS via the usual CLOS mechanisms (subclassing, writing methods, ...).
The macros are used, because they allow to give the Lisp developer a familiar surface syntax. Plus, and that is a big PLUS, macros are making it possible to use load- and compile-time side-effects.
If a Lisp compiler sees (ensure-class 'foo) the compiler itself has no special knowledge about that function call. It is just an ordinary function call. So, all the compiler will and can do is to create the byte- or machine-code for that function call.
But if the compiler sees (defclass foo () ()), the macro will be expanded (!) at compile time and the resulting (!) code will be compiled to byte- or machine-code. Since the macro code for DEFCLASS is run at compile time, the writer of that macro can execute arbitrary code at that time and he can generate arbitrary code.
For what is that good?
Typical examples are: DEFUN, DEFVAR, DEFMETHOD, DEFCLASS, DEFMETHOD, DEFSTRUCT, DEFINE-CONDITION, DEFPACKAGE, DEFTYPE, ...
To see what your Common Lisp implementation does, just try:
(pprint (macroexpand '(defclass foo () ())))
Example:
Welcome to OpenMCL Version (Beta: Darwin) 0.13.2! ? (pprint (macroexpand '(defclass foo () ()))) (PROGN (EVAL-WHEN (:COMPILE-TOPLEVEL) (CCL::%COMPILE-TIME-DEFCLASS 'FOO NIL)) (CCL::%DEFCLASS 'FOO '(STANDARD-OBJECT) NIL NIL NIL NIL 'NIL) (CCL::RECORD-ACCESSOR-METHODS 'FOO 'NIL))So, macros are especially powerful, because at compile-time or load-time the whole power of Common Lisp is available. And this is not just only for writing clever functions transforming code (like the LOOP macro), but also for larger facilities like CLOS who need to be integrated tightly into the Common Lisp system and its development environment.
For the necessary understanding, you might want to read the book TheArtOfTheMetaObjectProtocol (Kiczales, des Rivieres, Bobrow). The code shown in that book is just beautiful and everything seems to just fall in its place.
Oh, I agree. I'm suggesting that hiding lambda is such a common use of macros that it deserves to be made a first-class part of the language.
Take the with-scarce-resource macro described above, for example. Because it's a macro, it disappears after read-time and can't be passed around like an ordinary function could. Of course, with-scarce-resource could have been written as a function that takes a function as an argument, but then you'd have to type "lambda" every time you want to use it. So every time you want to do things with lambda, there's a decision to make - do you do it the pretty way, or the flexible way, or both?
In Smalltalk, you create a closure simply by writing the code inside square brackets. The syntax is easy to type, easy to read, and doesn't get in the way. The Smalltalk version of with-scarce-resource would be just as pretty as the Lisp version, and it would be an ordinary method that would still exist at runtime.
But the lambda-hiding thing is unusual, because it's so common. Most uses of macros, I'm imagining, are more application-specific, and aren't important enough to be worth adding more features to the language. For those, I'd be glad to have macros. It's certainly better than living without the macro and typing out a bunch of duplicated code.
So I suppose I'm saying that I think Smalltalk is flawed because it doesn't give me macros as an escape hatch, but I think Lisp is flawed because it relies too heavily on that escape hatch. I'm predicting that there's a sweet spot in the middle somewhere - a place where most of my ideas can be expressed directly as first-class elements in the language, and the remainder can at least be done through macros to avoid duplication. And the sweet spot is a moving target, because the more macros we write, the more patterns we'll notice and eliminate.
Does that make any sense?
-- AdamSpitz
You are making the same argument as the Scheme community WRT lambda-hiding. In Scheme, whenever you see something like "with-output-to-file," instead of being a macro like in Common Lisp, it'll be a function that takes a lambda as an argument. There are very few cases, such as "define" and "let", where the lambda is hidden.
The developers of the Racket language came up with one solution for the ugliness of typing "lambda": They added a macro, λ, which is equivalent to LAMBDA, and made an IDE in which CTRL+\ was bound to this character.
In Common Lisp you could define a read macro that dispatched on square brackets (the definition is omitted cos its tricky and I'm at work :)and expanded into a lambda.
So you could write
(defun with-scarce-resource (fn) ;;; warning. This code is untested,
(let ((resource (allocate-my-scarce-resource))) ;;; don't use it for writing air traffic control software
(unwind-protect
(funcall #'fn resource))
(deallocate-my-scarce-resource resource)))
So now I can write:
(with-scarce-resource [ (r)
(foo) (bar) (etc)])which will expand into
(with-scarce-resource (lambda (r)
(foo)
(bar)
(etc)))
And I don't have to change the implementation and this will work across all compliant implementations out of the box. This is kind of the point of macros. If something's broke, you don't have to appeal to the language designer to fix it.
Often the "best" use of macros involves having most of the work being done using functions. The functions can encapsulate most of the "work;" the macro then provides a convenient way to "manage" that work.
- You don't need macros for that. Functions are just as good. CLIM for example uses both.
instead a macro:
(with-open-file (stream "ccl:examples;test.lisp") (read stream))
use a function
(opening-file "ccl:examples;test.lisp" #'READ)
It is just as good. Implementation:
(defun opening-file (pathname thunk) (let (stream) (unwind-protect (funcall thunk (setf stream (open pathname)))) (when stream (close stream))))Actually, I much prefer using functions like that. I can change by them by replacing them with new definitions and it will have immediate effect (unless they were inlined). If I were using a macro and the macro implementation changes, I have to recompile every code that uses that macro. Bad!
-
For instance, it makes perfect sense to have functions for OpenFile? and CloseFile? and such. It is then nice to have a WITH-OPEN-FILE macro that allows the programmer to not need to worry about explicitly opening and closing the file.
There's a binding for Guile for PostgreSQL; it consists of a whole horde of functions for opening connections to databases, creating queries, accessing results, as well as for creating, accessing, and disposing of database cursors. Unfortunately, it exclusively consists of functions. It doesn't have any "WITH-DATABASE" or "WITH-CURSOR" or "WITH-SELECTION" or "WITH-TRANSACTION" macros; you're left managing that sort of thing by hand. A major value of Lisp-style macros is in allowing you to hide all the irritating "create/dispose" work.
- see above
It is _possible_ to handle this by passing around functions; it's rather nice to not be forced to plug pieces of program into lamba functions just because that's the convenient way of using some "convenience" functions. The "WITH-SELECTION" function lets you write the program the way YOU want to.
The excellent book OnLisp thoroughly explains macros - why they're useful, how to write them, etc.
Also of interest: LispMacroDiscussion
When I correctly understand what you guys are trying to say, macros are mainly used to get rid of quotes and to help the compiler generate better bytecode? So they are nearly useless in a interpreter which doesn't generate byte code but immediately evaluates source code. Is that correct??? -- Bruno
From reading OnLisp, i get the impression that while lots of macros are written just to escape the lambdas (the SchemeLanguage style is to use explicit lambdas), some of them really need to destructure the code, and have to be programmed as macros. It has nothing to do with bytecode generation. -- AnonymousDonor
Correct: Lisp has inline functions if all you want to do is to remove function call overhead. The point of macros is to (1) allow you to tune the language to suit the domain (think of it as like embedding a domain-specific language into Lisp), and (2) to provide another way of encapsulating frequently-used patterns for concise use. Some uses of both kinds can be kinda-sorta mimicked using functions instead of macros, by wrapping stuff in lambda forms, but that gets verbose and ugly very quickly. By the way, Bruno, almost all Common Lisp implementations compile to native code. (In a couple of cases, via C.) Only one compiles only to bytecode, and I don't know of any that are only interpreted. -- GarethMcCaughan
Is there a form of reflection which could make functions as powerful as macros? Something along the lines of "anything you can do with CodeGeneration, I can do with ReflectionMagic?"? Or is reflection strictly not as powerful as macros?
Also, for a technique which can eliminate 99.99 percent of those troublesome macros... see CutAndPaste (works with functions too!)
That's just plain wrong. CutAndPaste (AndSubsequentEdit?) certainly is a common way to program, and can substitute for non-recursive functions. But CutAndPaste cannot substitute for recursive functions, and in no way can it be used to do what Lisp macros do: compile-time computation.
Hardly... it just means that the programmer is the compiler. In some ways, it could even be considered to be better... instead of performing expensive calculations which extend compile times, we get all that non-sense done with at entry-time :)
Nay nay. What you said is as true or as false as the notion that functions are unnecessary because the programmer could be the computer and just do the whole computation himself. In principle that is of course true, but in practice, as we know, computers tend to be handy. Same thing with macros. Macros are merely functions that are evaluated at compile time, as I recently discussed at the top of this page.
Doubtless you are thinking of trivial syntactic sugar macros, which can largely be done as easily in e.g. C lexical macros as in Lisp semantic macros. For those trivial uses, sure, CopyAndPaste your heart out; those kinds of macros aren't all that important. But neither are they what this page is (theoretically, amidst the confusion) about. -- DougMerritt
I don't want to turn the summary at the top into another ThreadMode mess :), but I think there's a misunderstanding in there about my complaint about the non-uniformity between macros and functions. It's got nothing to do with Common Lisp's strange function-handling; I'm talking about using "apply" on the macro itself, not on the result of the macro. (I tried to explain my position on the LispMacroDiscussion page, in the part about whether to implement "square" as a function or a macro.) I really am talking about problems that I perceive in the fundamental nature of macros, not just the accidental CL weirdness. -- AdamSpitz
I know, I did read that. But no, it is precisely the strange function-handling. Try (defmacro a () ()) and (apply a foo). Nope, variable a has no value. Well, of course not, in Common Lisp you'd have to do (apply #'a foo). Nope, #'a is illegal because a is a function, not a macro.
Hmm...let's do it with functions first:
(defun twelve () 12) (apply twelve 1) ...*** - EVAL: variable TWELVE has no valueOops, there's that weirdness, "twelve" isn't a variable bound to a function, it's in a wholly different namespace; try it again:
(apply #'twelve 1) ...Lisp responds 12, ok, that worked. But we can't use #' with a macro because Common Lisp will arbitrarily say "no, that's a macro, not a function"; it won't even try.Well, let's make it happy:
(defun twelve () 12) (defun my-macro () #'twelve) (apply (my-macro) 1)...and now Lisp is happy; my-macro was expanded to #'twelve, a notation that Common Lisp understands is a function, and apply happily uses that and gets the final result 12.
So it's not impossible to use macros with apply even in Common Lisp, it's just that apply wants to see a functional parameter by the time it's invoked. -- AnonymousDonor
Um... I think you're still confused, and I don't know what I can say that will clear it up.
I think we can clear things up if we take them one step at a time.
I'm talking about using "apply" on the macro itself, not on the result of the macro. Notice that in your example, you're saying (apply (my-macro) 1), not (apply my-macro 1).
(Now you're thinking, "But (apply my-macro 1) doesn't even make any sense!" That's exactly my point.)
Well, yes, but never mind macros for a second, when does it ever make sense in Common Lisp? Never. Not for anything (unless I'm getting my Lisp dialects all mixed up and have a mental block, of course). Which is why I keep babbling about functions being treated oddly in Common Lisp. In some Lisp dialects it makes sense to say (apply name 1), because what happens is that the variable "name" is looked up, it turns out to refer to a function, that function is passed to apply, and everyone is happy.
Now, looking at it the other way around, in any dialect of Lisp you can imagine, what kind of parameter should apply accept? We know what apply means when it's called with a function. Does it mean anything to apply non-functions? How about (apply 1 1)? Doesn't make sense to me, but that doesn't mean that integers aren't first class! It also doesn't make sense to pass a comment to apply. Very very few functions are total in the technical sense, and that's probably just as well. Try to add 1 to a function. It usually doesn't mean anything, and we probably don't want to come up with a meaning for that. It's just illegal.
So back to macros. When does it even mean anything to pass a macro to apply? Only if apply sees a function. Considering that macros, by definition, are evaluated at compile time, when could their result be a function? Well, anytime that's the way that they were written. What if they expand to e.g. an integer instead? Then you end up with (apply 1 1), which doesn't mean anything. Or what if the macro evaluates to nothing at compile time, as is the case with comments? Then you get (apply 1), an error.
So it's not that macros are 2nd class, it's just that you've got to ponder what the meaning would be of an attempted operation with them.
The fundamental question here is: "How do you decide whether to implement an operation as a function or as a macro?" For example, should I write (defun square (x) (* x x)) or (defmacro square (x) `(* ,x ,x))?
And the answer is: "Always implement it as a function if the calling code will look good enough." So we'd implement "square" as a function, because (square 3) looks the same as (square 3) - making it a macro doesn't make the calling code look any better. But, for example, "if" should be implemented as a macro, because (if a b c) looks better than (if a (lambda () b) (lambda () c)).
I would put it more strongly. Always use functions, period. In the Lisp community, typically it is not the case that application programmers use macros. It is usually avoided and discouraged. But sometimes you really, really need one, and you then seek out the macro programmer in the group and ask them to provide one that is then available to everyone.
It's similar to compilers, perhaps. When should you write a compiler? For most programmers, never. Don't. But if you really have to for some reason, it won't matter that people told you not to - you'll do it if it's just plain necessary. The rest of the time, don't. Macros are code-generators, so they're like micro-compilers, so that's not all that fanciful of a comparison.
If we had written "square" as a macro, we wouldn't be able to write (mapcar square '(1 2 3)). That's what I mean when I talk about not being able to 'apply a macro. (And if you wrap the macro in a function, you lose the delayed-evaluation and the macro was pointless.) Macros in Lisp are second-class citizens - that's why we prefer to write things as ordinary functions whenever that's good enough. Because there are things you can do with runtime constructs that you can't do with read-time constructs.
Wait, this is exactly the way it should work. Ask yourself, do I want to square each element of the list at runtime or at compile time. If you want to do it at compile time, you don't want mapcar. If you want to do it at run time, you don't want macro-square.
No - and this is why it's the macro expert in the company that writes macros, not just everyone; doing them correctly is a specialty that is different than the skill of writing functions.
The "correct" way to write square depends very much on how it will be used. Almost always, it should be a function. If heavy syntactic sugar is needed, then ok, it should be a macro - one that expands into a function! And then that function can be passed to apply.
When should you write macros that don't expand into functions? Never! :-) Except sometimes you have to, and you'll know when that happens because you won't have any choice. Then you go get the macro expert (or become one yourself, of course). They come up when you write a new mini-language, for instance, and as everyone knows, implementing a new language, even a little language, shouldn't be done quite as casually as writing a new function.
Ok, I'm exaggerating to some extent, but look at the examples someone else wrote above, such as def-synonym-function - it's not defmacro, it's a wrapper that creates a macro in a careful controlled way. That's the common approach to more casual creation of macros - don't use defmacro directly, because you'll cause yourself grief, and you get into the usual problems with creating a new language others can't maintain. Instead, create a limited facility for your own convenience. This, too, can be abused; there are a lot of important programming style issues here.
This isn't so different than shell programming. It really annoys me when I have to use someone else's shell session, and they've used aliases or whatever to redefine every command under the sun, so it's as if you're working with somebody's private system, instead of the standard one.
Stylistically, the usual answer is, don't do that. But everyone will, once they understand the mechanism. But still, if they go overboard, they will be criticized, just as any bad programming practice will be. Macros should be used very judiciously.
The only answers I've gotten that make any sense to me are:
Does this help at all?
Yes, but if you're still not happy with my new answers, then just tell me this: what do you think it could possibly mean to apply a macro that doesn't expand to a function? Tell me that. If there's a possible meaning then there's likely a way to do it. If there's no possible meaning, then it's a moot point!
Just for argument's sake, what about... (or something, my knowledge of lisp is very limited...)
(defmacro foo (name) `(defun with-,name (x) ,(cond (x) ((file stream)) ((t) (`(print "Doing something to a ",name x)))) (do-something-complicated-with-,name x) (do-something-else-complicated-with-,name x) (do-something x))) (mapcar foo (file stream database))How about this? It still doesn't allow macro application though. -- Khumba
(defmacro macromap (macro &rest arglists)
(let (result)
(do ((rest arglists (cdr rest)))
((null rest) (cons 'progn (nreverse result)))
(push `(,macro ,@(car rest))))))
(macromap foo (file) (stream) (database))
If we ever reach a consensus, I want to blow away all of the ThreadMode junk on these macro pages and just write a summary. :) -- AdamSpitz
Agreed. -- DougMerritt
One thing to consider about macros is their synergetic nature: just like FirstClassFunctions?, they're not a KillerFeature? or a SilverBullet as such. They're valuable because of the way they interact with other language features. In and on themselves they "just" provide SyntacticAbstraction and a ProgrammableCompiler?: without macros you cannot build new syntaxes for your DomainSpecificLanguage, without macros (and CompilerMacros?) you cannot tell the compiler how to optimize specific idioms. -- NikodemusSiivola
First class functions aren't a killer feature? Why not?
Because there is no such thing as real KillerFeature?? Because anything you can do with them you can do without them? It's partially the good old TuringTarpit, and partially a synergy thing: feature X may be nice anyways, but to truly shine it must be harmonious with the rest of the language. One way to put it (FP people will probably disagree) is that without macros functional composition can become a form of ThreeStarProgramming; multiple levels of indirection with only weak indication of the actual semantics. With macros you can build the syntactic abstractions to deal with this, so that the semantics become manifest in code. Besides, first class functions are only half the story: LexicalClosures are much closer to being a killer feature (just nitpicking ;-).
I think, generally applying a macro does not make sense. Many macros do some kind of destructuring, treat their parameters specially or what have you. If you enable apply, mapcar, etc., on macros, they will fail more often than they'll work. Each macro is special. If you want to "mapcar" the macro foo from the above example, you can do it, by writing a trivial macro. It can even work with the whole class of macros similar to foo, but it still won't work for all macros. -- IvanToshkov?
I think this is getting far more complex than it needs to be. Macros are confusing to most folks because they're quite alien to modern programming languages. They're not really that scary or complex (as some would claim). They're a very large, powerful gun. You don't use them to swat flies. Let's look at what macros are really for:
The problem with this list is that it has to be taken as a whole, not one spoonfed bullet at a time. A lisp macro that, say, allows you to map over a list 3 entries at a time (like in Paul Graham's ANSI Common Lisp) gives you 1, 2 and 4 from that list. Clearly, it's a win. If you check out the macros used in the UnitTest suite in PracticalCommonLisp by PeterSeibel, you will see that he manages to hit all 4.
The whole idea of macros is to create cleaner, faster code. It's cleaner because you don't have to waste time explicitly passing anonymous functions (to prevent immediate evaluation of arguments). It's better abstracted because you can now build even more complex code off these structures as if they were primitives. It's got a shear zone because the code can be written without a care of what the macro is doing (save for the constraints the macro specifies). It's faster because, unlike say, Ruby, you don't incur the overhead of a function call and associated scope creation.
Also, macros allow you to establish scopes and variables. Check OnLisp's section on Anaphoric macros. As an exercise, try writing a macro that deliberately captures variables. Now, try it in Ruby with Ruby's odd scoping and lambdas. :)
As for the issue of "apply"; macros and functions are different. So different, in fact, that they even exist in different namespaces. In a hypothetical language, I suppose they could be the same thing, but the assumption of a macro is that it will generate code. That's what really differentiates compile-time macros from run-time functions. This means that for you to run (apply my-macro '(1 2 3)) your lisp environment would have to generate a list of expressions and evaluate them, collecting their results into a list. Sure, you could do it. Does it really make sense? Does it really validate the use of as powerful a hammer as macros? Sledgehammers are useful, but they're heavy and awkward compared to hand tools. Take the path of least resistance.
I think comp.lang.lisp mentioned the define-compiler-macro when they talked about this. If you're looking just to do optimizations on certain functions, http://www.lisp.org/HyperSpec/Body/mac_define-compiler-macro.html will show you how. This is a more targeted tool for the job than basic macros.
If you just have to have a macro that's applicable (which is fair, I suppose it's possible), then try defining a macro that expands to code that does something and a function that actually does it. Now when you say (apply #'my-macro '(1 2 3)) you get the function argument.
I hope this wasn't too redundant. :) -- DaveFayram
Okay, I've read this far, and I still have (almost) no clue what a lisp macro is. Where do I go from here?
Also, I tend to avoid C macros, not because they are too powerful or that they are awkward to read/write/maintain, but because they make debugging a PITA, since they are really just a code generator, but debuggers typically show you the original code, not the generated code (so you can't step into a macro). Does the same pitfall exist with lisp macros? If not, what fundamental part of their nature is responsible?
-- PaulBrannan?
Well, you could read any of the numerous introductory texts on Lisp, either books or what's on the Web. Some are referenced on this page and elsewhere on C2; also see the Common Lisp wiki, http://cliki.net.
But I'll try to succinctly describe a lisp macro for you: It's a Lisp function which takes as parameters pieces of Lisp code, and returns a new piece of Lisp code. The "pieces of Lisp code" are represented as Lisp data that represents s-expressions; it's a fundamental feature of Lisp that code can be represented this way. Macro calls look just like function calls in the textual code. They differ from functions in that the Lisp evaluator or compiler calls the macro immediately during processing, and then processes the results returned by the macro as if it replaced the original macro call. They also differ in that the arguments to the macro are not evaluated; instead, their s-expr representations are passed to the macro, and the macro can control when, if, and how many times they're evaluated, by the arrangement of the code that it returns. Since all of this occurs during interpretation or compilation, and you have the full power of the language available to you in examining and transforming the original macro call, you're essentially injecting your own code into the interpretation and compilation processes.
Since macros themselves are written in Lisp, and can operate on a convenient data representation of code, they are much more powerful than macros in most other languages. Debugging them can be easy or hard depending on the sophistication of the implementation - the problem you mention with C macros tends to be reversed in Lisp implementations, you usually see only the expansion when debugging. (The expansions of complex macros can be very confusing!)
One thing that helps a lot with debugging Lisp macros is that you can interactively ask the interpreter to expand a macro invocation form and show you the results; this feature is available in all Common Lisp implementations, since it's required by the language standard. (See MACROEXPAND and MACROEXPAND-1.)
-- DanMuller
s-expressions are an EXTERNAL text representation form. Macros don't get these as a parameter, but instead an internal representation of Lisp data (lists, symbols, numbers, you name it).
The evaluator or compiler does no reading, so it cannot process macros during reading. The evaluator/compiler gets Lisp data that already has been READ. If the evaluator/compiler gets some internal representation of code, it will do some macro processing during its computation. The compiler will compile the code for example, after it has been macro expanded.
What do you call something that evaluates its parameters at run time, but you don't want to quote it? Suppose I wanted to create my own control structure, but I want the ess-expression inside that call to be sent as a parameter(s) at run time instead of evaluated first without having to make the caller quote stuff. Is there a difference between compile-time macros and run-time macros?
Answer: The macros that people usually talk about in Lisp are expanded at compile-time or when code is interpreted. (I corrected some mistakes related to this in my description above.) These macros have explicit control over when, how often, and if the arguments are evaluated. There are also some other similar mechanisms in CommonLisp:
Interestingly, in IoLanguage one can handle the macro-ish cases where one wants to handle evaluation differently than the norm simply by using another route to get at the arguments. For instance, in EuropaProject?, we've introduced a method named ' (similar to using ' in Lisp) to easily allow us to pass methods and blocks around without necessarily invoking them. In short, methods are handed symbols, but the behind-the-scenes magic evaluates them in the sender before you get to them. If you want the symbols (IoLanguage has them as instances of Message, I believe), you refer to the element in the arguments list. Therefore our ' implementation (initially stolen from the iolanguage Yahoo group's mailing list, I believe) goes something like this:
Object setSlot("'", method( self getSlot(thisMessage arguments at(0) code) ))
So we're setting the ' slot on Object to a method which ensures the handling of the following Message as a single, passed Message rather than as a precalculated value. -- RobRix
It should be pointed out that the HygienicMacros pattern language in SchemeLanguage are radically different from traditional Lisp macros, and represent one of the more notable (and controversial) differences between Scheme and most of the other LispFamily languages. See DefineSyntax for an explanation of Scheme macros. -- JayOsako
A decent motivating example for C programmers might be the declaration of a pretty-printable enum. Just about every C programmer has at some point written something like
enum foo_t {
BAR = 8,
BAZ = 42,
FROB = 50
};
const char* foo_string (enum foo_t foo) {
static const char* lookup [] = {
[8] = "BAR",
[42] = "BAZ",
[50] = "FROB"
};
return lookup [foo];
}
This works, but it's tedious to extend, easy to get out of sync, and difficult to switch (no pun intended) to a switch-based lookup on a whim. It also feels kind of silly to have to manually type out both sets every time you want a printable enum. I'd be mighty impressed if you hid this all away with the preprocessor. Now, pretend for a moment Lisp has enums (and that we're more interested in symbols than strings):
(defenum foo
(bar 8)
(baz 42)
(frob 50))
(let ((lookup '((8 . bar)
(42 . baz)
(50 . frob)))) ;; Closures give a reasonable approximation of
;; static function-local variables
(defun foo->symbol (foo)
(cdr (assoc foo lookup))))
Well, in this case we can write a macro to do this for us:
(defmacro define-stringable-enum (name &rest defs)
`(progn
(defenum ,name ,@defs)
,(let ((func-name (intern (concatenate 'string (symbol-name name) "->SYMBOL")))
(param-name (gensym))
(lookup-alist-name (gensym))
(lookup-alist
(loop for (symbol value) in defs collecting
(cons value symbol))))
`(let ((,lookup-alist-name ',lookup-alist))
(defun ,func-name (,param-name)
(cdr (assoc ,param-name ,lookup-alist-name)))))))
Indeed, calling macroexpand:
(macroexpand '(define-stringable-enum foo (bar 8) (baz 42) (frob 50)))
=> (PROGN
(DEFENUM FOO (BAR 8) (BAZ 42) (FROB 50))
(LET ((#:G1415 '((8 . BAR) (42 . BAZ) (50 . FROB))))
(DEFUN FOO->SYMBOL (#:G1414) (CDR (ASSOC #:G1414 #:G1415)))))
we see something not entirely unlike what we wrote. This has the features we'd expect from a powerful macro system: each enum need only be enumerated once, the compiler does the dirty work of writing the lookup function for us, and perhaps most importantly, we can change the lookup method, e.g. to a hash-table, without touching any of the code that actually defines an enum. In fact, we can even write code to decide which lookup method to use for a given enumeration, in Lisp, called from the macro at compile time. Try doing that with the C preprocessor!
Oh, and if you actually did want (really dumb) enums in Lisp:
(defmacro defenum (name &rest defs)
(declare (ignore name)) ;; Doing something interesting with NAME is left as an
;; exercise for the reader. Maybe something with CLOS
;; for strong type semantics would be nice.
`(progn
,@(loop for def in defs collecting
`(defconstant ,@def))))
Boom. New language construct.
EditText of this page
(last edited December 16, 2014)
or FindPage with title or text search