Lisp Macro Discussion Two

LispMacroDiscussion got too big for me to be able to edit it, so I moved the last little bit over here, because I'm too tired to do the right thing. We might be getting closer to a conclusion, though, so I'm hoping that we'll be able to refactor this mess away soon. -- AdamSpitz

[Answer to Adam's questions; after this, RefactorMe or DeleteMe -- AlainPicard]

Adam -- thanks. Now I think I've gotcha. I was wrong after all; you don't understand read time. Well, you understand it technically, but you don't grok it.

No, you cannot step through a macro in the debugger, because the macro is gone, gone forever. The macro is a construct which is explicitly meant to take action at READ time only.

Yes, you don't need macros. You can just use lambda everywhere, and get something which is (as near as I can tell from your description) equivalent to SELF (which looks like a cool language, btw).

Lispers use macros anyway because it is the construct which allows them to build a new language in which to describe and express their problems. Macros are about building new syntax into the language (which is fully on par with the original language. This is important, because the seamlessness between the user's extensions and the original language allows even more future, regular extensions.) Lisper's aren't worried that they cannot mapcar #'IF; it's not an operation which makes a lot of sense. Similarly, the macros they define don't make sense in a functional sense (otherwise they'd have been defined as functions) so the loss of "uniformity" is not perceived as a loss.

Say you have an electrical engineering signals processing application. You might like to write things like: 

   (transform signal -> low-pass-filter -> high-pass-filter -> notch-filter)
This is just syntactic sugar, yes, but can you not see how this sugar makes it possible to write excruciatingly clear applications? The program is short, concise, and obviously correct because it has been expressed in a language close to the application domain.

I dunno if I'm helping, so I'll stop now. :-) --AlainPicard

Yes, I agree that that's very clear, and I value that. But are you sure that it's the syntactic sugar that's making your code clear?

I think that the thing that makes your code clear is the ability to give names to all the concepts in your mind and represent those concepts by their names. Lisp gives you the ability to give names to the concepts of "transform", "signal", "low pass filter", "high pass filter", and "notch filter", and that's why the code is clear - it says exactly what you want to say, and doesn't say anything else.

In Self, I might write: 

    signal transform: lowPassFilter & highPassFilter & notchFilter
(That's sending a message named "transform:" to the signal object, passing in a list of the three filter objects. The & message is just Self's idiom for constructing lists.)

I might be tempted to define a -> method for signal objects, so that I could write: 

    signal -> lowPassFilter -> highPassFilter -> notchFilter
...which I think is kinda cool, but not really the point; I think the first version is almost as readable. (Isn't it?) Self happens to give me the ability to define this particular kind of syntactic sugar, but the really big gains came from the ability to express the concepts of the problem directly in the language.

Notice that there are two requirements here: you need to be able to give names to your concepts, and you need to be able to use those names to represent the concepts. Macros let me give a name to just about anything, but don't let me use that name in some situations. I'm willing to believe that this isn't a problem, but I don't yet understand why. How did it come about that all the concepts that are more succinctly expressed through macros are concepts that don't need to be used as first-class values at runtime?

-- AdamSpitz

I'm not sure the first version is as readable. This example seems to go to the crux of the matter, as I understand it. This form: 

 (transform signal -> low-pass-filter -> high-pass-filter -> notch-filter)
Seems to me to read a lot like a description of what some actual signal processing hardware might look like, in that sense it is close to the domain. The ->s might transform into some explicit list-building code, or they might not, who cares? The point is exactly that this is a new language with new syntax, where -> means connect the output of the signal processing block on the left to the input of the block on the right. Whereas the form: 
 signal transform: lowPassFilter & highPassFilter & notchFilter
Definitely isn't a new language, it's still Self, and to use it requires understanding of building explicit lists and sending messages to objects, which is not very much like assembling signal processing hardware at all. I guess you either think that the Lisp macro approach here is a win or you don't. Then the form: 
 signal -> lowPassFilter -> highPassFilter -> notchFilter
is a lot more than just cute, it gets closer to the "looking like the hardware might" property, which again you either think is a win or not. But it still isn't a new language, and to understand it means understanding sending the signal object the message -> with an argument, the object you get from sending the message -> to the object... and so on. Again, not much like plugging together the hardware.

Now, Lisp supports a bunch of computational styles already, functional, procedural and so on, any one of which can of course be used to model any domain you like, but Lisp macros let you introduce new styles than map most directly onto the problem in hand. Signal processing in hardware is probably closer to functional programming than the others, but that doesn't necessarily mean that a functional style program most clearly expresses a given signal processing problem. But the signal-processing style language that the macros let us build can most clearly express a given signal processing problem. Macros or not isn't much of a formal, computer science type issue, it's a pragmatic, software engineering type issue. -- KeithBraithwaite

There are some interesting thought patterns emerging here.

Keith, the way you talked about object-oriented programming ("sending the signal object the message -> with an argument, the object you get from sending the message -> to the object... and so on") sounds very weird to me. That's not the way I read it at all. When I read that code, my brain doesn't use the words "object" or "message" or "argument". I don't think of "signal" as "a Self object representing the signal" - I think of it as "the signal". And "->" means "connect it to", and "lowPassFilter" means, "the low pass filter". So I read that code as, "Connect the signal to the low pass filter," which is remarkably similar to what you said (substituting "the signal" and "the low pass filter" for "the block on the left" and "the block on the right").

I was surprised when you said, "not much like plugging together the hardware," because I was thinking that it was exactly like plugging together the hardware. "signal" and "lowPassFilter" are the hardware, and "->" is how I plug them together.

I think you're looking at the language, and I'm looking at the objects. I think that Lisp doesn't make the objects real enough, and you think that Self doesn't make the language direct enough.

Does that make any sense? If it does, then I think I understand macros better. If you think of a program as a bunch of sentences in a language, then you want that language to be as powerful as possible (and so macros make perfect sense). But if you think of a program as a world of interacting objects, then you want those objects to be as real as possible (and so a concept that's only real part of the time makes no sense at all).

-- AdamSpitz

Adam, that's an excellent observation. I don't know what you mean by objects not being real enough, but I certainly do think of a program as a bunch of sentences, and I want to speak to the computer in a language that we're both comfortable with.

FWIW, when I showed the TRANSFORM example, I had in mind a replacement for an expression like: 

  (let  ((output (apply-filter #'notch-filter
                               (apply-filter #'low-pass-filter
                                             (apply-filter #'high-pass-filter
                                                           signal)))))
     (do-stuff output))
and the TRANSFORM lets me "invert" the direction in which I need to read the arguments (in a traditional functional programming style). This way, I get functional application but I can "see" filters being processed in the way I mean to connect them. This is why your observation (of "a bunch of sentences") resonates strongly with me, although I, of course, see it as a profound change. The above looks like LISP. The (transform foo -> bar -> baz ...) doesn't; when I read it, I don't have to do any "mental mapping" between my problem domain and my implementation language; they have become one and the same. -- ap

Quick, irrelevant question about that expression there: Couldn't you define a function called "transform" that takes a signal and a &rest parameter (is that the right term?), so that you could write: 

    (transform  signal  #'high-pass-filter  #'low-pass-filter  #'notch-filter)
That puts the filters in the proper order, and makes the code just as short as the macro version. (And you could still go ahead and write the macro to give you the cool -> notation, if you wanted to. It would just expand trivially into a call to this function.)

In any case, though, I think I'm happy. Here are some questions I'd like to think more about, regarding this bunch-of-sentences/world-of-objects thing:

1. Does it make a difference who you are? (Maybe some people think better about bunches of sentences, and some people think better about worlds of objects.)

2. Does it make a difference what the problem is? (Maybe some programs make more sense as a bunch of sentences, and some programs make more sense as a world of objects.)

3. Is there a way to combine the two worldviews? Is there any reason to?

See RealObject for my attempt to explain what I mean by "not real enough." -- AdamSpitz

Let me suggest a few nice full-scale examples of what you can do with Lisp macros, which I think give the full flavour:

-- LukeGorrie

Adam, you may be interested in "First Class Macros have Types" http://citeseer.nj.nec.com/494837.html.

EditText of this page (last edited February 20, 2006) or FindPage with title or text search