Lisp In Jak And Daxter

The post-mortem for Naughty Dog's game Jak and Daxter is at

http://www.gamasutra.com/features/20020710/white_02.htm [used to require free registration with e-mail confirmation]

It's very interesting reading from a process perspective, and also because of their use of LispLanguage. It seems they went completely ToolHappy, writing their own language and compiler (!) based on Allegro CL.

This had some benefits, but also some major downsides. SmugLispWeenies should take note of the difficulties they encountered next time they think about Lisp as the MagicBullet that can give a company a huge competitive advantage.

Well, SmugLispWeenies like to point out that the use of Lisp has a long tradition with Naughty Dog. The Crash Bandicoot series was written with Lisp, Jak & Daxter and then the latest Jak II. Somewhere inbetween Naughty Dog got bought by Sony. You may want to concentrate on the problems - I would concentrate on how a small company was able to write their own development environment for the Playstation I and II, created a few beautiful games with one of the best graphics and animation engines on the market. With Lisp.
Actually, in the first Crash Bandicoot, Lisp was only used to decide how to load objects, in the very limited Playstation architecture. Don't know about the later games though.
Actually I don't think this is true. You can read about the use of Common Lisp and GOOL (GOAL (not GOOL) then was later the language for Jak & Daxter and Jak II) in Crash Bandicoot here: http://www.franz.com/success/customer_apps/animation/naughtydog.php3 . Quote: Naughty Dog used Allegro CL for the character control portions of the game. Quote: For the Crash Bandicoot project, Naughty Dog used Allegro CL to create a programming language called "GOOL" (Game Object Oriented Language), which is specifically geared to game development. Using this Allegro CL-based language, the team was able to produce hundreds of different game objects with sophisticated real-time behavior and animation.
You are right. More exactly, GOOL was used to control objects in the game:

For Crash Bandicoot we implemented GOOL (Game Oriented Object LISP), a compiled language designed specifically for object control code that addresses the limitations of traditional languages.

Having a custom language whose primitives and constructs both lend themselves to the general task (object programming), and are customizable to the specific task (a particular object) makes it much easier to write clean descriptive code very quickly. GOOL makes it possible to prototype a new creature or object in as little as 10 minutes. New things can be tried and quickly elaborated or discarded. If the object doesn't work out, it can be pulled from the game in seconds without leaving any hard to find and wasteful traces behind in the source. In addition, since GOOL is a compiled language produced by an advanced register coloring compiler with reductions, flow analysis, and simple continuations, it is at least as efficient as C, more so in many cases because of its more specific knowledge of the task at hand. The use of a custom compiler allowed us to escape many of the classic problems of C.

You can read all about it in Gamasutra: http://www.gamasutra.com although, you will have to register.

Advantages:

Complete control over the language and compiler (including macros).
Ability to interact with a running game through a listener.
One high-level language for code compiled to different processors (note, that's different processors inside the PS2, not different platforms!)

Disadvantages:

The disadvantages listed below are largely problems inherent in a developer-driven (rather than customer- or goal-driven) development process, but they are to different degrees also directly related to the choice of Lisp as a development language.

Their development language (GOAL) was written by one person. Everything having to do with the language had to go through this one person (nobody else understood the system), creating a huge bottleneck. This is obviously a process problem rather than a lisp-related problem, but it is a danger when creating task-specific dialects of lisp.
"Also, it took over a year to develop the compiler, during which time the other programmers had to make do with missing features, odd quirks, and numerous bugs."
Memory usage (duh!), but more in the development process than the running game, it seems.
Gave up all existing tools (debuggers, profilers, etc.) and libraries, including existing Naughty Dog code.
"Compared to the thousands of programmers with many years of C++ experience, there are relatively few programmers with Lisp experience, and no programmers (outside of Naughty Dog) with GOAL experience, making hiring more difficult."

Other problems:

What tools existed came very late in the process and had very programmer-oriented (command-line) interfaces that artists found very hard to use.
All the time spent on tools meant little time for actually working on the game!

Honestly, after reading the post-mortem, and knowing that Jak and Daxter came together to be an artistic and commercial success, I'm very impressed by the levels of skill (and luck) of the Naughty Dog crew (especially the poor designers and artists!).

Mostly the stuff above is superseded already by the latest game Jak II. Naughty Dog wrote even more GOAL (= Lisp) for this game ;-) : Jak II. Jak II is about 900000 lines (!!) of GOAL (Lisp) plus some more C/C++/Common Lisp/MEL... . So, they have not given up on Lisp, but instead worked even harder to create an absolutely stunning game.

Ahh the good ol WikiNow (it's like a discussion blog, but without dates) ... Jak III is "now" the latest game. Given how hot on the heels it came after Jak II, I suspect it's largely the same engine, perhaps tuned up a smidge. Any reports on the amount of GOAL in that game?

Jak III was mostly GOAL like Jak II. In future Naughty Dog won't be using their lisp-like language and tools because the guy who wrote it all will no longer be available to support them. So they will be moving to C++ possibly using Open C++ to bend the language to their needs.

From the following thread http://lists.midnightryder.com/pipermail/sweng-gamedev-midnightryder.com/2004-November/002611.html

"After using a completely custom language for our last three PS2 projects, coming back to C++ has been a pretty rude awakening. We were hoping we could work around some of the deficiencies of C++ with a good preprocessor, ideally one that provided more than simple hygienic macro substitution; we really would like the capability to actually parse C++ so that we can write metacode that actually understands our code structure and class hierarchy. So far, the only available packages I've found that appear to fit the bill are OpenC++ and FOG (Flexible Object Generator), although there may be others." -- Scott Shumaker, Naughty Dog, Inc.

They need to stop thinking like this if they want to make Jak IV. They're going to end up rewriting Lisp in C++ and doing the Yahoo! Store thing all over again. C++ isn't Lisp and if they're committed to changing (I imagine the effort involved in porting their compiler to new consoles is significant), they need to stop thinking about how they'd solve problems in Lisp and more about how they'd solve them in C++. If your first instinct when you move back to C++ is that you need a new preprocessor, you need to stop, take some time off, and reimmerse yourself in C++.

Alternately, they need to spend a year or so getting some other people up to speel on GOAL internals and keep using that. It'll probably cost less, both in time and money, anyway.

There are certain deficiencies in C++ that GOAL addresses neatly. Simply "re-immersing" oneself in C++ doesn't make these problems go away (not to mention the fact that pretty much all the ND programmers are already extremely proficient in C++). One trivial example: GOAL permits compile-time select/inserts on a set of shared SQL tables (containing all kinds of art asset information) - the existing C++ preprocessor certainly won't let you do this.

Well, GOAL appears to be pretty tightly bound to the track record your outfit has accumulated for prompt shipment and high quality projects. Your main GOAL dude took a hike, and now you have no resident genius to support your custom language. Stick with the custom language and push it into the world as a GPL'd dialect of Lisp. Build a community around it, and you will find you have less hassle bringing the kids up to speed as you hire them on. ;) that doesn't solve your immediate problems, though... I think I'd try to work through the patch sticking with what you have spent all this time developing to date, rather than dispatch the baby with the bathwater.

There are certain deficiencies in C++ that GOAL addresses neatly.

I'm sure there are

Simply "re-immersing" oneself in C++ doesn't make these problems go away (not to mention the fact that pretty much all the ND programmers are already extremely proficient in C++).

That's as may be, but they're thinking like GOAL programmers and not like C++ programmers. If they aren't going to use GOAL anymore, they need to stop doing that.

One trivial example: GOAL permits compile-time select/inserts on a set of shared SQL tables (containing all kinds of art asset information) - the existing C++ preprocessor certainly won't let you do this.

That's a cute trick, which I see absolutely no benefit in making part of your language, but if that sort of thing is what they need, then they really need to go with the second option I mentioned - suck it down and get some real support and documentation in place for GOAL and go forward using that. Trying to force C++ to act like GOAL with all these no doubt hacked together tools they'll write will waste a crapload of time and money, just like the Yahoo! Stores debacle did. If they're committed to using C++ instead of GOAL, they need to think like C++ programmers and solve things in C++ ways, including using separate toolchains instead of a do-it-all custom built language. I'm not speaking at all about the relative merits of the Lisp/GOAL/whatever vrs C++, I'm talking about the sort of process flaws that result in Greenspunning. Don't re-write GOAL in C++ because that's what you're used to - that's stupid. Either stick with GOAL or fully commit (this includes re-thinking your workflow, processes, and design decisions) to C++.
- Start thinking like a game developer. That compile-time SQL processing could be keeping track of a number of things. For example, it could keep an accurate count of how many times each art resource is being used in the current build. That would allow the devs to easily eliminate unused assets once the game is ready to ship.

Interesting article; I am a huge proponent for using the right tool for the job. If it turns out that C/C++ isn't up to the task for a given job (especially considering time to market requirements, which is pretty vital in the gaming industry), then by golly I will not use it. It is unfortunate that doing this necessarily means restricting the pool of developers you can hire, provided you want to minimize training costs. I have some experience with Lisp, and except for the ugly source code, I generally like the language. I'm also a fan of ForthLanguage as well (particularly in the early days, Forth had a large presence in the gaming industry).

What seems to me they should do is research the possibility of embeddable languages. I know that there are a variety of Scheme, Lisp, Forth, etc. embeddable language variants. And, of course, perhaps the most famously known language used for embedding, Lua.

Is there any discussion on why they didn't persue this course of action? --SamuelFalvo?

The difficult part, as I understand the situation, was coordinating assembler code for multiple processors. Porting an existing tool to that environment would have been the same amount of work or more, since the end result included a fast compile-and-test loop, lisp macros, and other goodies.

One of the prevailing themes of this story, and possibly a cautionary one against the power of Lisps: Too much proprietary technology was invented at once. Naughty Dog's team managed to ship three games because it worked, but they were unable to draw from the "WorseIsBetter community" of C++, and had also become too distant from mainstream Lisp to get help from that area. Without successors to maintain it, the language became a bottleneck. A more cautioned approach where the game code started from a existing well-supported dialect, and gradually added in the hooks, features, and optimizations necessary to ship, might have successfully spread knowledge around the team. (on the other hand, the system specs and available implementations when GOAL was started may have made this unfeasible.)

Without successors to maintain it, the language became a bottleneck. This is a management problem. Management failed to realize that engineers migrate from company to company. This is why Google holds regular tech-talks -- the goal is to distribute knowledge of proprietary and innovative technologies to as wide an audience as possible. This helps defend Google from BrainDrain.
they were unable to draw from the "WorseIsBetter community" of C++. Under no circumstances is WorseIsBetter a trait of CeePlusPlus, or for that matter, even of CeeLanguage. Nor were they the innovators of the concept (remember, CeeLanguage had its predecessors, BeeLanguage and BcplLanguage). Hence, WorseIsBetter could just as easily have been taken advantage of with GOAL as with CommonLisp, C++, or dare I say it, even BasicLanguage. I might even argue that the WorseIsBetter philosophy was singularly responsible for BasicLanguage's huge success in the 8-bit home computer era. Were there better languages? Yeah. But it was good enough.

Update: 2012-02-15 From a 20-minute behind-the-scenes video of the PlayStation 3 port of the Jak and Daxter PlayStation 2 trilogy, there is a shot of commented out LISP in a C source file, at 10m37s. Naughty Dog handed off the source code and assets to developer Mass Media, while proclaiming "This is an impossible project." The non-technical discussion about porting starts at 7m10s. http://www.joystiq.com/2012/02/15/jak-and-daxter-collection-trailer-gets-deep-super-nostalgic

Haven't you heard? They're not writing C++, they're generating it from Racket. Half of everything above is irrelevant. Now see where wild speculation gets you? http://www.gameenginebook.com/resources/gdc09-statescripting-uncharted2.pdf

CategoryHistory CategoryGame