Whenever I write client-server applications, I run into the same problem trying to separate the code. To send a message from the server to the client, the server has to serialize that message and the client has to unserialize that message. The server doesn’t need to unserialize that message. The client doesn’t need to serialize that message.

It seems wrong to include both the serialization code and the unserialization code in both client and server when each side will only be using 1/2 of that code. On the other hand, it seems bad to keep the serialization and unserialization code in separate places. You don’t want one side serializing A+B+C and the other side trying to unserialize A+C+D+B.

One approach: data classes

Some projects deal with this situation by making every message have its own data class. You take all of the information that you want to be in the message and plop it into a data class. You then serialize the data class and send the resulting bytes. The other side unserializes the bytes into a data class and plucks the data out of the data class.

The advantage here is that you can have some metaprogram read the data class definition and generate a serializer or unserializer as needed. You’re only out-of-sync if one side hasn’t regenerated since the data class definition changed.

The disadvantage here is that I loathe data classes. If my top-level interface is going to be (send-login username password), then why can’t I just serialize straight from there without having to create a dippy data structure to hold my opcode and two strings?

Another approach: suck it up

Who cares if the client contains both the serialization and unserialization code? Heck, if you’re really all that concerned, then fmakunbound half the universe before you save-lisp-and-die.

Of course, unless you’re using data classes, you’re either going to have code in your client that references a bunch of functions and variables that only exist in your server or your client and server will be identical except for:

(defun main ()
  #+server (server-main)
  #-server (client-main))

Now, of course, your server is going to accidentally depend on OpenGL and OpenAL and SDL and a whole bunch of other -L’s it never actually calls. Meanwhile, your client is going to accidentally depend on Postmodern and Portable-Threads and a whole bunch of other Po-’s it never actually calls.

Another approach: tangle and weave, baby

Another way that I’ve got around this is to use literate programming tools to let me write the serialiization and unserialization right next to each other in my document. Then, anyone going to change the serialize code would be immediately confronted with the unserialize code that goes with it.

The advantage here is that you can tangle the client code through an entirely separate path than the server code keeping only what you need in each.

The disadvantage here is that now both your client code and your server code have to be in the same document or both include the same sizable chunk of document. And, while there aren’t precisely name-capturing problems, trying to include the “serialize-and-send” chunk in your function in the client code still requires that you use the same variable names that were in that chunk.

How can Lisp make this better?

In Lisp, we can get the benefits of a data-definition language and data classes without needing the data classes. Here’s a snippet of the data definition for a simple client-server protocol.

;;;; protocol.lisp
(userial:make-enum-serializer :opcode (:ping :ping-ack))
(defmessage :ping     :uint32 ping-payload)
(defmessage :ping-ack :uint32 ping-payload)

I’ve declared there are two different types of messages, each with their own opcode. Now, I have macros for define-sender and define-handler that allow me to create functions which have no control over the actual serialization and unserialization. My functions can only manipulate the named message parameters (the value of ping-payload in this case) before serialization or after unserialization but cannot change the serialization or unserialization itself.

With this protocol, the client side has to handle ping messages by sending ping-ack messages. The define-sender macro takes the opcode of the message (used to identify the message fields), the name of the function to create, the argument list for the function (which may include declarations for some or all of the fields in the message), the form to use for the address to send the resulting message to, and any body needed to set fields in the packet based on the function arguments before the serialization. The define-handler macro takes the opcode of the message (again, used to identify the message fields), the name of the function to create, the argument list for the function, the form to use for the buffer to unserialize, and any body needed to act on the unserialized message fields.

;;;; client.lisp
(define-sender  :ping-ack send-ping-ack (ping-payload) *server-address*)
(define-handler :ping     handle-ping   (buffer) buffer
  (send-ping-ack ping-payload))

The server side has a bit more work to do because it’s going to generate the sequence numbers and track the round-trip ping times.

;;;; server.lisp
 
(defvar *last-ping-payload* 0)
(defvar *last-ping-time*    0)
 
(define-sender :ping send-ping (who) (get-address-of who)
  (incf *last-ping-payload*)
  (setf *last-ping-time*    (get-internal-real-time)
        ping-payload        *last-ping-payload*))
 
(define-handler :ping-ack handle-ping-ack (who buffer) buffer
  (when (= ping-payload *last-ping-payload*)
    (update-ping-time who (- (get-internal-real-time) *last-ping-time*))))

Problems with the above

It feels strange to leave compile-time artifacts like the names and types of the message fields in the code after I’ve generated the functions that I’m actually going to use. But, I guess that’s just part of Lisp development. You can’t (easily) unload a package. I can makunbound a bunch of stuff after I’m loaded if I don’t want it to be convenient to modify senders or handlers at run-time.

There is intentional name-capture going on. The names of the message fields become names in the handlers. The biggest problem with this is that the defmessage calls really have to be in the same namespace as the define-sender and define-handler calls.

I still have some work to do on my macros to support &key and &optional and &aux and &rest arguments properly. I will post those macros once I’ve worked out those kinks.

Anyone care to share how they’ve tackled client-server separation before?

I wrote (in Java) an XML parser generator that took an XML description of how you’d like the native-language data structures to look and where in the XML it could find the values for those data structures. The Java code-base for this was ugly, ugly, ugly. I tried several times to clean it up into something publishable. I tried to clean it up several times so that it could actually generate the parser it used to read the XML description file. Alas, the meta-ness, combined with the clunky Java code, kept me from completing the circle.

Fast forward to last week. Suddenly, I have a reason to parse a wide variety of XML strings in Objective C. I certainly didn’t want to pull out the Java parser generator and try to beat it into generating Objective C, too. That’s fortunate, too, because I cannot find any of the copies (in various states of repair) that once lurked in ~/src.

What’s a man to do? Write it in Lisp, of course.

Example

Here’s an example to show how it works. Let’s take some simple XML that lists food items on a menu:

<menu>
        <food name="Belgian Waffles" price="$5.95" calories="650">
                <description>two of our famous Belgian Waffles with plenty of real maple syrup</description>
        </food>
        <!-- ... more food entries, omitted here for brevity ... -->
</menu>

We craft an XML description of how to go from the XML into a native representation:

<parser_generator root="menu" from="/menu">
  <struct name="food item">
    <field type="string" name="name" from="@name" />
    <field type="string" name="price" from="@price" />
    <field type="string" name="description" from="/description/." />
    <field type="integer" name="calories" from="@calories" />
  </struct>
 
  <struct name="menu">
    <field name="menu items">
      <array>
        <array_element type="food item" from="/food" />
      </array>
    </field>
  </struct>
</parser_generator>

Now, you run the parser generator on the above input file:

% sh parser-generator.sh --language=lisp \
                           --types-package menu \
                           --reader-package menu-reader \
                           --file menu.xml

This generates two files for you: types.lisp and reader.lisp. This is what types.lisp looks like:

(defpackage :menu
  (:use :common-lisp)
  (:export #:food-item
             #:name
             #:price
             #:description
             #:calories
           #:menu
             #:menu-items))
 
(in-package :menu)
 
(defclass food-item ()
  ((name :initarg :name :type string)
   (price :initarg :price :type string)
   (description :initarg :description :type string)
   (calories :initarg :calories :type integer)))
 
(defclass menu ()
  ((menu-items :initarg :menu-items :type list :initform nil)))

I will not bore you with all of reader.lisp as it’s 134 lines of code you never had to write. The only part you need to worry about is the parse function which takes a stream for or pathname to the XML and returns an instance of the menu class. Here is a small snippet though:

;;; =================================================================
;;; food-item struct
;;; =================================================================
(defmethod data progn ((handler sax-handler) (item food-item) path value)
  (with-slots (name price description calories) item
    (case path
      (:|@name| (setf name value))
      (:|@price| (setf price value))
      (:|/description/.| (setf description value))
      (:|@calories| (setf calories (parse-integer value))))))

Where it’s at

I currently have the parser generator generating its own parser (five times fast). I still have a little bit more that I’d like to add to include assertions for things like the minimum number of elements in an array or the minimum value of an integer. I also have a few kinks to work out so that you can return some type other than an instance of a class for cases like this where the menu class just wraps one item.

My next step though is to get it generating Objective C parsers.

Somewhere in there, I’ll post this to a public git repository.

Many years back, I wrote some ambient music generation code. The basic structure of the code is this: Take one queen and twenty or so drones in a thirty-two dimensional space. Give them each random positions and velocities. Limit the velocity and acceleration of the queen more than you limit the same for the drones. Now, select some point at random for the queen to target. Have the queen accelerate toward that target. Have the drones accelerate toward the queen. Use the average distance from the drones to the queens in the -th dimension as the volume of the -th note where the notes are logarithmically spaced across one octave. Clip negative volumes to zero. Every so often, or when the queen gets close to the target, give the queen a new target.

It makes for some interesting ambient noise that sounds a bit like movie space noises where the lumbering enemy battleship is looming in orbit as its center portion spins to create artificial gravity within.

I started working on an iPhone application based on this code. The original code was in C++. The conversion to Objective C was fairly straightforward and fairly painless (as I used the opportunity to try to correct my own faults by breaking things out into separate functions more often).

Visualization troubles

The original code though chose random positions and velocities from uniform distributions. The iPhone app is going to involve visualization as well as auralization. The picture at the right here is a plot of five thousand points with each coordinate selected from a uniform distribution with range [-20,+20]. Because each axis value is chosen independently, it looks very unnatural.

What to do? The obvious answer is to use Gaussian random variables instead of uniform ones. The picture at the right here is five thousand points with each coordinate selected from a Gaussian distribution with a standard-deviation of 10. As you can see, this is much more natural looking.

How did I generate the Gaussians?

I have usually used the Box-Muller method of generating two Gaussian-distributed random variables given two uniformly-distributed random variables:

(defun random-gaussian ()
  (let ((u1 (random 1.0))
        (u2 (random 1.0)))
    (let ((mag (sqrt (* -2.0 (log u1))))
          (ang (* 2.0 pi u2)))
      (values (* mag (cos ang))
              (* mag (sin ang))))))

But, I found an article online that shows a more numerically stable version:

(defun random-gaussian ()
  (flet ((pick-in-circle ()
           (loop as u1 = (random 1.0)
                as u2 = (random 1.0)
                as mag-squared = (+ (* u1 u1) (* u2 u2))
                when (< mag-squared 1.0)
                return (values u1 u2 mag-squared))))
    (multiple-value-bind (u1 u2 mag-squared) (pick-in-circle)
      (let ((ww (sqrt (/ (* -2.0 (log mag-squared)) mag-squared))))
        (values (* u1 ww)
                (* u2 ww))))))

For a quick sanity check, I thought, let’s just make sure it looks like a Gaussian. Here, I showed the code in Lisp, but the original code was in Objective-C. I figured, If I just change the function declaration, I can plop this into a short C program, run a few thousand trials into some histogram buckets, and see what I get.

The trouble with zero

So, here comes the problem with zero. I had the following main loop:

#define BUCKET_COUNT 33
#define STDDEV       8.0
#define ITERATIONS   100000
 
  for ( ii=0; ii < ITERATIONS; ++ii ) {
    int bb = val_to_bucket( STDDEV * gaussian() );
    if ( 0 <= bb && bb < BUCKET_COUNT ) {
      ++buckets[ bb ];
    }
  }

I now present you with three different implementations of the val_to_bucket() function.

int val_to_bucket( double _val ) {
  return (int)_val + ( BUCKET_COUNT / 2 );
}
 
int val_to_bucket( double _val ) {
  return (int)( _val + (int)( BUCKET_COUNT / 2 ) );
}
 
int val_to_bucket( double _val ) {
  return (int)( _val + (int)( BUCKET_COUNT / 2 ) + 1 ) - 1;
}

As you can probably guess, after years or reading trick questions, only the last one actually works as far as my main loop is concerned. Why? Every number between -1 and +1 becomes zero when you cast the double to an integer. That’s twice as big a range as any other integer gets. So, for the first implementation, the middle bucket has about twice as many things in it as it should. For the second implementation, the first bucket has more things in it than it should. For the final implementation, the non-existent bucket before the first one is the overloaded bucket. In the end, I used this implementation instead so that I wouldn’t even bias non-existent buckets:

int val_to_bucket( double _val ) {
  return (int)lround(_val) + ( BUCKET_COUNT / 2 );
}

(define (dtft samples)
  (lambda (omega)
    (sum 0 (vector-length samples)
        (lambda (n)
          (* (vector-ref samples n)
             (make-polar 1.0 (* omega n)))))))

My first thought was That’s way too short. Then, I started reading through it. My next thought was, maybe I don’t understand scheme at all. Then, my next thought was, I do understand this code, it just didn’t do things the way I expected.

Here, I believe is a decent translation of the above into Common Lisp:

(defun dtft (samples)
  #'(lambda (omega)
      (loop for nn from 0 below (length samples)
           summing (let ((angle (* omega nn)))
                     (* (aref samples nn)
                        (complex (cos angle) (sin angle)))))))

Now, what I find most interesting here is that most implementations you’ll find for the DTFT (Discrete-Time Fourier Transform) take an array of samples and a wavelength, and returns a result. This, instead, returns a function which you can call with a wavelength omega. It returns an evaluator. This is an interesting pattern that I will have to try to keep in mind. I have used it in some places before. But, I am sure there are other places that I should have used it, and missed. This is one where I would have missed.

Usage for the above would go something like:

(let ((dd (dtft samples)))
  (dd angle1)
  (dd angle2)
  ...)

For those not into Lisp either (who are you?), here is a rough translation into C++.

#include <cmath>
#include <complex>
 
class DTFT {
  unsigned int len;
  double* samples;
 
public:
  DTFT( double* _samples, unsigned int _len ) {
    this->len = _len;
    this->samples = new double[ this->len ];
    for ( unsigned int ii=0; ii < this->len; ++ii ) {
      this->samples[ ii ] = _samples[ ii ];
    }
  }
 
  ~DTFT( void ) {
    delete[] this->samples;
  }
 
  std::complex< double > operator () ( double omega ) const {
    std::complex< double > sum( 0.0, 0.0 );
    for ( unsigned int ii=0; ii < this->len; ++ii ) {
      sum += this->samples[ ii ] * std::polar( 1.0, omega );
    }
    return sum;
  }
};

With usage like:

DTFT dd( samples, 1024 );
dd( angle1 );
dd( angle2 );
...

So, six lines of Scheme or Lisp. Twenty-five lines of C++ including explicit definition of a class to act as a pseudo-closure, explicit copying and management of the samples buffer, etc. I suppose, a more direct translation would have used a std::vector to hold the samples and would have just kept a pointer to the buffer. That would have shaved off six or seven lines and the whole len and _len variables.

Most of the C++ stuff is done with classes. Most of the methods of those classes are virtual methods. Many of the methods will be called multiple times every hundredth of a second. Very few of the virtual methods ever get overridden by subclasses.

So, I got to thinking. Does it make sense for me to use CLOS stuff at all for most of this? Would it be significantly faster to use (defstruct …) and (defun …) instead of (defclass …) and (defgeneric …)?

My gut instinct was: Yes. My first set of test code didn’t bear that out. For both the classes and the structs, I used (MAKE-INSTANCE …) to allocate and (WITH-SLOTS …) to access. When doing so, the classes with generic functions took about 1.5 seconds for 10,000,000 iterations under SBCL while the structs with non-generic functions took about 2.2 seconds.

For the next iteration of testing, I decided to use accessors instead of slots. I had the following defined:

(defclass base-class ()
  ((slot-one :initform (random 1.0f0) :type single-float
             :accessor class-slot-one)
   (slot-two :initform (random 1.0f0) :type single-float
             :accessor class-slot-two)))
 
(defclass sub-class (base-class)
  ((slot-three :initform (random 1.0f0) :type single-float
               :accessor class-slot-three)
   (slot-four  :initform (random 1.0f0) :type single-float
               :accessor class-slot-four)))
 
(defstruct (base-struct)
  (slot-one (random 1.0f0) :type single-float)
  (slot-two (random 1.0f0) :type single-float))
 
(defstruct (sub-struct (:include base-struct))
  (slot-three (random 1.0f0) :type single-float)
  (slot-four (random 1.0f0) :type single-float))

I then switched from using calls like (slot-value instance ‘slot-three) in my functions to using calls like (class-slot-three instance) or (sub-struct-slot-three instance). Now, 10,000,000 iterations took 2.6 seconds for the classes and 0.3 seconds for the structs in SBCL. In Clozure (64-bit), 10,000,000 iterations with classes and with method-dispatch and with accessors took 11.0 seconds, with classess and without method-dispatch but with accessors took 6.1 seconds, and with structs and without method-dispatch and with accessors took 0.4 seconds.

There is much more that I can explore to tune this code. But, for now, I think the answer is fairly easy. I am going to use structs, functions, and accessors for as many cases as I can. Here is the generic-function, class, struct test code with accessors. The first loop uses classes and generic functions. The second loop uses classes and regular functions. The third loop uses structs and regular functions.

A Quick Look at Macros

First, let’s review macros for a moment to make sure we’re all on the same page. The C/C++ macros are the simplest to understand so we’ll start there. The C and C++ compilers run in two stages. First, the C preprocessor runs to expand all of your macros. Then, the compiler runs to actually compile your code. In C and C++, macros are declared using the #define command to the preprocessor. For example, you might do something like the following:

#define START_TIMER  __start_time__ = get_current_time();
#define END_TIMER    __elapsed__ = get_current_time() - __start_time__;

Then, later in you code, you might do something like:

START_TIMER
    for (ii = 0; ii < MAX_ITERATIONS; ++i) {
        do_foo();
    }
END_TIMER

The C preprocessor will replace the START_TIMER and END_TIMER with the code you declared in the #define. Your resulting code would then go into the actual compile phase looking like this:

__start_time__ = get_current_time();
    for (ii = 0; ii < MAX_ITERATIONS; ++i) {
        do_foo();
    }
__elapsed__ = get_current_time() - __start_time__;

The preprocessor did some simple string replacements on your behalf. [Actually, it is somewhat helpful to think of it as doing token replacement instead since the name of your macro has to be a valid C token and you have to jump through some ugly, implementation-specific hoops to get it to concatenate things rather than whitespace separate them.] You can also create macros with arguments like:

#define WARN(x,y)     log_message( __LOG_LEVEL_WARN__, x, y )
...
WARN( "buffer_copy()", "Output buffer is not properly aligned" );

If you look back at the START_TIMER and END_TIMER macros above, you will notice that the semicolons, which delimit statements in C and C++, are included in the macro definitions. As such, they aren’t used explicitly where the macros are invoked. So? Well, that means that your program needn’t bear any resemblance to compilable C before the preprocessor is run. Some sets of macros exploit this to make your C code look like FORTRAN or Bourne Shell or PASCAL or C++.

Lisp macros are a different beast entirely. Invoking a macro in Lisp looks just like everything else in Lisp. [To be fair, some macros (like (LOOP ...)) go a little nutty and make things look like a Lisp wrapper around some non-Lisp. Most macros are much more Lispy. Additionally, Lisp has two different sorts of macros: macros and reader macros. With reader macros, you could probably make your Lisp code look something like PASCAL if you were exceedingly ambitious and exceedingly misguided. But, from there on out, when I say macro, I specifically mean just macro as distinguished from reader macro.]

For pedagogic purposes, let’s look at those same macros defined in Lisp:

(defmacro START_TIMER () '(setf *start-time* (get-current-time)))
(defmacro END_TIMER () '(setf *elapsed* (- (get-current-time) *start-time*)))
(defmacro WARN (x y) `(log_message +log-level-warn+ ,x ,y))

The first two macros are straight substitutions: wherever you find the Lisp form (START_TIMER), substitute the form (setf *start-time* (get-current-time)), and similarly for the (END_TIMER) form. The single quote in front of the (SETF …) forms in those first two macros indicates that the following form, while parsed, is not to be evaluated at the time the macro is expanded. In the (WARN …) macro, things are a bit more complicated. The backquote before the (LOG_MESSAGE …) form indicates that the form itself is not to be evaluated at expansion time, but that some portions within it (those demarcated with commas) will need to be substituted. As you would expect:

(WARN "square-root" "Not interested in handling negative arguments today")

would expand into:

(log_message +log-level-warning+ "square-root" "Not interested in handling negative arguments today")

What makes Lisp macros so great?

From what I’ve shown you so far, you might be thinking: What make Lisp macros so great? Everything you’ve done with Lisp macros, I can do with the C preprocessor. Further, I can do things with the C preprocessor that are syntactically nothing at all like C. You already said you can’t do Lisp macros that are syntactically nothing like Lisp.

There is, of course, an aesthetic argument about why it would be better if the C preprocessor enforced some semblance of C on you. I wouldn’t respect that argument much either if someone were poking at my language. Aesthetics is good. Aesthetics is important. Should the compiler enforce them? Only if it’s going to give me something much better in return.

What do Lisp macros give you? They give you Lisp! What more could you possible want?

Let’s say that you’ve written code in Assembly Language or BASIC so long that the code you’re writing today screams for a Jump table or computed GOTO. Heck, the code you wrote yesterday used a jump table, too? Maybe you could factor out jump tables. You could do this in C by creating a function:

void do_jump_table( int choice, void (*array_of_functions[])( void ) )
{
    (*(array_of_functions[choice]))();
}
...
void (*my_jump_table[])( void ) = { foo, bar, baz, cat, dog, horse, tick };
do_jump_table( choice, my_jump_table );

That will work for you as long it’s okay that all of your functions have the same signature, you don’t mind the overhead of the extra function call to do_jump_table(), you don’t mind making a separate function for each different jump table option, and your choice never stumbles past the end of the jump table. You can do it this way in Lisp if you like, too (without having to write a different do_jump_table for every different return-type you might like from your jump tables.

(defun do-jump-table (choice array-of-functions)
  (funcall (aref array-of-functions choice)))
...
(do-jump-table choice #< #'foo, #'bar, #'baz, #'cat, #'dog, #'horse, #'tick >)

What if you do care about the overhead of the extra function call to do_jump_table(), you do mind making a separate function for each different jump table option, or you are concerned about doing something graceful when your choice stumbles past the end of the jump table? You could address that last concern with more error checking and, in the C case, more parameters to the function. The other concerns are tougher though.

If we don’t want to have to write separate foo, bar, baz, etc. functions, then we want to try this with macros. Rather than taking an array of function pointers, we’re going to let it take some variable number of statements—one for each case in the jump table. Let’s start with the Lisp version:

(defmacro do-jump-table ( choice &rest options )
  `(case ,choice
     ,@(loop for ii from 0
            for oo in options
            collecting (list ii oo))
     (otherwise (do-something-graceful))))

Then, we could use it with some code like this:

(do-jump-table op-code
               (write "NOOP")
               (write "SYNC")
               (when (>= +protocol-version+ 1)
                 (write "QUIT")))

This would expand to:

(case op-code
  (0 (write "NOOP"))
  (1 (write "SYNC"))
  (2 (when (>= +protocol-version+ 1)
       (write "QUIT")))
  (otherwise (do-something-graceful)))

I challenge you to make such a macro in C, C++, TeX, m4, or nroff. I’ve written lots of custom C programs in the past to preprocess my code before sending it to the C preprocessor and the C compiler. In Lisp, it’s all right there. I can use all of Lisp’s magic to turn my code into the code I want it to become. No temporary files. No fancy Makefile tricks to get everything squared away. It’s just all there.

What don’t I know?

The do-jump-table macro we just defined expands to one statement (one lisp form): (CASE …). There are many times when I want my macro to expand to more than one form, however. This isn’t legit. I still end up wanting to do it. For example, I might think that since I’m doing things with macros, that I might want the whole (CASE …) construct to be in the main code and just have the macro expand the different cases:

(case op-code
    (do-jump-table-cases (write "NOOP")
                         (write "SYNC")
                         (when (>= +protocol-version+ 1)
                             (write "QUIT")))
    (otherwise (do-something-especially-graceful)))

Lisp says no. I can understand why it says that. In this case, it’s easy enough to work around. But, in other cases, I end up flailing.

I feel that once I understand Lisp macros more completely, I won’t keep trying to jam that gorgeous, round peg into all of these square holes in my logic.

Perl

I admit: I was surprised myself. Seriously, I tried several other languages first. Let’s just focus on the first loop here. Here’s the Perl again for reference:

my $total = 0.0;
foreach my $item ( @list_of_items ) {
    $total += weight( $item );
}

Objective C

The actual code that I was attempting to explain was in Objective C with the NeXTStep foundation classes. I couldn’t imagine that anyone would want to try to understand it:

double total = 0.0;
NSEnumerator* ee = [itemList objectEnumerator];
Item* item;
while ( ( item = (Item*)[ee nextObject] ) != nil ) {
    total += [item weight];
}

C

My first attempt for the article was to write it in C using an array of items.

double total = 0.0;
unsigned int ii;
for ( ii = 0; ii < itemCount; ++ii ) {
    total += weight( itemArray[ ii ] );
}

That’s not too terrible. There is, however, a great deal of syntax and code devoted to handling the array including an extra variable to track its length. Beyond that, the loop counter is of little use to someone trying to understand the concept.

C++

My next go was C++. My hope was that iterators would make the looping less painful. C++ iterators, however, are the very model of pain.

double total = 0.0;
std::list< Item* >::const_iterator it;
for ( it = itemList.begin(); it != itemList.end(); ++it ) {
    total += weight( *it );
}

This requires the reader to wade through a syntactic thicket to get the iterator into the picture at all. It also requires a certain comfort with pointers that one wouldn’t have coming from another language.

Java

My next attempt was Java. I hate Java. It may not be terrible for this code with the new looping constructs that came in Java 1.5. Alas, I quit coding it before I made it through this first loop.

Lisp

Of course, I wanted to code this in Lisp:

(let ((total (reduce #'+ list-of-items :key #'weight)))
   ...)

That would be perfectly clear to someone who has used Lisp or Scheme. Alas, not everyone has.

So… Perl…

If you want to write the Perl code, you have to understand the difference between scalars and arrays. To read the code, however, you don’t have to grok that distinction to see what’s going on. I didn’t try it in Python. Maybe it would be clear to a wider audience. I may have to try it next time.

In Lisp, I will pull something out into a separate function if it makes the current function more self-contained, more one idea. In C++, I will only pull something out into a separate function if I need the same functionality in multiple places.

Actually, it’s even worse than that in C++. For stuff that is less than six or so lines, I might maintain it in several functions. Or, if I’m using some Literate Programming tool, I will just use the same chunk in multiple places. The notable exception in C++ is when I want to use something as a loop conditional, I may bother to break it out into its
own function:

while ( incrementCounter( cntr, min, max, dimensions ) ) {
    // body of loop here
}

In C++ or Objective-C, I might do something like this:

// some portion of my function
 
unsigned int choice = random() % length;
void* currentChoice = options[ choice ];
    // yes, I know I can memcpy(), but that's not as obvious
for ( unsigned int ii=choice+1; ii < length; ++ii ) {
    options[ ii-1 ] = options[ ii ];
}
--length;
 
// some code using currentChoice

In Lisp, I would never consider keeping that code inline. I would put it in another function right away.

This, this is one of the things I meant when I said Lisp is just plain fun. It’s easy to make a new function. I can just do it. I don’t have to fret too much over the name. I don’t have to fret too much over the argument
list. I don’t have to pre-declare it in this header file with the same signature as that implementation file. I
don’t have to pretend its a method when it’s really just a function. I can return multiple values if I need to do so. I don’t have to worry much about which compilation units will need to see this to compile.

Part of the maze generation code that I wrote in Objective-C needs to track walls. I don’t need the same structure during generation that I will use once it’s generated. So, I have a Wall class declared. It feels wrong to declare it right in the header file for the Maze. It feels silly to break it out into its own header file. What I should be doing is making a separate MazeFactory and have its implementation include the declaration of this Wall class. But, that is such overkill here. I just want the darn maze.

In Lisp, I would just be done with no feelings of guilt at all.

There are a bunch of things that I wanted to do with it for a long time, but it’s been too slow and rigid to make any of those things fun.

Enter Lisp. As soon as it made it through my skull that Lisp is actually compiled (honest-to-goodness your-CPU instructions), I wanted to rewrite the whole thing in Lisp. I have finally gotten started on doing that. And, I just made it to the point where I’m actually tracing rays. Here is a stereo pair of a three-dimensional scene:

It use’s xach‘s ZPNG library for output, my OpenMPI library for sharing work across machines, and a thin layer that I wrote on top of Portable Threads for threading within a machine.

It doesn’t yet do reflections and refractions, directional lights, or most of the shapes that my old raytracer does.
But, it’s already got multithreading, MPI, more meaningful camera parameters, and functional color characteristics.

So, here’s the source code that generated the above images. Note that the one sphere has checkboarded diffuseness and the other has gradated phong-exponent and positional coloring.

(defun make-checkerboard-func (a b)
  #'(lambda (&key position)
      (let ((sum (reduce #'+
			 position
			 :key #'(lambda (vv)
				  (if (< 1.0 (mod vv 2.0)) 1 0)))))
	(if (evenp sum) a b))))
 
(defun my-universe ()
  (rt:universe (:spatial-dimensions 3
		:color-dimensions 3)
    (let ((look-at (rt:v 18.0 0.0 0.0))
	  (eye-offset (rt:v 0.0 0.25 0.0))
	  (aspect (rt:v 16.0 9.0))
	  (field-of-view 120.0)
	  (object-scale (rt:v 3.0 3.0 3.0))
	  (x-axis (rt:v 1.0 0.0 0.0))
	  (y-axis (rt:v 0.0 1.0 0.0))
	  (z-axis (rt:v 0.0 0.0 1.0))
	  (angle 10.0))
      (rt:with-transforms (:translation (rt:v* eye-offset -1.0)
			   :look-at look-at)
	(rt:camera :name :main-camera-l
		   :aspect aspect
		   :field-of-view field-of-view))
      (rt:with-transforms (:translation (rt:v* eye-offset 1.0)
			   :look-at look-at)
	(rt:camera :name :main-camera-r
		   :aspect aspect
		   :field-of-view field-of-view))
      (rt:with-translation (rt:v 0.0 -5.0 -2.0)
	(rt:light :color (rt:v 0.7 0.7 0.7)))
      (rt:with-translation (rt:v -10.0 -5.0 8.0)
	(rt:light :color (rt:v 0.7 0.7 1.0)))
      (rt:with-transforms (:translation look-at
			   :scaling object-scale
			   :look-at (rt:v 0.0 0.0 0.0)
			   :translation (rt:v 0.0 0.0 -1.0)
			   :rotation (x-axis y-axis angle))
	(rt:with-transforms (:translation (rt:v 0.0 -1.5 0.0)
			     :color-scaling (rt:v 0.75 0.75 0.75)
			     :color-rotation (y-axis z-axis angle))
	  (rt:with-color (rt:c (rt:v 0.2 0.8 0.2)
			       :diffuseness (make-checkerboard-func 0.2 0.6)
			       :specularity 0.4)
	    (rt:sphere)))
	(rt:with-transforms (:translation (rt:v 2.0 2.0 1.5)
			     :scaling (rt:v 1.5 1.5 1.5)
			     :color-translation (rt:v 0.0 0.2 0.0))
	  (rt:with-color (rt:c #'(lambda (&key position)
				   (map 'vector #'abs position))
			       :diffuseness 0.6
			       :specularity 0.4
			       :phong-exponent #'(lambda (&key position)
						   (+ 10
						      (abs (* 100
							     (aref position 2)
							     )))))
	      (rt:sphere)))
	(rt:with-rotation (x-axis y-axis angle)
	  (rt:with-translation (rt:v -15.0 0.0 0.0)
	    (rt:with-color (rt:c (rt:v 0.8 0.2 0.2))
	      (rt:halfspace :name :hspace1))))))))
 
#+:openmpi
(mpi:init)
 
(rt:with-workers (1)
   (let ((uu (my-universe))
	 (dpd (rt:v 2.0 2.0)))
     (rt:render-png :dots-per-degree dpd
		    :universe uu
		    :filename #P"output-l.png"
		    :camera-name :main-camera-l)
     (rt:render-png :dots-per-degree dpd
		    :universe uu
		    :filename #P"output-r.png"
		    :camera-name :main-camera-r)))
 
#+:openmpi
(mpi:finalize)

And, here are the same images swapped for those of you who prefer cross-eyed stereo pairs: