Today, working on it, I’m not quite sure what’s up, but I am not succeeding in communicating with the server using encryption. I’ll have to look more closely. Last time that I worked on it, I extended Ironclad, but I never got those changes pushed fully into Ironclad’s main line. But, I think I’m using the same version of Ironclad that I was using when I tested against the Windows Growl Application. *shrug*

I’ve also run into a snag with the Callbacks. Essentially, your Lisp program could get a callback when the user has clicked on your Growl notification. This actually works except for the fact that I am calling READ-SEQUENCE into a buffer that is longer than the message. The server, I believe, is supposed to close the socket after the callback. But, it does not. So, I am stuck waiting for more bytes that will never come.

Now, I either have to do one of the following:

• refactor it to use READ-LINE instead
• switch from using USocket to using IOLib (and hope that :dont-wait works as expected)
• extend USocket to support SOCKET-RECEIVE even on TCP sockets

Anyone have a preference?

Whenever I write client-server applications, I run into the same problem trying to separate the code. To send a message from the server to the client, the server has to serialize that message and the client has to unserialize that message. The server doesn’t need to unserialize that message. The client doesn’t need to serialize that message.

It seems wrong to include both the serialization code and the unserialization code in both client and server when each side will only be using 1/2 of that code. On the other hand, it seems bad to keep the serialization and unserialization code in separate places. You don’t want one side serializing A+B+C and the other side trying to unserialize A+C+D+B.

One approach: data classes

Some projects deal with this situation by making every message have its own data class. You take all of the information that you want to be in the message and plop it into a data class. You then serialize the data class and send the resulting bytes. The other side unserializes the bytes into a data class and plucks the data out of the data class.

The advantage here is that you can have some metaprogram read the data class definition and generate a serializer or unserializer as needed. You’re only out-of-sync if one side hasn’t regenerated since the data class definition changed.

The disadvantage here is that I loathe data classes. If my top-level interface is going to be (send-login username password), then why can’t I just serialize straight from there without having to create a dippy data structure to hold my opcode and two strings?

Another approach: suck it up

Who cares if the client contains both the serialization and unserialization code? Heck, if you’re really all that concerned, then fmakunbound half the universe before you save-lisp-and-die.

Of course, unless you’re using data classes, you’re either going to have code in your client that references a bunch of functions and variables that only exist in your server or your client and server will be identical except for:

(defun main ()
  #+server (server-main)
  #-server (client-main))

Now, of course, your server is going to accidentally depend on OpenGL and OpenAL and SDL and a whole bunch of other -L’s it never actually calls. Meanwhile, your client is going to accidentally depend on Postmodern and Portable-Threads and a whole bunch of other Po-’s it never actually calls.

Another approach: tangle and weave, baby

Another way that I’ve got around this is to use literate programming tools to let me write the serialiization and unserialization right next to each other in my document. Then, anyone going to change the serialize code would be immediately confronted with the unserialize code that goes with it.

The advantage here is that you can tangle the client code through an entirely separate path than the server code keeping only what you need in each.

The disadvantage here is that now both your client code and your server code have to be in the same document or both include the same sizable chunk of document. And, while there aren’t precisely name-capturing problems, trying to include the “serialize-and-send” chunk in your function in the client code still requires that you use the same variable names that were in that chunk.

How can Lisp make this better?

In Lisp, we can get the benefits of a data-definition language and data classes without needing the data classes. Here’s a snippet of the data definition for a simple client-server protocol.

;;;; protocol.lisp
(userial:make-enum-serializer :opcode (:ping :ping-ack))
(defmessage :ping     :uint32 ping-payload)
(defmessage :ping-ack :uint32 ping-payload)

I’ve declared there are two different types of messages, each with their own opcode. Now, I have macros for define-sender and define-handler that allow me to create functions which have no control over the actual serialization and unserialization. My functions can only manipulate the named message parameters (the value of ping-payload in this case) before serialization or after unserialization but cannot change the serialization or unserialization itself.

With this protocol, the client side has to handle ping messages by sending ping-ack messages. The define-sender macro takes the opcode of the message (used to identify the message fields), the name of the function to create, the argument list for the function (which may include declarations for some or all of the fields in the message), the form to use for the address to send the resulting message to, and any body needed to set fields in the packet based on the function arguments before the serialization. The define-handler macro takes the opcode of the message (again, used to identify the message fields), the name of the function to create, the argument list for the function, the form to use for the buffer to unserialize, and any body needed to act on the unserialized message fields.

;;;; client.lisp
(define-sender  :ping-ack send-ping-ack (ping-payload) *server-address*)
(define-handler :ping     handle-ping   (buffer) buffer
  (send-ping-ack ping-payload))

The server side has a bit more work to do because it’s going to generate the sequence numbers and track the round-trip ping times.

;;;; server.lisp
 
(defvar *last-ping-payload* 0)
(defvar *last-ping-time*    0)
 
(define-sender :ping send-ping (who) (get-address-of who)
  (incf *last-ping-payload*)
  (setf *last-ping-time*    (get-internal-real-time)
        ping-payload        *last-ping-payload*))
 
(define-handler :ping-ack handle-ping-ack (who buffer) buffer
  (when (= ping-payload *last-ping-payload*)
    (update-ping-time who (- (get-internal-real-time) *last-ping-time*))))

Problems with the above

It feels strange to leave compile-time artifacts like the names and types of the message fields in the code after I’ve generated the functions that I’m actually going to use. But, I guess that’s just part of Lisp development. You can’t (easily) unload a package. I can makunbound a bunch of stuff after I’m loaded if I don’t want it to be convenient to modify senders or handlers at run-time.

There is intentional name-capture going on. The names of the message fields become names in the handlers. The biggest problem with this is that the defmessage calls really have to be in the same namespace as the define-sender and define-handler calls.

I still have some work to do on my macros to support &key and &optional and &aux and &rest arguments properly. I will post those macros once I’ve worked out those kinks.

Anyone care to share how they’ve tackled client-server separation before?

After a whole bunch of running in circles, I have discovered a combination that works (with these clues). I have my TERM variable set to xterm-color. I configured the Terminal.app using its Keyboard settings to have it send the string “\033[1;5C” for C-<right arrow> and “\033[1;5D” for C-<left arrow>. (The “\033″ is the escape key.)

This works for me even through screen.

Bonus.

• Fix (make-int-serializer) to be big-endian 2′s complement
• Add :symbol and :keyword serializers
• Add (make-vector-serializer)
• Add (make-key-slot-serializer) and (make-key-accessor-serializer)
• Add (define-serializing-funcall)

Obtaining

• The USerial home page: http://nklein.com/software/unet/userial/
• The tarball: userial_0.8.2011.06.02.tar.gz
• The signature: userial_0.8.2011.06.02.tar.gz.asc
• The main git repository: http://git.nklein.com/lisp/libs/userial.git/
• Browsable git mirror: https://github.com/nklein/userial
• Issue reporting: https://github.com/nklein/userial/issues

Here’s one slide from that presentation:

Today, I started from scratch in an empty account on an eight-year-old Linux box. I installed Quicklisp and used it to install Weblocks. I had a Weblocks server up and running in under three minutes.

Obtaining

Getting the USerial library:

• The home page: http://nklein.com/software/unet/userial/
• The tar-ball: userial_0.7.2011.05.24.tar.gz
• The GPG signature for the tar-ball: userial_0.7.2011.05.24.tar.gz.asc
• The main git repository: http://git.nklein.com/lisp/libs/userial.git
• A browsable mirror of the git repository: http://github.com/nklein/userial

Differences

The differences between this version and earlier versions of this library include:

• Use of ContextL layered functions instead of CLOS methods
• Elimination of :buffer parameter in favor of using the *buffer* special variable
• Cleaning up macros which no longer required the :buffer parameter
• Serializers for arbitrarily large integers and unsigned integers
• Serializer for raw sequence of bytes
• New make-list-serializer macro

By using ContextL layered functions, one has the ability to define a serializer and/or unserializer in a particular ContextL layer. This can be used to create new versions of the serializer without losing the ability to use the older version when required.

In the process, I have created macros to assist in creating completely custom serializers. This both streamlines their definition and should allow any future modifications to the USerial library to fly under the radar. Code that before looked like this:

(defmethod serialize ((key (eql :foo)) (value foo-struct)
                      &key (buffer userial:*buffer*) &allow-other-keys)
  ... some code ...
  buffer)
 
(defmethod unserialize ((key (eql :foo))
                        &key (buffer userial:*buffer*) &allow-other-keys)
  (values (progn ... some code ...)
          buffer))

Should now look like this:

(define-serializer (:foo (value foo-struct))
  ... some code ...)
 
(define-unserializer (:foo)
  ... some code ...)

And, when you find you need to add a new version of your :foo serializer but you don’t want to lose the old one, you can add:

(contextl:deflayer new-version)
 
(define-serializer (:foo (value foo-struct) :layer new-version)
  ... some new code ...)

Without the :buffer parameter everywhere, code that used to look like this:

(serialize* (:string aa :uint8 bb) :buffer buf)
(buffer-rewind :buffer buf)
(unserialize-slots* (:string name :uint8 age) object :buffer buf)

Should now look like this:

(with-buffer buf
  (serialize* :string aa :uint8 bb)
  (buffer-rewind)
  (unserialize-slots* object :string name :uint age))

There are now :int and :uint serializers that encode arbitrarily large integers and unsigned integers, respectively. There is also a serializer that copies a sequence of bytes as is without any prefix or suffix. To unserialize, you either have to provide a buffer of the appropriate length with the :output parameter or provide appropriate :start and :end keywords.

(serialize :raw-bytes uchar-array &key (start 0)
                                       (end (length uchar-array)))
(unserialize :raw-bytes &key output
                             (start 0)
                             (end (length output)))

And, if you have a serialize/unserialize pair for type :foo you can use the make-list-serializer macro to create a serialize/unserialize pair for a list of items that can be serialized with the :foo serializer.

(make-list-serializer :list-of-uint8 :uint8)
(serialize :list-of-int8 '(0 1 1 2 3 5 8 13 21 34 55 89))

At the USerial home page, you can find more complete documentation.

Obtaining

Getting the USerial library:

• The home page: http://nklein.com/software/unet/userial/
• The tar-ball: userial_0.7.2011.05.24.tar.gz
• The GPG signature for the tar-ball: userial_0.7.2011.05.24.tar.gz.asc
• The main git repository: http://git.nklein.com/lisp/libs/userial.git
• A browsable mirror of the git repository: http://github.com/nklein/userial

Differences

The differences between this version and earlier versions of this library include:

• Use of ContextL layered functions instead of CLOS methods
• Elimination of :buffer parameter in favor of using the *buffer* special variable
• Cleaning up macros which no longer required the :buffer parameter
• Serializers for arbitrarily large integers and unsigned integers
• Serializer for raw sequence of bytes
• New make-list-serializer macro

By using ContextL layered functions, one has the ability to define a serializer and/or unserializer in a particular ContextL layer. This can be used to create new versions of the serializer without losing the ability to use the older version when required.

In the process, I have created macros to assist in creating completely custom serializers. This both streamlines their definition and should allow any future modifications to the USerial library to fly under the radar. Code that before looked like this:

(defmethod serialize ((key (eql :foo)) (value foo-struct)
                      &key (buffer userial:*buffer*) &allow-other-keys)
  ... some code ...
  buffer)
 
(defmethod unserialize ((key (eql :foo))
                        &key (buffer userial:*buffer*) &allow-other-keys)
  (values (progn ... some code ...)
          buffer))

Should now look like this:

(define-serializer (:foo (value foo-struct))
  ... some code ...)
 
(define-unserializer (:foo)
  ... some code ...)

And, when you find you need to add a new version of your :foo serializer but you don’t want to lose the old one, you can add:

(contextl:deflayer new-version)
 
(define-serializer (:foo (value foo-struct) :layer new-version)
  ... some new code ...)

Without the :buffer parameter everywhere, code that used to look like this:

(serialize* (:string aa :uint8 bb) :buffer buf)
(buffer-rewind :buffer buf)
(unserialize-slots* (:string name :uint8 age) object :buffer buf)

Should now look like this:

(with-buffer buf
  (serialize* :string aa :uint8 bb)
  (buffer-rewind)
  (unserialize-slots* object :string name :uint age))

There are now :int and :uint serializers that encode arbitrarily large integers and unsigned integers, respectively. There is also a serializer that copies a sequence of bytes as is without any prefix or suffix. To unserialize, you either have to provide a buffer of the appropriate length with the :output parameter or provide appropriate :start and :end keywords.

(serialize :raw-bytes uchar-array &key (start 0)
                                       (end (length uchar-array)))
(unserialize :raw-bytes &key output
                             (start 0)
                             (end (length output)))

And, if you have a serialize/unserialize pair for type :foo you can use the make-list-serializer macro to create a serialize/unserialize pair for a list of items that can be serialized with the :foo serializer.

(make-list-serializer :list-of-uint8 :uint8)
(serialize :list-of-int8 '(0 1 1 2 3 5 8 13 21 34 55 89))

At the USerial home page, you can find more complete documentation.

With my library, you set up some versions and a variable to track the current version:

(method-versions:define-method-version :v1.0)
(method-versions:define-method-version :v1.1 :v1.0)
 
(declaim (special *protocol-version*))
(defparameter *protocol-version* :v1.0)

In ContextL, you just set up some layers:

(contextl:deflayer :v1.0)
(contextl:deflayer :v1.1 (:v1.0))

In my library, you then set up a generic function that uses a special method combination that keys off of the special variable:

(defgeneric send-cmd (cmd)
  (:method-combination method-versions:method-versions-method-combination
                       *protocol-version*))

In ContextL, you declare a layered function:

(contextl:define-layered-function (cmd))

In my library, you then declare different methods using the version as a method qualifier.

(defmethod send-cmd ((cmd login-cmd))
  (send-string (login-name cmd))
  (send-string (login-password cmd)))
 
(defmethod send-cmd :v1.0 ((cmd login-cmd))
  (send-string (login-name cmd))
  (send-string (login-password cmd))
  (send-string (login-location cmd)))

In ContextL, you declare layered methods specifying which layer the functions belong to:

(contextl:define-layered-method send-cmd ((cmd login-cmd))
  (send-string (login-name cmd))
  (send-string (login-password cmd)))
 
(contextl:define-layered-method send-cmd :in :v1.0 ((cmd login-cmd))
  (send-string (login-name cmd))
  (send-string (login-password cmd))
  (send-string (login-location cmd)))

In my library, you set your special variable appropriately and invoke the method:

(let ((*protocol-version* :v1.1))
  (send-cmd cmd))

In ContextL, you declare which layer you want to be active when you go to invoke the method:

(contextl:with-active-layers (:v1.1)
  (send-cmd cmd))

My library does not let you specify other method qualifiers like :around or :after. ContextL does.

I am going to leave my library published because I think it is a reasonably understandable, yet non-trivial, use of non-standard method combinations. However, I am going to end up using ContextL for the projects that I had intended for my library.

Edit: After re-reading some of the ContextL papers, I believe that I am actually just going to use ContextL as it’s a much more flexible superset of this library. I will probably still keep this library published as an example of a non-trivial, but glarkable, method combination.

I am releasing a new library that allows one to dispatch generic methods based on the value of a global parameter.

There are situations where one might like to dispatch a method on some information other than the required parameters of the method. For many situations, it is sufficient to switch between those methods based on some external parameter. The method-versions library allows one to do just that.

Obtaining

• The home page: http://nklein.com/software/method-versions/
• The tar-ball: method-versions_0.1.2011.05.18.tar.gz
• The GPG signature for the tar-ball: method-versions_0.1.2011.05.18.tar.gz.asc
• The main git repository: http://git.nklein.com/lisp/libs/method-versions.git
• A browsable mirror of the git repository: http://github.com/nklein/method-versions

Internationalization Example

In this example, we do a silly form of internationalization. To that end, we will use English as the default language and define some other languages.

 (method-versions:define-method-version latin)
 (method-versions:define-method-version pig-latin)
 (method-versions:define-method-version french latin)
 (method-versions:define-method-version spanish latin)

We will prepare a language parameter and a welcome method that is versioned on the language.

 (declaim (special *language*))
 (defparameter *language* nil)
 
 (defgeneric welcome ()
   (:method-combination method-versions:method-version-method-combination
                        *language*))

And, we define welcome methods for the various languages (accidentally forgetting spanish).

 (defmethod welcome () :welcome)
 (defmethod welcome :latin     () :velkominum)
 (defmethod welcome :pig-latin () :elcomeway)
 (defmethod welcome :french    () :bonjour)

Then, we will try each of the languages in turn.

 (mapcar #'(lambda (ll)
             (let ((*language* ll))
               (welcome)))
         '(nil :latin :pig-latin :french :spanish))
 => (:welcome :velkominum :elcomeway :bonjour :velkominum)