Coder’s Asceticism October 14th, 2012
Patrick Stein

A month or two ago, I watched a bunch of videos of Uncle Bob speaking at various conferences. He’s thought a lot about what makes code testable and maintainable, and he’s an excellent speaker. I’ve even paid to watch some of his hour-ish videos at his Clean Coders site.

In Clean Code, Episode 3 – Functions, he makes the case that we should be striving to keep each function under five lines long. So, over the past month or so, I’ve been trying this out in all of my Scala and Lisp endeavors.

Once you get each function down to four lines or fewer, you start running into symbol-proliferation. You pretty much throw out (LABELS ...) and (FLET ...) unless you need the closure (and then try to say that when you can’t that it’s good enough to just keep the subfunctions under five lines and the body under five, too). You create bunches of functions that should never be called anywhere except the one place it’s called now.

Enter Package Proliferation!

Then, I remembered some Lisp coding style guidelines that I read early this year sometime. Those guidelines advocated having one package for each Lisp file and explicitly making peer files have to import or qualify symbols used from other peer files. This will help me with the symbol-proliferation! Let’s do it!

Enter File Proliferation!

After two weeks of that, I woke up yesterday morning realizing that I wasn’t getting the full benefit from this if I was exporting more than one symbol from each package. If keeping each function tiny is Good and limiting a package to one file is Good, then it is certainly Bad to have one file’s package exporting eight different functions.

So, today, I reworked all of the UNet that I wrote in the last couple of weeks to have one package per file and as few exported symbols per package as possible.

In the interest of testability and modularity, I had broken out a subpackage of UNet called UNet-Sockets that is to be the interface the UNet library uses to interact with sockets. I had realized that I had a file called methods.lisp with four different publicly available (DEFMETHOD ...) forms in it. Now, each is in its own file and package. I did the same for various other declarations.

Now, there is a top-level package which uses Tim Bradshaw’s Conduit Packages to collect all of the symbols meant to be used externally from these packages. If there is an exported function in a package, that is the only exported symbol in that package. If there is an exported generic function declaration in a package, that is the only exported symbol from that package. If there is an exported method implementation in a package, there aren’t any exported symbols from that package. If there is an exported condition in a package, that condition and its accessors are the only exported symbols from that package. If there is an exported class in a package, that class and its accessors are the only exported symbols from that package.

Example

The top-level package of the UNET-SOCKETS system, that defines the interface that UNet will use to access its sockets, consists of just a (DEFPACKAGE ...) form. You can see the full package on github in the unet-sockets.asd and the sockets/ directory.

(org.tfeb.conduit-packages:defpackage :unet-sockets
  (:use :cl)
  (:extends/including :unet-sockets-interface
                      #:sockets-interface)
 
  (:extends/including :unet-sockets-base-socket
                      #:base-socket)
 
  (:extends/including :unet-sockets-hostname-not-found-error
                      #:hostname-not-found-error
                      #:hostname-not-found-error-hostname)

  (:extends/including :unet-sockets-get-address
                      #:get-address)

  (:extends/including :unet-sockets-address+port-not-available-error
                      #:address+port-not-available-error
                      #:address+port-not-available-error-address
                      #:address+port-not-available-error-port)

  (:extends/including :unet-sockets-create-datagram-socket
                      #:create-datagram-socket)
 
  (:extends/including :unet-sockets-send-datagram
                      #:send-datagram)
 
  (:extends/including :unet-sockets-poll-datagram
                      #:poll-datagram))

Conclusion?

The jury’s still out on this for me. On the one hand, the self-flagellation is an interesting exercise. On the other hand, if I’m going to have to export everything useful and import it in multiple places then I’m taking away some of the fun. I feel like I’m maintaining C++ header files to some extent.

I think I’ll keep it up for the UNet project, at least. It makes good documentation to have each exported symbol in a package and file named for that exported symbol. You can open up any file and read the three-to-four line exported function in it. You can see, right there with it, any supporting function it uses that no other files need to use. You can see from the (DEFPACKAGE ...) form at the top where any other supporting functions came from. And nothing else is cluttering up the landscape.

We’ll see if I develop religion on this topic. At the moment, I think it’s eventually going to fall into the moderation in all things bucket. Time will tell.

Delayed Evaluation Across Packages April 24th, 2011
Patrick Stein

For my networking layer library, I wanted to provide ubiquitous logging. At the same time, I did not want to tie the application to my choice of logging library. I wanted the user to be able to pass me a function where I could give them a logging category and something to log.

(in-package :unet)

(defvar *logger-function* nil)

(defmacro log-it (category thing-to-log)
  `(when *logger-function*
     (funcall *logger-function* ,category ,thing-to-log)))

This seems simple enough, right? Now, throughout my code, I can do things like:

(log-it :incoming-packet packet)
(log-it :list-of-unacked-packets (get-list-of-unacked-packets))
(etc)

The application can register a *logger-function* something like this:

(defun app-log-network-msgs (category thing-to-log)
  (cl-log:log-message category thing-to-log))

Here’s the problem though: most (all?) logging libraries are good about not evaluating any arguments beyond the category unless something is actually listening for messages of that category. This makes it reasonable to do stuff like this and only take the speed hit when it’s actually important to do so:

(log-it :excruciating-detail
        (mapcar #'get-excrutiating-detail (append everyone everything)))

With my macro above, I have no way of knowing whether something is listening on a particular category or not. Further, most logging libraries don’t offer a way to query that sort of information.

What to do?

What I wanted to do was to pass a macro from the application package into my networking library instead of passing a function. I spent way too long trying to find a way to make this work (especially considering I ran into the same trouble in October, 2009 trying to use some combination of (macroexpand ...) and (eval ...) to let the caller decide which forms to execute).

All it took was posting to comp.lang.lisp to answer my own question: Closures. I changed my macro to create a closure:

(defmacro log-it (category thing-to-log)
  `(when *logger-function*
     (funcall *logger-function* ,category #'(lambda () ,thing-to-log))))

Now, the application’s logger function changes slightly and my forms are only evaluated when the logging library evaluates them:

(defun app-log-network-msgs (category thing-to-log-generator)
  (cl-log:log-message category (funcall thing-to-log-generator))

Hopefully, I will remember next time I run into this.

USerial — v0.4.2011.04.11 April 12th, 2011
Patrick Stein

Made an addition to the USerial library to support logging of binary messages (so far only cl-log supported).

(serialize-log :log-category :uint32 uint-to-log :string "string to log" ...)

Here is the latest tarball: userial_0.4.2011.04.11.tar.gz and its signature: userial_0.4.2011.04.11.tar.gz.asc.

Binary Logging with CL-Log April 6th, 2011
Patrick Stein

One of the things that my current work does better than anywhere I’ve worked before is logging. When something goes wrong, there is a log file that you can dig through to find all kinds of information about what you were doing and how things were going.

As I move forward programming a game with my UNet library, I want to make sure that I can easily log all the network traffic during testing runs at least.

In looking through the various Lisp logging packages out there, I decided on Nick Levine’s cl-log library.

I installed it in no time with quicklisp.

Then, I set to work trying to figure out how I could use it to log binary data.

Here’s what I ended up with. If you want to do something similar, this should give you a good starting point.

Serializing, unserializing, and categorizing

With my USerial library, I defined a serializer to keep track of the different categories of log messages. And, I made corresponding categories in cl-log.

(make-enum-serializer :log-category (:packet :error :warning :info))

(defcategory :packet)
(defcategory :error)
(defcategory :warning (or :error :warning))
(defcategory :info (or :warning :info))

Specializing the classes

There are two major classes that I specialized: base-message and base-messenger. For my toying around, I didn’t end up adding any functionality to the base-message class. I will show it here though so that you know you can do it.

(defclass serialized-message (base-message)
  ())

(defclass serialized-messenger (base-messenger)
  ((filename :initarg :filename :reader serialized-messenger-filename)))

Then, I overrode the messenger-send-message generic function to create a binary header with my USerial library and then write the header and the message out.

(defmethod messenger-send-message ((messenger serialized-messenger)
                                   (message serialized-message))
  (let ((header (make-buffer 16)))
    (serialize* (:uint64 (timestamp-universal-time
                              (message-timestamp message))
                 :log-category (message-category message)
                 :uint64 (buffer-length :buffer (message-description message)))
             :buffer header)
    (with-open-file (stream (serialized-messenger-filename messenger)
                            :direction :output
                            :if-does-not-exist :create
                            :if-exists :append
                            :element-type '(unsigned-byte 8))
      (write-sequence header stream)
      (write-sequence (message-description message) stream))))

Using it

To get things going, I then made a log manager that accepts my serialized-message type and started one of my serialized-messenger instances.

(setf (log-manager)
      (make-instance 'log-manager
                     :message-class 'serialized-message))

(start-messenger 'serialized-messenger :name "binary-logger"
                                       :filename "/tmp/binary-log.dat")

Once these were started, I made a little utility function to make it easy for me to make test messages and then invoked log-message a few times.

(defun make-info (string)
  (serialize :string string :buffer (make-buffer)))

(log-message :warning (make-info "Warning"))
(log-message :info (make-info "This is info"))

Conclusions

In all, it has taken me about four times as long to write blog post as it did to install cl-log with quicklisp, peek through the cl-log documentation and source code enough to figure out how to do this, and write all of the code.

To really use this, I will probably separate out the category of a message from the serialized type of the message. This will probably involve adding a field to the serialized-message class to track the message type, adding an initialize-instance :before method for that class to look through the arguments to pull out the type, and then adding the type as an extra argument to log-message.

USerial Library — v0.3.2011.03.05 March 4th, 2011
Patrick Stein

I have released a new version of my serialization library. I hope no one has dug in too far on using it yet because I rearranged the interface a fair bit in this release. To accommodate more complex serializers and unserializers as well as supporting a with-buffer macro, the buffer is no longer the first argument to the serialize and unserialize methods. Now, it is a &key argument to the serialize and unserialize generics. Further, the serialize and unserialize generics also &allow-other-keys.

In an intervening and unannounced release, I added serializers for slots and accessors.

In this release, I have also really fleshed out the documentation and examples.

For instructions on obtaining and using the USerial library, please refer to the USerial library web page.

Edit: This had been v0.3.2011.03.04, but I made a minor update to add MIT License and correct a few glitches in the docs. Now, it’s v0.3.2011.03.05.

RSS Feeds

l