lisp | nklein software

Nick Levine’s cl-log rocks… December 10th, 2012
Patrick Stein

Whenever I need logging in my Common Lisp applications, I use Nick Levine’s cl-log library. For many applications, spending the time to run CL:FORMAT during runtime is unreasonable. It is often much better to log something in a raw form and do any formatting/presentation offline rather than online. The cl-log library is the only one that I know of which supports binary logging readily.

Additionally, Nick is incredibly responsive. I found an oversight in a corner-case of cl-log last Friday at 1am. I emailed him about it at 1:19am. In under 5 hours, I had email back pointing me to a new release with that case corrected.

I hope to do a more complete comparison of the existing log libraries at some point. At this point, know that I’ve tried all of the ones on the Cliki which had obvious documentation or examples, and I’ll be using cl-log.

Coder’s Asceticism October 14th, 2012
Patrick Stein

A month or two ago, I watched a bunch of videos of Uncle Bob speaking at various conferences. He’s thought a lot about what makes code testable and maintainable, and he’s an excellent speaker. I’ve even paid to watch some of his hour-ish videos at his Clean Coders site.

In Clean Code, Episode 3 – Functions, he makes the case that we should be striving to keep each function under five lines long. So, over the past month or so, I’ve been trying this out in all of my Scala and Lisp endeavors.

Once you get each function down to four lines or fewer, you start running into symbol-proliferation. You pretty much throw out (LABELS ...) and (FLET ...) unless you need the closure (and then try to say that when you can’t that it’s good enough to just keep the subfunctions under five lines and the body under five, too). You create bunches of functions that should never be called anywhere except the one place it’s called now.

Enter Package Proliferation!

Then, I remembered some Lisp coding style guidelines that I read early this year sometime. Those guidelines advocated having one package for each Lisp file and explicitly making peer files have to import or qualify symbols used from other peer files. This will help me with the symbol-proliferation! Let’s do it!

Enter File Proliferation!

After two weeks of that, I woke up yesterday morning realizing that I wasn’t getting the full benefit from this if I was exporting more than one symbol from each package. If keeping each function tiny is Good and limiting a package to one file is Good, then it is certainly Bad to have one file’s package exporting eight different functions.

So, today, I reworked all of the UNet that I wrote in the last couple of weeks to have one package per file and as few exported symbols per package as possible.

In the interest of testability and modularity, I had broken out a subpackage of UNet called UNet-Sockets that is to be the interface the UNet library uses to interact with sockets. I had realized that I had a file called methods.lisp with four different publicly available (DEFMETHOD ...) forms in it. Now, each is in its own file and package. I did the same for various other declarations.

Now, there is a top-level package which uses Tim Bradshaw’s Conduit Packages to collect all of the symbols meant to be used externally from these packages. If there is an exported function in a package, that is the only exported symbol in that package. If there is an exported generic function declaration in a package, that is the only exported symbol from that package. If there is an exported method implementation in a package, there aren’t any exported symbols from that package. If there is an exported condition in a package, that condition and its accessors are the only exported symbols from that package. If there is an exported class in a package, that class and its accessors are the only exported symbols from that package.

Example

The top-level package of the UNET-SOCKETS system, that defines the interface that UNet will use to access its sockets, consists of just a (DEFPACKAGE ...) form. You can see the full package on github in the unet-sockets.asd and the sockets/ directory.

(org.tfeb.conduit-packages:defpackage :unet-sockets
(:use :cl)
(:extends/including :unet-sockets-interface
#:sockets-interface)

(:extends/including :unet-sockets-base-socket
#:base-socket)

(:extends/including :unet-sockets-hostname-not-found-error
#:hostname-not-found-error
#:hostname-not-found-error-hostname)

(:extends/including :unet-sockets-get-address
#:get-address)

(:extends/including :unet-sockets-address+port-not-available-error
#:address+port-not-available-error
#:address+port-not-available-error-address
#:address+port-not-available-error-port)

(:extends/including :unet-sockets-create-datagram-socket
#:create-datagram-socket)

(:extends/including :unet-sockets-send-datagram
#:send-datagram)

(:extends/including :unet-sockets-poll-datagram
#:poll-datagram))

Conclusion?

The jury’s still out on this for me. On the one hand, the self-flagellation is an interesting exercise. On the other hand, if I’m going to have to export everything useful and import it in multiple places then I’m taking away some of the fun. I feel like I’m maintaining C++ header files to some extent.

I think I’ll keep it up for the UNet project, at least. It makes good documentation to have each exported symbol in a package and file named for that exported symbol. You can open up any file and read the three-to-four line exported function in it. You can see, right there with it, any supporting function it uses that no other files need to use. You can see from the (DEFPACKAGE ...) form at the top where any other supporting functions came from. And nothing else is cluttering up the landscape.

We’ll see if I develop religion on this topic. At the moment, I think it’s eventually going to fall into the moderation in all things bucket. Time will tell.

Polylinguistic Code Improvement September 24th, 2012
Patrick Stein

I have started the Coursera course: Functional Programming Principles in Scala. The first assignment was a few simple recursion problems: calculate a given entry in Pascal’s triangle, verify that the parens are balanced in a list of characters, and count the number of ways one can give change for a given amount with given coin denominations. I did those assignments six days ago.

Last night, for grins, I thought I would implement them in Common Lisp. I started out doing them without referencing the Scala code that I’d written. Then, I got lazy and pulled up the Scala code for the last problem and a half.

For all three problems, I came up with better solutions that translated back into the Scala. Some of it was just the benefit of looking at the previous solution again. Some of it was thinking about it with a new language in mind. Some of it was being more comfortable with Lisp. It was such a dramatic improvement in the code that even though I had scored 10/10 six days ago, I still reworked the Scala code and resubmitted it last night.

As it relates to Common Lisp development specifically…. One of the first Lisp books that I read was Paul Graham’s On Lisp. In that book, he does a great deal of turning recursive functions into tail-recursive functions. I had followed his idiom of calling the internal function rec:

(defun factorial (n)
(labels ((rec (n accum)
(if (zerop n)
accum
(rec (1- n) (* n accum)))))
(rec n 1)))

For the Scala, I started out naming the internal function with a Rec suffix on the function name:

def factorial (n: Int): Int =
def factorialRec (n: Int, accum: Int): Int =
if (n == 0) accum
else factorialRec(n-1, n*accum)
factorialRec(n,1)

Now, I’m partial to just re-using the function name:

(defun factorial (n)
(labels ((factorial (n accum)
(if (zerop n)
accum
(factorial (1- n) (* n accum)))))
(factorial n 1)))

Visualizing Galois Fields (Follow-up) May 21st, 2012
Patrick Stein

In my previous article, I should have finished by remapping the multiplication and addition tables so that the multiplication table was in cyclic order. In cyclic order, the zeroth column (or row) represents zero and the $(i+1)$ -th column (or row) (for $i \neq 0$ ) represents $a^i$ for some generator $a$ . As such, the multiplication table is simply $0$ in the zeroth row and column and $1 + ((i+j) \mod 255)$ for the spot $(i+1,j+1)$ .

Once resorted by the order of the powers of a generator $a$ , the multiplication table look the same regardless of the $a$ . The addition, however, looks different for different generators. Below are the addition tables for two of them: $a = (1+x)$ on the right and $a = x^3 + x^6 + x^7$ on the left. They make as decent a stereo as any of the other pairs so far.

Here’s the code that I used to generate the remapping for a given generator.

(defun make-cyclic-remapping (fn generator &optional (limit 256))
(let ((to (make-array (list limit) :initial-element 0))
(from (make-array (list limit) :initial-element 0))
(used (make-hash-table :test #'equal)))
;; fill up the lookup tables
(setf (gethash 0 used) t)
(nlet fill-tables ((exp 1) (acc 1))
(when (< exp limit)
(setf (aref to exp) acc
(aref from acc) exp
(gethash acc used) t)
(fill-tables (1+ exp) (funcall fn acc generator))))
;; return a closure around the lookup tables
(when (= (hash-table-count used) limit)
(lambda (direction n)
(ecase direction
(:to (aref to n))
(:from (aref from n)))))))

If you’ve read it, you can probably tell by my code that I’m still under the influence of Let Over Lambda. If you haven’t read it, it is quite worth the read.

Then, I used a modified version of the draw-heightmap function which also takes in the remapping function.

(defun draw-mapped-heightmap (op map filename &optional (limit 256))
(let ((png (make-instance 'zpng:pixel-streamed-png
:color-type :grayscale
:width limit
:height limit)))
(with-open-file (stream filename
:direction :output
:if-does-not-exist :create
:element-type '(unsigned-byte 8))
(zpng:start-png png stream)
(dotimes (xx limit)
(dotimes (yy limit)
(let* ((a (funcall map :to xx))
(b (funcall map :to yy))
(c (funcall map :from (funcall op a b))))
(zpng:write-pixel (list c) png))))
(zpng:finish-png png)))
(values))

Visualizing Galois Fields May 17th, 2012
Patrick Stein

Galois fields are used in a number of different ways. For example, the AES encryption standard uses them.

Arithmetic in Galois Fields

The Galois fields of size $2^n$ for various $n$ are convenient in computer applications because of how nicely they fit into bytes or words on the machine. The Galois field $GF(2^n)$ has $2^n$ elements. These elements are represented as polynomials of degree less than $n$ with all coefficients either 0 or 1. So, to encode an element of $GF(2^n)$ as a number, you need an $n$ -bit binary number.

For example, let us consider the Galois field $GF(2^3)$ . It has $2^3 = 8$ elements. They are (as binary integers) $000$ , $001$ , $010$ , $011$ , $100$ , $101$ , $110$ , and $111$ . The element $110$ , for example, stands for the polynomial $x^2 + x$ . The element $011$ stands for the polynomial $x + 1$ .

The coefficients add together just like the integers-modulo-2 add. In group theory terms, the coefficients are from $\mathbb{Z}/2\mathbb{Z}$ . That means that $0 + 0 = 0$ , $0 + 1 = 1$ , $1 + 0 = 1$ , and $1 + 1 = 0$ . In computer terms, $a + b$ is simply $a$ XOR $b$ . That means, to get the integer representation of the sum of two elements, we simply have to do the bitwise-XOR of their integer representations. Every processor that I’ve ever seen has built-in instructions to do this. Most computer languages support bitwise-XOR operations. In Lisp, the bitwise-XOR of $a$ and $b$ is (logxor a b).

Multiplication is somewhat trickier. Here, we have to multiply the polynomials together. For example, $(x + 1) \cdot x = x^2 + x$ . But, if we did $(x^2 + x) \cdot x$ , we’d end up with $x^3 + x^2$ . We wanted everything to be polynomials of degree less than $n$ and our $n$ is 3. So, what do we do?

What we do, is we take some polynomial of degree $n$ that cannot be written as the product of two polynomials of less than degree $n$ . For our example, let’s use $x^3 + x + 1$ (which is $1011$ in our binary scheme). Now, we need to take our products modulo this polynomial.

You may not have divided polynomials by other polynomials before. It’s a perfectly possible thing to do. When we divide a positive integer $a$ by another positive integer $b$ (with $b$ bigger than 1), we get some answer strictly smaller than $a$ with a remainder strictly smaller than $b$ . When we divide a polynomial of degree $m$ by a polynomial of degree $n$ (with $n$ greater than zero), we get an answer that is a polynomial of degree strictly less than $m$ and a remainder that is a polynomial of degree strictly less than $n$ .

Dividing proceeds much as long division of integers does. For example, if we take the polynomial (with integer coefficients) $x^3 + 2x + 5$ and divide it by the polynomial $x + 3$ , we start by writing it as:

$x + 3 \overline{\left) x^3 + 0x^2 + 2x + 5\right.}$ We notice that to get $(x+3) \cdot q(x)$ to start with $x^3$ , we need $q(x)$ to start with $x^2$ . We then proceed to subtract $(x+3) \cdot x^2$ and then figure out that we need a $-3x$ for the next term, and so on. We end up with $x^2 - 3x + 11$ with a remainder of $-28$ (a degree zero polynomial).

$\begin{array}{ccrcrcrcr}& & & & x^2 & - & 3x & + & 11 \\& \multicolumn{8}{l}{\hrulefill} \\(x+3) & ) & x^3 & + & 0x^2 & + & 2x & + & 5 \\& & x^3 & + & 3x^2 & & & & \\& & \multicolumn{3}{l}{\hrulefill} & & & & \\& & & - & 3x^2 & + & 2x & & \\& & & - & 3x^2 & - & 9x & & \\& & & \multicolumn{4}{l}{\hrulefill} & & \\& & & & & & 11x & + & 5 \\& & & & & & 11x & + & 33 \\& & & & & & \multicolumn{3}{l}{\hrulefill} \\& & & & & & & - & 28\end{array}$ For the example we cited earlier we had $x^3 + x^2$ which we needed to take modulo $x^3 + x + 1$ . Well, dividing $x^3 + x^2$ by $x^3 + x + 1$ , we see that it goes in one time with a remainder of $x^2 + x + 1$ . [Note: addition and subtraction are the same in $GF(2^n)$ .]

$\begin{array}{ccrcrcrcr}& & & & & & & & 1 \\& \multicolumn{8}{l}{\hrulefill} \\(x^3+x+1) & ) & x^3 & + & x^2 & + & 0x & + & 0 \\& & x^3 & + & 0x^2 & + & x & + & 1 \\& & \multicolumn{7}{l}{\hrulefill} \\& & & & x^2 & + & x & + & 1\\\end{array}$ For a reasonable way to accomplish this in the special case of our integer representations of polynomials in $GF(2^n)$ , see this article about Finite Field Arithmetic and Reed Solomon Codes. In (tail-call style) Lisp, that algorithm for $GF(2^8)$ might look something like this to multiply $a$ times $b$ modulo $m$ :

(flet ((next-a (a)
(ash a -1))
(next-b (b)
(let ((overflow (plusp (logand b #x80))))
(if overflow
(mod (logxor (ash b 1) m) #x100)
(ash b 1)))))
(labels ((mult (a b r)
(cond
((zerop a) r)
((oddp a) (mult (next-a a) (next-b b) (logxor r b)))
(t (mult (next-a a) (next-b b) r)))))
(mult a b 0)))

How is the Galois field structured?

The additive structure is simple. Using our 8-bit representations of elements of $GF(2^8)$ , we can create an image where the pixel in the $i$ -th row and $j$ -th column is the sum (in the Galois field) of $i$ and $j$ (written as binary numbers). That looks like this:

Just before the above-mentioned article hit reddit, I got to wondering if the structure of the Galois field was affected at all by the choice of polynomial you used as the modulus. So, I put together some code to try out all of the polynomials of order 8.

Remember way back at the beginning of multiplication, I said that the modulus polynomial had to be one which couldn’t be written as the product of two polynomials of smaller degree? If you allow that, then you have two non-zero polynomials that when multiplied together will equal your modulus polynomial. Just like with integers, if you’ve got an exact multiple of the modulus, the remainder is zero. We don’t want to be able to multiply together two non-zero elements to get zero. Such elements would be called zero divisors.

Zero divisors would make these just be Galois rings instead of Galois fields. Another way of saying this is that in a field, the non-zero elements form a group under multiplication. If they don’t, but multiplication is still associative and distributes over addition, we call it a ring instead of a field.

Galois rings might be interesting in their own right, but they’re not good for AES-type encryption. In AES-type encryption, we’re trying to mix around the bits of the message. If our mixing takes us to zero, we can never undo that mixingâthere is nothing we can multiply or divide by to get back what we mixed in.

So, we need a polynomial that cannot be factored into two smaller polynomials. Such a polynomial is said to be irreducible. We can just start building an image for the multiplication for a given modulus and bail out if it has two non-zero elements that multiply together to get zero. So, I did this for all elements which when written in our binary notation form odd numbers between (and including) 100000001 and 111111111 (shown as binary). These are the only numbers which could possibly represent irreducible polynomials of degree 8. The even numbers are easy to throw out because they can be written as $x$ times a degree 7 polynomial.

The ones that worked were: 100011011, 100011101, 100101011, 100101101, 100111001, 100111111, 101001101, 101011111, 101100011, 101100101, 101101001, 101110001, 101110111, 101111011, 110000111, 110001011, 110001101, 110011111, 110100011, 110101001, 110110001, 110111101, 111000011, 111001111, 111010111, 111011101, 111100111, 111110011, 111110101, and 111111001. That first one that worked (100011011) is the one used in AES. Its multiplication table looks like:

Here’s it is again on the left with the multiplication table when 110011111 is the modulus on the right:

So, the addition image provides some insight into how addition works. The multiplication tables, at least for me, provide very little insight into anything. They don’t even make a good stereo pair.

To say two of the multiplication tables have the same structure, means there is some way to map back and forth between them so that the multiplication still works. If we have table $X$ and table $Y$ , then we need an invertible function $f$ such that $f(a \cdot_X b) = f(a) \cdot_Y f(b)$ for all $a$ and $b$ in the table $X$ .

What’s next?

If there is an invertible map between two multiplication tables and there is some element $a$ in the first table, you can take successive powers of it: $a$ , $a^2$ , $a^3$ , $\ldots$ . There are only $2^n$ elements in $GF(2^n)$ no matter which polynomial we pick. So, somewhere in there, you have to start repeating. In fact, you have to get back to $a$ . There is some smallest, positive integer $k$ so that $a^{k+1} \equiv a$ in $GF(2^n)$ . If we pick $a = 0$ , then we simply have that $k = 1$ . For non-zero $a$ , we are even better off because $a^k \equiv 1$ .

So, what if I took the powers of each element of $GF(2^n)$ in turn? For each number, I would get a sequence of its powers. If I throw away the order of that sequence and just consider it a set, then I would end up with a subset of $GF(2^n)$ for each $a$ in $GF(2^n)$ . How many different subsets will I end up with? Will there be a different subset for each $a$ ?

I mentioned earlier that the non-zero elements of $GF(2^n)$ form what’s called a group. The subset created by the powers of any fixed $a$ forms what’s called a subgroup. A subgroup is a subset of a group such that given any two members of that subset, their product (in the whole group) is also a member of the subset. As it turns out, for groups with a finite number of elements, the number of items in a subgroup has to divide evenly into the number of elements in the whole group.

The element zero in $GF(2^n)$ forms the subset containing only zero. The non-zero elements of $GF(2^n)$ form a group of $2^n - 1$ elements. The number $2^n - 1$ is odd (for all $n \ge 1$ ). So, immediately, we know that all of the subsets we generate are going to have an odd number of items in them. For $GF(2^8)$ , there are 255 non-zero elements. The numbers that divide 255 are: 1, 3, 5, 15, 17, 51, 85, and 255.

It turns out that the non-zero elements of $GF(2^n)$ form what’s called a cyclic group. That means that there is at least one element $a$ whose subset is all $2^n - 1$ of the non-zero elements. If take one of those $a$ ‘s in $GF(2^8)$ whose subset is all 255 of the elements, we can quickly see that the powers of $a^3$ form a subset containing 85 elements, the powers of $a^5$ form a subset containing 51 elements, …, the powers of $a^{85}$ form a subset containing 3 elements, and the powers of $a^{255}$ form a subset containing 1 element. Further, if both $a$ and $b$ have all 255 elements in their subset, then $a^k$ and $b^k$ will have the same number of elements in their subsets for all $k$ . We would still have to check to make sure that if $a^i + a^j = a^k$ that $b^i + b^j = b^k$ to verify the whole field structure is the same.

This means there are only 8 different subset of $GF(2^8)$ ‘s non-zero elements which form subgroups. Pictorially, if we placed the powers of $a$ around a circle so that $a^0 = 1$ was at the top and the powers progressed around the circle and then drew a polygon connecting consecutive points, then consecutive points in the $a^3$ sequence and consecutive points in the $a^5$ sequence, etc…. we’d end up with this:

If we don’t reorder them and just leave them in the numerical order of their binary representation, the result isn’t as aesthetically pleasing as the previous picture. Here are the same two we used before 100011011 (on the left) and 110011111 (on the right). They are easier to look at. They do not lend much more insight nor make a better stereo pair.

*shrug* Here’s the source file that I used to generate all of these images with Vecto and ZPNG: group.lisp

nklein software

software development and consulting

Nick Levine’s cl-log rocks… December 10th, 2012
Patrick Stein