Building Supplies January 8th, 2016
Patrick Stein

I have a couple of nascent projects on my plate which require custom, server-side software. I’ve been trying to use these projects to explore the Clean Architecture concepts using Test-Driven Development (TDD).

I can’t even begin.

According to Clean Architecture, I should start with my application logic independent of whether it will be a command-line tool, a series of tools, a web-application, or what-have-you. So, let me start on the application code.

This is TDD though. So, I need to start with a test. How do I do that?

(ql:quickload :nst)

(nst:def-test-group querying-data () ...)

Wait a minute? I just committed myself to Lisp. I just made a huge business decision, a huge implementation decision, a decision that will shape my life for the next n years, and neither TDD or Clean Architecture had anything to say.

I need building supplies. I can’t go anywhere with my architecture or my development without a programming language.

I want to write my apps in Lisp. One of them, I already have mostly done in Lisp. I am still trepidatious about deploying Lisp.

Why am I trepidatious about deploying Lisp? Is it because there is inadequate support for web programming in Lisp? Absolutely not. Is it because I have some doubt in Lisp’s staying power? Absolutely not.

It’s email. Email is holding me back.

I use my current web hosting provider to give me a LAMP stack upon which I run WordPress for this blog and some git repositories and a bug-tracking database (that I can’t remember how to log into). All of that, I could move with confidence in a few hours.

What I dread is having to collect the several hundred mail-forwards that I currently have along with the half-dozen IMAP accounts and move them anywhere, let alone to somewhere that I have to manage them myself and deal with SPAM and mail queues and crap.

It seems that for only a few bucks more per month than I’m paying now, I can get Plesk on a VPS that should be big enough for my purposes. Does anyone have any experience with Plesk? Is it going to make email painless for me on Ubuntu? Or, am I going to hate my life? Does someone have a VPS provider they strongly recommend?

Do I really have to write my application in PHP on the chance that I’ll want to deploy it on the web just because email is scary?

Population Density in Politics January 3rd, 2016
Patrick Stein

I am taking a Coursera course by Wesleyan University titled Data Management and Visualization.

During Bush v. Gore, there was a ton of freely available data about that election. I did a quick-and-dirty graph showing that Gore won the most populous counties by wide margins and Bush won most of the rest of the counties by wide margins.

For this course, I am going to revisit that analysis with the 2012 election data and maybe the 2008 election data.

The Hypothesis

In the United States, there is a strong correlation between population density and voting for the Democratic candidate for president.

The Data

The 2012, by-county results are available through The Guardian newspaper website. The 2008 election data is available for purchase through Dave Leip’s Election Data Store. The US Census Data website has information available about the population and land area of each U.S. county.

Related Work

My first web search for related data was: correlation population density political party.

This search turns up several articles about a scatterplot by Conor Sen relating the Cook Partisan Voting Index (PVI) plotted against population density based on 2012 data. There are related heat-maps by others from the same data.

That search also turns up a paper by Jowei Chen of the University of Michigan and Jonathan Rodden of Stanford University about why compact voting districts are bad for Democrats. That paper focuses mostly on the shapes of voting districts in Florida and how they have be gerrymandered to make those in population-dense areas very compact while those in less populated areas are tentacled and sprawling and how this results in a higher number of Republican representatives than is warranted by overall population numbers.

A related aspect that shows up in this search is that on specific issues, like transit infrastructure, the congressional voting record is strongly correlated with the population density of the congressperson’s district. This effect is a second-order effect, however. The vote of a congressperson will likely be entwined with what the party as a whole wants as much as (or even more than) their constituents want.

ArcGIS contains a map correlating political affiliation of congresspersons with the population density of their districts.

The Tools

I will, of course, being me, use Common Lisp for all of this. I suspect that I will use Fare-CSV for ingesting CSV data. If I have to parse TIGRE data, I will likely rely on some blend of esrap and CL-EWKB or custom geometry code. For plotting, I will likely rely on Vecto but may also try out some of the other libraries like adw-charting or finally get around to making my own multi-backend charting library.

Grid-Generators v0.4.20151016 (and List-Types v0.4.20151029) Released October 29th, 2015
Patrick Stein

Two new Common Lisp libraries released: GRID-GENERATORS and LIST-TYPES.

GRID-GENERATORS and GRID-ITERATE packages

I often find myself wanting to iterate over the points in a rectangular region of a grid in an arbitrary number of dimensions. If my grid is only one-dimensional or two-dimensional then I often just write nested loops to traverse the points of interest.

(loop :for y :from 0 :to 10 :by 1/2
    :do (loop :for x :from 0 :to 10 :by 1/2
             :do (something x y)))

However, when I want the code to be flexible in the number of dimensions, I always end up writing application-specific code to increment arbitrarily long lists of integers with given bounds. I finally got sick of repeating this code every time I needed it and created a library.

For the particular application that I have in mind though, I wanted more than just walking the points inside some rectangular hyper-prism. I wanted to traverse all of the points at a give taxicab distance from a starting point.

The GRID-GENERATORS package facilitates generating points in a rectangular hypergrid, generating points based on taxicab distance, and generating points based upon the number of steps in an arbitrary lattice (given the generators of the fundamental parallelpiped of the lattice). The GRID-ITERATE package provides ITERATE clauses for those generators.

For example, one could iterate over the same points in my nested example above like this:

(loop :with generator := (make-grid-generator '(10 10) :by '(1/2 1/2))
    :for (x y) := (funcall generator)
    :while x
    :do (something x y))

Or, with iterate:

(iterate:iterate
  (iterate:for point on-grid-to '(10 10) by '(1/2 1/2))
  (destructuring-bind (x y) point
     (something x y)))

LIST-TYPES package

The LIST-TYPES package provides a way to generate `SATISFIES` type clauses that ensure that a list contains elements all of the given type. For example, if I wanted to ensure that my list was entirely rational numbers, I could use the declaration:

(check-type my-list (list-types:list-of rational))

Say What You Mean September 19th, 2015
Patrick Stein

There is a scene in the movie The Birdcage where the son tells his father that he (the son) has met a girl and is going to get married. The father begins gulping down the glass of wine that he has in hand. The son asks, Are you upset? The father finishes the glass of wine and says, Let me tell you why.

Here is a function that I wrote several years ago.

(sheeple:defreply mouse-move ((item =draggable=) xx yy)
  (let ((dragging (dragging item)))
    (when dragging
      (let ((dx (- xx (car dragging)))
            (dy (- yy (cdr dragging))))
        (incf (offset-x item) dx)
        (incf (offset-y item) dy))
      (let ((pp (parent item)))
        (when pp
          (when (< (width pp) (+ (offset-x item) (width item)))
            (setf (offset-x item) (- (width pp) (width item))))
          (when (< (height pp) (+ (offset-y item) (height item)))
            (setf (offset-y item) (- (height pp) (height item))))))
      (when (< (offset-x item) 0) (setf (offset-x item) 0))
      (when (< (offset-y item) 0) (setf (offset-y item) 0))
      t)))

This is awful! Am I upset? Let me tell you why.

Is it the Single Responsibility Principle (SRP)? No.

Is it Don’t Repeat Yourself (DRY)? No.

Is it Mixing Levels of Abstraction? Closer, but not quite.

Those are all clearly violated by this code. But, that’s not really the problem. The problem is Why. Nothing about this code tells you why it is here or what is doing.

There is no way to glance at that function and have any idea what’s going on. You have to read it carefully. You have to understand things that aren’t even in this source file to make head nor tail of it. Once you understand the second LET block, you will have nine more lines of code without the least inkling of why there should be nine more lines of code. Anyone care to hazard a guess as to why this function returns T (only) when we’re dragging?

Encapsulation

Two years ago, a colleague and I were tasked with providing docstrings for every function in all of the code we’d written in the last year. We’d done well on providing docstrings to the outward-facing functions, but now we had to do the rest. He started at one end of the directory (in alphabetical order), and I started at the other end. This gave me a good opportunity to look closely at a boat-load of code he’d written that I’d never really delved into before.

He was absolutely religious about encapsulating containers. If he had a hash-table or a p-list or a flat list in a DEFVAR, there was one and only one function that retrieved items from it and at most one function that added items to it. Those functions were one or two lines each (two if they needed a mutex). Those functions were named after what the collection was storing not what mechanism was used to store them.

A lot of times when people talk about the value of encapsulating, they talk about shielding the rest of the code from the implementation details so that if you need to replace how it’s actually implemented on the back end you can do it without breaking any existing code. You are protecting your precious implementation from how people will use it so that you can someday replace the implementation with an even more precious implementation next year (when your language finally gets first-class functions).

I’ve been coding for a good long time now. I’m going to let you in on a little secret. Code almost never gets replaced. When code does get replaced, it almost never continues to adhere to the old API (there was always a semantic leak). If there is a business justification strong enough to let you replace the code, it’s because the old code has become an unmaintainable mess of people subverting the interface or the code as it is didn’t scale and now synchronous things need to happen asynchronously or local things have to happen remotely and hiding that under your old API isn’t going to relieve the bottlenecks.

Trying to insulate your code so that it’s easy to replace is looking down the wrong end of the telescope. The real benefit of encapsulation is that the people who read your code later can be half-asleep and still get everything—your code will scream its meaning. The real benefit of encapsulation is that the person debugging your code can set a break-point in a place that means something—not in the seventeen places the state might have changed but in the only place it could change.

Making It Better

Any ideas what the body in this function does?

(sheeple:defreply mouse-move ((item =draggable=) xx yy)
  (if (being-dragged-p item)
      (handle-event ()
         (let ((dx (- xx (drag-starting-x item)))
               (dy (- yy (drag-starting-y item))))
           (translate-widget item dx dy)
           (keep-widget-inside-parent item)))
      (ignore-event ())))

The new functions BEING-DRAGGED-P, DRAG-STARTING-X, and DRAG-STARTING-Y are just wrappers around what had been explicitly treated as an (OR NULL (CONS INTEGER INTEGER)).

(defun being-dragged-p (item)
  (dragging item))

(defun drag-starting-x (item)
  (car (dragging item)))
(defun drag-starting-y (item)
  (cdr (dragging item)))

It is still an (OR NULL (CONS INTEGER INTEGER)) but nobody ever has to care. Nobody ever has to try to remember what the integers mean. Sure, you could replace it with a structure or a complex number, but why would you ever bother? Why would you ever look at it again?

The new macros HANDLE-EVENT and IGNORE-EVENT encapsulate the return value of this function into something with meaning.

(defmacro handle-event (() &body body)
  `(prog1
       (values t)
     ,@body))

(defmacro ignore-event (() &body body)
  `(prog1
       (values nil)
     ,@body))

It might still be too easy to write an event-handler with a path which doesn’t end in one of these two macros, but it is way better than that dangling T was. It looks like it’s really supposed to be there, and it looks like what it means rather than what it is.

The TRANSLATE-WIDGET and KEEP-WIDGET-INSIDE-PARENT functions can benefit greatly with some further helper functions (and analogous functions for top and bottom):

(defun left (item)
  (offset-x item))
(defun (setf left) (x item)
  (setf (offset-x item) x))
(defun right (item)
  (+ (left item) (width item)))
(defun (setf right) (x item)
  (setf (offset-x item) (- x (width item))))

Some Rules of Thumb

If you find that when you want to check (PRED1 ...) you instead have to check:

(and (PRED0 ...)
     (PRED1 ...))

Then you should consider making a function that does them both. Consider the difference between these two blocks of code:

(when (and (connectedp (player1 g))
           (connectedp (player2 g))
           (not (pausedp g)))
  ...)

(when (game-active-p g)
  ...)

If you find that you are depending on the NULL-ness or positiveness or some other property of some number of state variables to decide which course of action to take, then you should consider making predicates named after your state. In many OO scenarios, you may even want to explicitly track (or calculate) which state you are in at all times.

(defmacro state-case (g &body clauses)
  `(ecase (calculate-or-fetch-state-of g)
     ,@clauses))

(state-case g
 (:pause-screen-showing
  ...)
 (:settings-menu-showing
  ...))

In more imperative languages, it may even be beneficial to keep a STATE member variable in your class. When doing that, make sure that there is one and only one function which actually mutates the value of that STATE member. This will let you:

  1. Log all state transitions without having to hunt for all of them.
  2. Quickly hunt for all of them if you want to do that
  3. Set a break point on all state changes.
  4. Enforce the validity of transitions (or at least scream loudly when something transitions from STOPPED to PAUSED without having passed through PLAYING first).

If you have to check whether some resource is being used by some instance, don’t ask it which resource it is using, ask it whether it is using the one you want.

;;; Common: Reader is forced to know each player has one socket and
;;;    that sockets are comparable with #'=
(loop :for player :in all-networked-players
      :until (= socket-with-something-happening
                (player-socket player))
      :finally (return player))

;;; Better: All I wanted to know is, "Is this yours?"
(loop :for player :in all-networked-players
      :until (player-using-socket-p player
                                    socket-with-something-happening)
      :finally (return player))

Encapsulation is about protecting the person who has to read your code. It’s not about protecting your code.

Syntactic Corn Syrup June 16th, 2015
Patrick Stein

I’ve been bouncing around between Java and C++ and C and loads of JNI cruft in between. At some point today, I accidentally used a semicolon to separate parameters in my C function declaration:

void JNI_myJNIMethod( int paramA; int paramB; int paramC )
{
  ...
}

It looked wrong to me. But, I had one of those brain-lock moments where I couldn’t tell if it was wrong. I was pretty sure that it was wrong by the time my brain locked on pre-ANSI K&R:

void
JNI_myJNIMethod(paramA, paramB, paramC)
  int paramA;
  int paramB;
  int paramC;
{
  ...
}

Regardless, it got me thinking about the programming maxims: Deleted code has no bugs and Deleted code is debugged code.

I never have this kind of brain-lock in Lisp. Some of that is because my Emacs configuration has been molded to my Lisp habits better than to my C/C++/Java habits. Most of it, though, is that Lisp understands the difference between syntactic sugar and syntactic cruft.

Lisp decided long ago that writing code should be easy even if it makes writing the compiler tougher. C and C++ and Java all decided that LALR(1) was more important than me. As if that weren’t bad enough, C++ and Java have thrown the lexers and parsers under the bus now, too. No one gets a free ride.

Updates In Email

Email:

l