Fog of Light – Starting to Add Star-Fields May 28th, 2017
Patrick Stein

I have finally written my first OpenGL code using GLSL. Whew. That took way too long to get all working correctly. I promise, soon, I will upload some sample code so that others may not have to stumble as long as I did.

For the star-field, I generate a few thousand 2-D points. Each point has its own radius, its own opacity, and its own color.

I put these all into an OpenGL array buffer. Then, the vertex shader copies data out of my struct to set the color and the point size. Then, the fragment shader turns the color into two, overlapping radial gradients (one that is half the radius of the other) by modulating the color’s opacity.

screenshot of sample starfield

Next up will be nebulae, then planets/asteroids in the local system.

Fog of Light – Getting Underway May 15th, 2017
Patrick Stein

Dauntless (The Lost Fleet, Book 1) was the first science-fiction book I read that tried to deal with space combat with the real-world constraint that light only travels so fast. It takes light eight minutes to get from the Sun to Earth. It takes light more than a second to get from the Earth to the Moon. Depending on where they are in their orbits, it takes between three minutes and twenty-two minutes to get light from Mars to Earth.

Imagine that you’re a star-ship. You and your companions have just warped into a new star system. You see a flotilla of enemy ships about 45 light-minutes away. That means, you’ve got 45 minutes before that flotilla can possibly even know that you’re in their star system. How much can you get done in that time? Once they can see you, how much can you mislead them on your target if they’re going to be operating on data about where you were heading more than half an hour ago?

For years, I have been batting around this concept, hammering it into a game. I have finally gotten started on it.

Armed with some functions like these, I am constructing values which change at points in space-time and querying the value visible from other points in space-time.

(defgeneric get-nearest-value (space-time-value space-time-point)
  ...
  (:documentation "Find the observable value of a quantity
SPACE-TIME-VALUE when observed from a given location
SPACE-TIME-POINT. This method finds the most-recent
value V0 (at location P0) for this data when viewed from
the given location. This method returns (VALUES V0 P0).
This method makes no effort to interpolate the results."
))

Here are my first, visually-demonstrable results:

Hopefully, there will be plenty more coming in the near future.

The Metaphor Soup Blender Train March 10th, 2017
Patrick Stein

I had two days of SAFe training this week. SAFe stands for Scaled Agile Framework. Its goal is to take Agile software practices and use them in a big program, big product, big system scenario.

The training itself went well. It was 10,000 times better than the training that I had 25 years ago on the Carnegie Mellon Capability Maturity Model. The instructor knew his stuff and wasn’t afraid to weigh in on the good/bad of how SAFe has been rolled out in our program.

But, I gotta say, the Capability Maturity Model took five vague concepts and enshrined them into a mythical religion. SAFe, on the other hand, took a whole slew of disparate imagery and jammed them all together in some of the most crowded, grindingly-bad metaphoric combinations I can imagine.

Starting with Agile

As it starts with Agile, SAFe is already on shaky metaphoric ground. In Agile practice, one breaks up the work into User Stories. If a particular story is super-broad and long, then it is called an Epic. So far, so good, eh? But, stories are fairly static and we’re trying to be Agile. So, we take a group of stories and put them together into the work we’re going to do for the next two weeks and we call it, an Anthology?, an Issue?, a Serial?, no… we call it a Sprint. Because there is nothing like running with a good book to boost productivity.

And, I should mention that some User Stories are more complicated than others. So, each User Story has a number of Points assigned to it, usually through Planning Poker. We use a gambling metaphor to come to a team consensus on the number of Points each Story is worth.

Are you following me? Good.

Okay, so, in Agile, too, we want to break down the barriers between those who design and those who code and those who test. How does our team do this? We Scrum. Scrum is term from the sport of Rugby. A scrum, in Rugby, is a big crowd of players from both teams, fighting over the ball in a mob, trying
to push it down the field to their team’s advantage. In Agile, we are all on the same team, so I’m not sure why we’re fighting over the ball. And, I’m not sure how this is helping us Sprint our User Stories. But, that’s what we do.

Really though, for some reason, we use the word Scrum to mean a quick daily meeting to update everyone on what we’re working on and raise anything that is blocking us. Cuz, that’s nothing like a Rugby scrum. And, we use the term Swarm to refer to those times when our whole team is focused on a singular goal to push it through, or when other teams dive in to help our team with a singular goal.

So, in order to finish our User Stories in a Sprint, we Scrum every day and occasionally Swarm.

And, like every good Sprint, the next Sprint starts immediately after the last one ends.

The last thing you do in a Sprint is demo what you accomplished in the Sprint. You know, like in a foot race, where you finish the race and then show everyone how you just finished the race. And, then, you have a Readout of what you’ve done. And, you start the next sprint with a Readout of what you’re going to do the next Sprint. Because, it wasn’t scientific enough to have just finished your Stories in your Sprint, you need an old piece of galvanized steel equipment to spit out a small paper tape with the total on it.

All Aboard

So, how do you take something organized around Sprints full of Stories by a Scrum team who can Swarm and make it scale to something big?

Well, you start by adding trains. You add multiple, ARTs (Agile Release Trains). These trains each have multiple Scrum teams Scrumming and Swarming on Stories.

But, now, we need some stuff in between Epics and User Stories. Are we going to stick with writing metaphors? Of course not. We’re going to make Epics broken up into multiple Capabilities and Capabilities broken up into multiple Features and those Features get broken up into User Stories.

Oh, we need to know where the Agile Release Trains are heading, right? So, we will have a Roadmap and a Vision which will be the Guardrails for our train. And, the train, itself, will be following a Value Stream with Milestones. So, your train doesn’t run on train tracks. It runs on a stream. It does this so it can Pivot if needed. But, it is okay, because there are guardrails and a roadmap.

The train provides Transparency. Your train full of scrums on a stream with a roadmap is transparent.

There is, of course, also a Product Manager or Product Owner who steers the train. We also have Release Train Engineers. They don’t get to steer though. They only get to drive the train straight along the stream.

Are We Flying Yet?

So, in Software Development, there are tons of small decisions that we make each day that, together, form the Architecture of our application. Most big projects have dedicated Software Architects who are responsible for keeping the Architecture coherent.

You may rightly be thinking that this whole bevy of Scrum Teams piled on our Agile Release Trains with their heads focused on the User Stories in their own Sprints might not be the best thing for the overall Architecture. But, that’s okay, because running along the Value Stream there is also an Architectural Runway. I’m not 100% sure if this is so that our product can fly (or land?) or so that our product can sell its new line of shoes in Milan. But, we’ve got a Runway.

The Crooked House

At some point, someone decided that SAFe needed Lean. So, they added it in, too. In fact, they have what is called the SAFe House of Lean. Now, I am quite glad that my house house does not lean.

The foundation of the SAFe House of Lean is Leadership. The roof is Value. The pillars are: Respect for people and culture, Flow, Innovation, and Relentless Improvement. We’ve now built a house out of abstract concepts. I’m not sure where it sits compared to our Stream or our Runway or our Train or our Sprint, but it makes for a hell of a User Story.

The Less You Know (An Unspoken Tenet of Clean Architecture) February 7th, 2016
Patrick Stein

In response to my previous article, Holger Schauer asked why I implemented the use-cases with classes rather than having the book repository passed into the browse-books method or putting it into the abstract base-class browse-books-use-case rather than in the implementation.

(defstruct browse-books-request
  book-repository)

(defgeneric browse-books (request response))

;; -- or --

(defgeneric browse-books (book-repository request response)

;; -- or --

(defclass browse-books-use-case ()
  ((book-repository :initarg :book-repository :reader book-repository)))

I started to reply, but I found that I was having trouble keeping my response to a reasonable length. So, here, you are subjected to my unreasonably long response.

For reference, here is the repository from the previous post.

Java’s Influence

I chose to model the use-cases as classes to be faithful to all of the implementations that I’ve seen of Clean Architecture. Uncle Bob, in particular, does most of his videos and talks using Java or Ruby. When he uses Ruby, he still tends to use the same OO organization that he would use if he were writing Java.

One aspect of Java is that everything has to be a method on some class. If you want to declare a method somewhere but have its implementation somewhere else, you can do this with an abstract base class or an interface. You cannot, for example, declare an abstract static method as you can in CLOS (or even C).

Java’s single-inheritance makes it very awkward in general to use abstract base classes. If the book-repository slot were on the browse-books-use-case rather than on the implementation, this would have to be an abstract base-class and direct inheritance if it were in Java. If the book-repository slot is on the implementation rather than on the browse-books-use-case, then someone writing this in Java could use an interface and an implementation of that interface.

I wrote this blog post using Lisp/CLOS. However, for the actual project that I am working on, I have opted to use PHP. PHP shares Java’s inheritance restrictions. A class can only have one superclass but it can implement any number of interfaces.

The Stated Advantages of Clean Architecture

When Uncle Bob talks about why to use Clean Architecture, he tends to focus on these things:

  • Ability to defer decisions about the frontend and backend
  • Ability to test the core logic of the application easily
  • Ability to deploy your application in pieces

The adapters depend on interfaces defined in the application. The application does not depend on anything defined by the frontend or the backend. As such, deciding which frameworks to use on the frontend or which database to use on the backend can be deferred for a long time during the development process. Uncle Bob often describes how on FitNesse they developed using the filesystem as the database backend fully intending to plug in a relational database later in the project. In the end, they decided that the filesystem backend they had created served their purposes just fine. When they came upon a client who had requirements that all of the dynamic content be stored in their corporate database, they rewrote an adapter for that customer to use the database rather than the filesystem.

Clean Architecture applications are easy to test because the application code isn’t interwoven with the presentation code or the database code. One can create mocks (even test-specific mocks) for the repositories and front-ends that do whatever the test needs to have done. They can do it quickly, without temporary databases or network overhead.

In a Clean Architecture deployment of my application in Java, I might have one JAR file for the book repository implementation, one JAR file for the Console frontend, one JAR file for the application core logic, and one JAR file that wires all of those other pieces together on startup. If I need to incorporate a different database, then I only need to update the book repository implementation JAR and (maybe) the JAR file that wires the pieces together. I don’t have to redeploy my entire application every time the database changes. I don’t have to retest every bit of application logic. Everything is nicely contained.

An Unspoken Benefit of Clean Architecture

One of the ways that Clean Architecture keeps its dependencies in check is by making all of the interfaces going into or out of your application use primitive types or dumb structures for parameter passing. The unspoken benefit of this is that it encourages you to keep the interfaces as spartan as possible.

If the controller (the console in my sample application) that is invoking the use case doesn’t absolutely need to know there is a book repository involved, then it shouldn’t know about it. The book repository cannot be passed from the controller to the use case using primitive types and dumb structures.

The controller’s job is to take input from the user, turn it into a request, invoke the use case with that request, and present the response back to the user. There is no input that the controller can get from the user which will allow it to instantiate a book repository. It could, of course, pull it from global state as it pulled the use case. However, the less it needs to know, the better.

This separation has other benefits, as well. In Lisp, especially, one could imagine interacting with the application through the REPL. So, rather than implementing the simple REPL that I did with my console, I could have made a browse function that invokes the use-case and returns the results.

(defun browse ()
  (let ((request (make-browse-books-request))
        (response (make-browse-books-response)))
    (browse-books *browse-books-use-case* request response)
    (mapcar #'book-summary-to-plist
            (browse-books-response-list-of-book-summaries response))))

If everything is going to happen within my one Lisp image, then it would be fine to keep a *book-repository* global to pass into the use case. However, if I want to share the book repository between multiple users, each at their own REPL, then it no longer makes sense that each REPL would need a handle to the book repository.

Remote Proxy class-diagram

My browse function doesn’t need to know whether the use case it is interacting with is the actual implementation or just a proxy for the implementation. In the client-server case, it makes little sense for the client to have any idea that there is a book repository instance.

I suppose there are some classes of application where one might want to work on a remote repository but keep the application logic local. If that were the case, then one would want the book repository to be the interface which has the proxy and server. However, nothing in the architecture mentioned in the previous post precludes that use. If we had put the book repository into the interface for the use case rather than in its implementation, then we need to at least pretend there is a book repository on the local client even when everything is going to happen on the server.

Conclusion

An accidental benefit of Clean Architecture’s insistence on primitive or dumb-struct parameters to use-cases is that the parameters end up only reflecting the minimal amount of information that is specific to the current request. Any state that is common across requests is a part of the internal state of the use-case.

Because of this, code that is interacting with the use case only to know the little bit that makes them different from other people using the use case. This results in very simple interfaces to the use case and the a great deal of flexibility in implementation.

Exploring Clean Architecture February 4th, 2016
Patrick Stein

This is the first in what will likely be a series of blog posts about Clean Architecture. Uncle Bob Martin has written numerous blog posts and given lots of talks about it.

The goal of Clean Architecture is to have the directory structure of your application shout out what your application does rather than what framework was used to present your application or what database is nestled in the depths of your application. Your program is divided into Entities, Use Cases, and Interface Adapters.

Entities encapsulate “Enterprise-wide business rules.” Use Cases encapsulate “Application-specific business rules.” Interfaces and Adapters represent how your Use Cases want to interact with the outside world (e.g. databases, users, printers, etc.).

In Clean Architecture, the Entities cannot know that the Use Cases exist and the Use Cases cannot know anything about the Adapters except for the Interface to them which is defined by the Use Case rather than by the Adapter. The Use Case does not know whether the application is being used from the command-line or from the web or from a remote service calling into it. The Use Case does not know whether the data is being stored in the file system or in a relational database (or conjured from the ether as needed). Nothing in the Adapters can know anything about the Entities.

Simple Example

I have a project that I am just starting. I thought I would use this new project to see how Clean Architecture works for me.

There are large number of talks and videos about Clean Architecture. However, there are not many examples of it despite several years of Stack Overflow questions and blog posts asking for examples.

There are a few simple examples around the web. The most notable is Mark Paluch’s Clean Architecture Example. It is just big enough to get a sense of how things hang together. If you’re willing to put up with Java’s insane directory hierarchies, you can get a pretty good idea of what the application does just by poking around the Use Cases directory.

My First Use Case

My first Use Case is to let the User browse a list of Book Summaries. The User should be able to sort by Title, Author, Publication Date, or Date the Book was acquired. The User should be able to filter the list based upon Genre or Keyword. The Use Case should allow the caller to implement pagination, so the Use Case needs to support returning up to a given number of Book Summaries starting with a specific number.

Some might argue that that is multiple Use Cases glommed together. If that were the case, then I would need some way to pipeline together Use Cases if I’m going to make any sort of reasonably navigable app atop my Use Cases.

But, let’s start with baby steps.

The Simplified Version of my First Use Case

Let’s just say the User wishes to see a list of all of the Book Summaries. The User is fine with seeing all of them at once in whatever order my app sees fit.

This simple version of the Use Case is implemented in an accompanying repository under the tag the-dream.

Class Diagram (explained below)

The architecture consists of some simple structures with no “business logic” in them at all: book-summary and book.

(defstruct book-summary
  isbn
  title
  author
  cover-page-thumbnail)

(defstruct book
  isbn
  title
  author
  cover-page-thumbnail
  cover-page-fullsize
  list-of-thumbnails
  table-of-contents
  synopsis
  publication-date
  list-of-genres
  list-of-keywords)

There is one use case browse-books which defines the use-case interface browse-books-use-case along with its input structure browse-books-request and its output structure browse-books-response. The use case defines the method browse-books which must be called with a browse-books-use-case instance, a browse-books-request instance, and browse-books-response instance.

(defstruct browse-books-request)

(defstruct browse-books-response
  list-of-book-summaries)

(defclass browse-books-use-case ()
  ())

(defgeneric browse-books (use-case request response))

In my implementation, the browse-books-response is a simple data structure. One could easily imagine that the browse-books method would return one rather than filling one in that was passed to it. In some variants of Clean Architecture (like the Paluch example cited above), the response model is a class instance upon which a method is called to complete the interaction. But, it would have to be clear from the outset that anyone using this Use Case cannot depend on it being synchronous or asynchronous.

The use case also defines the book-repository interface that it needs.

(defclass book-repository ()
  ())

(defgeneric find-book-by-isbn (book-repository isbn))
(defgeneric all-books (book-repository))

In the Paluch example, all of the use cases share the same repository interfaces (though Paluch and others have separate repository interfaces for Users and Invoices and Items). In several of Uncle Bob’s videos, he makes the claim (or claims equivalent to the claim) that each use case should define an interface for just the methods it needs to use on a Book repository. In this use case, it would need only the ability to retrieve the list of Books and so I should not have defined find-book-by-isbn here at all, and I should have named this interface browse-books-book-repository.

I wrote browse-books-impl class which extends the browse-book-use-case. It takes an instance of book-repository on construction.

(defclass browse-books-impl (browse-books-use-case)
  ((book-repository :initarg :book-repository :reader book-repository)))

(defun make-browse-books-use-case (book-repository)
  (check-type book-repository book-repository)
  (make-instance 'browse-books-impl :book-repository book-repository))

It uses that to retrieve the list of Books. Then, it creates a book-summary from each book instance retrieved from the book-repository.

(defun summarize-book (book)
  (check-type book book)
  (make-book-summary :isbn (book-isbn book)
                     :title (book-title book)
                     :author (book-author book)
                     :cover-page-thumbnail (book-cover-page-thumbnail book)))

(defmethod browse-books ((use-case browse-books-impl)
                         (request browse-books-request)
                         (response browse-books-response))
  (let* ((books (all-books (book-repository use-case)))
         (summaries (mapcar #'summarize-book books)))
    (setf (browse-books-response-list-of-book-summaries response) summaries))
  response)

To test the design so far, I wrote in-memory-book-repository backend which implements the book-repository interface that was defined in the Use Case.

(defclass in-memory-book-repository (book-repository)
  ((books :initarg :books :reader books)))

(defun make-in-memory-book-repository (books)
  (check-type books list)
  (assert (every #'book-p books))
  (make-instance 'in-memory-book-repository :books books))

(defmethod all-books ((book-repository in-memory-book-repository))
  (mapcar #'copy-book (books book-repository)))

I also wrote a console frontend which invokes the browse-books-use-case.

(defun console-browse-books ()
  (let ((request (make-browse-books-request))
        (response (make-browse-books-response)))
    (browse-books *browse-books-use-case* request response)
    (mapcar #'console-print-summary
            (browse-books-response-list-of-book-summaries response))
    (values)))

...

(defun console-main-loop ()
  (catch 'console-quit
    (with-standard-io-syntax
      (loop
         :do (mapc #'console-print
                   (multiple-value-list
                      (console-eval (console-read))))))))

To tie it all together, I wrote app-context which holds the current browse-books instance.

(defvar *browse-books-use-case*)

And, I wrote the app which creates an instance of the in-memory-book-repository and creates the browse-books-impl for the app-context. Then, it runs the main loop of the console frontend.

(defun run-console-app-with-memory-db (&optional (books *book-list*))
  (let* ((book-repo (make-in-memory-book-repository books))
         (*browse-books-use-case* (make-browse-books-use-case book-repo)))
    (console-main-loop)))

Trouble In Paradise

Already, in this simple interface, I am torn. For this Use Case, I do not need the repository to return me the list of Books. I could, instead, ask the repository to return me the list of Book Summaries. If I do that, my application is just a fig-leaf over the repository.

(defgeneric all-book-summaries (book-repository))

Well, the argument against asking the repository for Book Summaries is that it should not be up to the database to decide how I would like to have my Books summarized. That certainly seems like it should be “business logic” and probably “application specific” business logic at that.

So, fine. I will have the repository return Books and the Use Case will summarize them.

Now, let me extend the Use Case the next little bit forward. What if I want to support pagination? My choices are to push the pagination down to the repository so that I can ask it to give me up to 20 Books starting with the 40th Book. Or, I can let the repository give me all of the books and do the pagination in the Use Case.

(defstruct browse-books-request
   start max-results)

Here, I can find no guidance in any of the Clean Architecture videos that I have watched nor in the examples that I have found online. Everyone seems happy with the repositories being able to return one item given that item’s unique identifier or return all of the items.

If the repository is going to return all of the Books, then why wouldn’t my Use Case just return them all and leave the caller to do any pagination that is needed?

This works fine when there are a few dozen books and they are small. It does not scale, and I don’t know how it is supposed to scale without pushing most of the responsibility onto the repository.

(defgeneric all-books-in-range (book-repository start max-results))

Sure, I can push the responsibility onto the repository. But, one of the reasons that Clean Architecture is so structured is to allow easy testing of all of the application logic. The more that I push into the repository, the less that I actually exercise when I run my unit tests with my mock repository (and the more complex my mock repository should probably be).

One possible approach would be to have the all-books method instead be all-isbns. Then, I can retrieve all of the ISBNs and use find-book-by-isbn to get all of the books.

Now, if I want to sort by Author then by Title, I need to:

  • fetch all of the ISBNs all-isbns,
  • fetch each ISBN’s title,
  • sort my ISBNs by title,
  • fetch each ISBN’s author,
  • stable-sort my ISBNs by author,
  • clip to my range,
  • fetch each book in my range,
  • summarize each fetched book

Or, I have to write an SQL query, that can do all of that in one database call instead of 2N + R + 1 calls (where N is the number of books in the database and R is the number of books in my range), making my Use Case a fig-leaf again.

Updates In Email

Email:

l