The quest for modularity

I was reading some of Avery Pennarun’s blog posts on system architecture and design. There was a set of bullet points that stood out to me that I will copy below:

The top-line goals of module systems are always the same:

1. Isolate each bit of code from the other bits.
2. Re-connect those bits only where explicitly intended (through a well-defined interface).
3. Guarantee that bits you change will still be compatible with the right other bits.
4. Upgrade, downgrade, and scale some bits without having to upgrade all the other bits simultaneously.

This article has some interesting heuristics as well:

State – If you don’t have state, just pure logic, you probably have a library, not a service. If you have state, but it doesn’t change with user requests what you have is configuration, either global or environment-specific. If changes to configuration are not time-sensitive, it can be a parameter to the deployment, which gets triggered when it changes. If they are time-sensitive you might need a configuration service that serves them continuously, rarely a bespoke one. If you do have global state that can change with, and across requests, you may have a service.

I have thought about this quite a bit over the years. My own heuristic revolves around units of deployment vs units of reuse. It’s kind of the same logic around static vs dynamic dependencies in C++. We make tradeoffs that might make sense initially (size vs complexity) but change as constraints change (more RAM/Disk, lower cost).

In the past I have favored library-driven development. I’ve come to learn that many times people are creating monolithic services (even if they are “micro”) that are really business logic libraries. When I architected the UberNote service tier, I started with a monolithic service, but the logic was all in a .NET assembly that was nicely organized into namespaces that would loosely be called “domain driven” today I think. If we needed to split things into multiple services, we could just use that library anywhere we needed to and the service would only call what it needed, or we could have split it into more than one dll to decouple the “deployment”.

I think that the most important part of a system design is the logical decomposition initially, and secondarily the topology (servers and pods, networks, etc). If the aspects of the logical design (domain?) are correct we should be able to arrange things in different topologies as we grow.

Some of the other things that commonly become issues are configuration, provisioning/defining environments, tenancy and maybe some others. Some aspects of the “12 factor app” are orthogonal here. I’m not talking about logging, etc.

Middleware everywhere

I’m a huge fan of composition via middleware. Some people call them plugins, as we did back when I was working on Ubernote. We had plugins for all sorts of things. The only thing you need is a stable API and a default way of invoking things.

This turns out to be important later when considering how to do things in the Node/Express ecosystem. Express doubled down on plugins and I think it worked out pretty well. React has something now called Hooks which I think of in a similar way. It’s a way of plugging behavior into the page render flow without relying on fixed page lifecycle methods. Page lifecycles are a pain. It’s one of my worst memories of ASP.NET (viewstate, etc) and initially React had a similar pain with things like componentWillRender.

My current role involves a large Ruby codebase. I’m not as familiar with that ecosystem yet and I was reading some code that used an HTTP library called Faraday. Recently a co-worker was walking me through some code that used this library and I commented on having seen it and noted that it must be like requests or similar.

Well, it is, but it has some pretty neat features like… plugins! So we can do things like OAuth token refreshes in the client middleware. Pretty cool. Middleware everywhere.

Modern C#

Those of you that have followed me for a while know that my first significant industry experience was with .NET. Of course my first programming experiences were with C64 BASIC and Pascal on the Mac along with college courses in C++ but it’s always those first real codebases where one cuts their teeth.

Anyway, it was very modern at the time with a real type system and ahead-of-time compilation. A competitor to Java. I like to think that C# managed to learn from Java’s mistakes and made it a better language. That and Anders Hejlsberg is a genius.

I wrote a lot of great software in C# including my own note-taking startup UberNote. Once I moved on in the startup world, Microsoft was a total non-starter. No one was using SQL server or .NET or followed any of that ecosystem. From where I was sitting, the real issue with .NET was the community’s tendency to just clone the projects that Java did in the same way (log4NET, etc) and the Web frameworks were lagging behind significantly. MS clung to ASP for way too long.

Fast forward to some recent conversations. I joked sometimes that I would bring Clojure in as a language, which is based on the JVM. Later on some folks turned that joke around on me with C#, and I asked “why not”? Turns out that a lot of other devs around me have significant experience with C#. C# influenced the design of the now-popular TypeScript language. MS open-sourced a bunch of the .NET Core runtime. I wonder if it’s becoming time for C# to be really unchained from MS and come into its own?

Looking at the modern language landscape, the most interesting things I see now are Rust and Go. Maybe some things like Scala. I really don’t know what can dethrone Python right now. I was on the dynamic language bandwagon for a long time and was a champion of Javascript the whole way through to today where it’s one of the most modern language ecosystems (yes there are warts but the rapid pace of development of the language is IMO unmatched – it reached true internet-scale development). TypeScript is in a place where it could take over as a systems language, but I fear that its roots in Javascript will forever hamstring its chances. Maybe with Deno this will change?

What is best systems language to pick right now for back-end distributed systems? It’s kind of crazy to me that Python was the winning horse for so long. C++ was dethroned long ago due to sheer complexity. I think I need to learn Rust and do some tests with .NET on Kubernetes before I make any judgements.

Process stability and Software Engineering

I’m reading some of Edward Deming’s writing and I can’t help but try to draw parallels between manufacturing fundamentals and statistical process control and the software development lifecycle.

Deming was able to show scientific ways of managing hidden variables without trying to make wild guesses about systems up-front. I have seen management systems come and go in Software Engineering and seen backlash against things like the Capability Maturity Model and various ISO certification systems. We ended up coming back to simple things like Agile. Why?

I think agile has become a sort of cargo cult, but some parts of it fundamentally address the right questions about process viewed through the lens of statistical process control. The concept of measuring velocity is a key part, as is tracking estimation errors. I think where the cargo cult comes in is the assumption that the goal is to have the maximum velocity for a given sprint. I’d argue that the more valuable goal is to achieve a stable velocity over time, even if it seems slow.

Complicating matters a bit is that software development is a double-production process. Artifacts are produced which in turn produce other behaviors and further artifacts. Now we are talking about another concept that is reliability. How does reliability relate to quality and to process stability? I think that there are statistical answers to this question.

I was listening to the CyberWire Daily podcast where one of the hosts was discussing cybersecurity insurance. The thesis is that threat models are so complex now that they can’t be calculated classically/deterministically. However, statistical models are very powerful and are able to find very good solutions to otherwise intractable problems.

This is going to be a short post, but hopefully it sows some seeds for me to return to in depth in a later series of posts.

All code is legacy code

I was doing some one-on-one meetings with some folks on my dev team and my colleague showed me an informal document that he was keeping. Sprinkled throughout were references to legacy code and proposals to move to different technologies.

I wanted to dig into some specifics but I casually mentioned that all of our code is legacy code. Code is kind of like copyright. It’s legacy once it hits the disk (“fixed form of expression”). I can’t recall exactly where I picked this up but it’s a distillation of an opinion that I’ve formed during my career working with codebases at varying levels of maturity. I’ve seen cheeky references to legacy code as “any code that works”.

I’m part-way through reading an excellent book about legacy systems entitled “Kill it with fire” https://www.amazon.com/Kill-Fire-Manage-Computer-Systems/dp/1718501188

I will have to read back and see if I can trace it back to this book. Either way you should read it (Disclaimer: I have not finished it).

Responses to my comment in the one-on-one have been trickling in from the rest of the team. Everyone seems to have interpreted it a little differently but I think that each interpretation actually embodies a facet of what I’m trying to convey. The context of my utterance was a system that we needed to keep up-to-date so it makes sense to treat the system like a flowing stream. It’s going to have some dynamics all on its own, and to make sure we can guide it we need to manage it. (Software is performance? Software development is a generative system? Too much for this post).

Managing the entire lifecycle of code from the start means treating new code as legacy. Sounds crazy but it makes you think about your dependencies, are they well maintained? Is the code using a build system that is different than the rest of the system? What about testing plans (not the unit tests themselves – the strategy for maintaining quality over time).

The thing I didn’t expect was that it triggered a revelation in one of the other Senior Engineers that the decisions and technologies are always up for discussion. A codebase at rest can be an archaeological dig at worst and a collection of Just-So stories at best. It’s encouraging that this is getting people to ask why. I think that’s pretty amazing.

Immediacy

The last few years I have been going out to a festival in the Nevada desert. You can probably guess which one but it’s not really important for this discussion. One of the core tenets or “principles” of that gathering is embracing immediacy. In the context of the festival that generally manifests itself in a kind of a fractally recursive series of pleasant distractions. One of our rules is if someone asks “what is that?” we have to go find out.

Naturally in the real world this lack of prioritization can lead to trouble. However, I have begun to explore the idea of optimizing for immediate action. There is rarely a better time to do a thing than the current moment. Unfortunately this comes at the cost of potentially missing high priority tasks that just don’t happen to be immediately in front of us.

There are several “triage”/inbox zero type of strategies that rely on one pass over potential tasks to see what can be knocked out in 5 minutes or less. My approach here is to try to optimize for speed so some things can be approached in smaller chunks of time.

Digressing a bit here – a useful thing done in a small amount of time is also dependent on there being a nice stopping point for the task. So really there are 2 things to optimize for. Quick to start and quick to stop. Stopping is important because the gains can be lost if the exploration is forgotten (if it is an info-task) or the intermediate result is misplaced only to be done again next time it’s immediately before us.

Before I do any more explaining I will give an example. Say I am looking at a broken power jack on a circuit board I’m holding. I think, ok just need to unsolder this thing. I have more barrel plugs in a little drawer in my organizer. The soldering stuff is in a plastic bin in a stack of bins. What do I have to do to accomplish this task? There are move moving parts than one would think. I have to put the board down somewhere. That might be a challenge if my bench isn’t clear. I need to pull the soldering bin out from under some other stacked bins potentially and figure out where to put it while I pull out the desoldering iron. I have to find a place to plug it in.

Immediacy can come into play in many places here. One is realizing that a good place for a power outlet is right in front of the bench, maybe mounted to the leg of the bench. Having a ready supply of double sided tape would be something that could enable this. The ability to slap a power strip right there and use it right away for this task is what I’m talking about. It should take less than 5 minutes to evaluate whether it’s a good idea, and if it works out it’s much easier to remember and maybe I’ll use screws or something more permanant. Kind of like paving the worn paths across the grass.

Other potential things in this example are, “where do I put the hot iron?” I have a little soldering iron holder on a clip now. The things that led to this was my cat knocking something heavy onto my soldering iron holder, breaking it off of the base. I delayed throwing it out since I didn’t have anything else and figured I would fix it eventually. However it was broken on the metal and I was thinking I could weld it so fast but I had to remember to take it to my shop, which never happened. I had some steel wire and started looking for somewhere to lash it, and I had a binder clip lying around and the idea worked and stuck. Pretty contrived, but how to enable more immediate actions like this?

One touch systems. I did the clip-on soldering iron holder while the iron was hot in my hand. This is what I would call “one touch”. I was already holding the problem. Another example of this would be washing a pan before letting it go after serving out the food it contained. It’s probably not going to get any easier to clean after it’s cool.

Go to the H1 of the class

I’m refactoring some HTML right now and it occurs to me that headings are problematic for non-frontend devs and inconsistently used at best among frontenders.

When I was at Yahoo a big part of the styling philosophy revolved around Atomic CSS. These were the kind of predecessor to React’s JS-like inline style declarations. Styles were generated by a post processor based on descriptive short class names. Eg, display: inline-block becomes D-ib for example (not the real syntax anymore but similar idea). The compiler would generate a the stylesheet to make the class name work, eg

.D-ib {display: inline-block}

The reason I’m bringing this up is that headings typically have some styling associated with them. The base stylesheets of browsers define them, and they are block-level at the very least. Most CSS template systems further define them and/or do some kind of “reset” that redefines them.

At a basic level, heading tags are semantic, telling an agent at what level in the hierarchy of information this markup resides. That’s basically it. I’m coming to think that in the age of components we need to treat headings as relative and purely semantic. On one extreme I see developers using just div everywhere. This gets around the problem of default stylings but throws away any attempt at semantic hierarchy. Maybe they sprinkle in some non-standard class name scheme for that.

Ideally H tags would have no styling different than div. Components always start with H1 at their top level, and would be rewritten during page composition to the appropriate depth based on “root” heading level. Styles are applied separately. Possibly inverting the “additional semantics” pattern by supplying “additional styling” even using .h1 class names. This seems like an anti-pattern at first but then some common trends in React styling started as anti-patterns as well.

Simple Ubuntu Desktop with Vagrant

I wanted to spin up a Linux development environment to hack on some code that needed epoll. I could have run everything in a Docker container, but I kinda wanted to be in that environment for total hackage.

I thought maybe I’d just do it in a Virtualbox instance. Then I didn’t want to install Ubuntu or anything. Then I realized that Vagrant is supposed to solve this problem. Installed Vagrant and used Ubuntu Trusty – https://app.vagrantup.com/ubuntu

$ brew cask install vagrant
$ vagrant init ubuntu/trusty64
$ vagrant up

Then I realized I wanted a minimal desktop.

So googling.

Yep, XFCE FTW. https://stackoverflow.com/questions/18878117/using-vagrant-to-run-virtual-machines-with-desktop-environment

Then install chrome. It’s easiest to SSH into the instance and past this stuff in through the native shell.

$ wget -q -O – https://dl-ssl.google.com/linux/linux_signing_key.pub | sudo apt-key add –
$ echo ‘deb [arch=amd64] http://dl.google.com/linux/chrome/deb/ stable main’ | sudo tee /etc/apt/sources.list.d/google-chrome.list
$ sudo apt-get update
$ sudo apt-get install google-chrome-stable

 

In Virtualbox

$ startxfce4&

Forgotten Javascript

Every once in a while I get reminded of some dark corner of Javascript even though I’m developing with it every day. I give a lot of JS coding interviews and every once in a while something comes up that I hadn’t anticipated in the questions that I usually ask.

Recently someone tried to solve one of my coding problems using arguments.callee. I hadn’t thought about this variable in a while, and I realized why once I looked it up on MDN.

The short answer is that it doesn’t work in strict mode, which of course everyone is using right?

The long answer is that you are effectively looking at the call stack, which in most languages is sketchy due to compiler optimizations (function inlining, etc). In C# I have seen some code that relied on applying the attribute [MethodImplAttribute(MethodImplOptions.NoInlining)] to the function to keep the compiler from inlining it.

This got me thinking about today’s highly-optimized Javascript runtimes. Surely they can do inlining when generating target code right? It turns out that yes, they can, and even better, in the case of V8 we can trace how the optimizations are applied.

I took the example I found here and ran it myself to verify that my version of V8 installed via homebrew on OSX was performing the same optimizations.

 

 

Micro thing-uses

Hello friends. I’m still writing code every day, believe it or not. I’m back in startup land. Away from big tech land mostly. Seeing both sides of the coin has been helping me have some insights I didn’t have before on the open-source adoption of technologies that trickle down from big organizations.

In the past I have mostly held the position that Your’e Not Gonna Need It with most things involving scale or performance, or (insert x thingy). Big companies need to deal with multiple facets of scale that you don’t need when your entire engineering department fits at one desk pod.

However, there are some success stories around the adoption of some technologies and some failures. I’m not sure yet if I can draw conclusions yet on what tips the scales one way or another. I think the elephant in the room right now is microservices. Leading up to microservices was containerization.

What are the technologies? Docker. Kubernetes. Lambda. Graphql. Apollo.

These don’t seem like they have that much in common but they are all attempts at granularization of the solution space in some way.

The monolithic app has been derided since I started in the software industry. My entire career we have been splitting things into libraries, creating dependencies and fighting the ugly monolith. He’s an easy enemy. A strawman. Kill the monolith!

Companies like Microsoft, Google and Facebook have been  known to keep all their code in one big repository. A monolith? Not necessarily but in some cases yes. Death to the monolith, long live the monrepo?

The distinction between units of code reuse and units of deployment is still something that confuses the issue. Remember n-tier apps? Of course we will handle x in the n tier. In reality there was one server with all the tiers on it. Was the database a tier? How many cycles of splitting and joining database schemas did the app go through?

It was an imperfect abstraction at best. What is a tier?

Now I ask, “what is a service”? What makes a service “micro”?

This post is not for answering questions but for asking them.

When we wrote .NET remoting objects, were those microservices? Why was DCOM so hated? Were they microservices? Remember registries for COM components – service locators in Java? Is this the old new thing, microservices?

Two ideas in computer science that have endured like viruses are functional programming and the UNIX philosophy. Why are these good ideas? Maybe they are bad ideas but somehow they keep popping up like bad pennies. Functional means never having to remember that you said you were sorry. UNIX means dance to your own drum beat, but only one.

Are UNIX commands proto-microservices? Is Bash the proto-orchestration framework? We are giving way too much credit to the shell here. It was UNIX pipes that was the stroke of genius. Does UNIX have side effects? Does it ever. However it’s interesting that there are some general patterns that hold. Standard I/O, return codes, and file handles cover almost everything. Pipes are special files. Sockets can be as well. So two basic subsystems cover almost everything in UNIX.

This brings us back to making some generalizations about service models. Why didn’t CORBA deliver on its promises? .NET remoting? I’m going to put out a strawman here and say – state and side effects.

Sounds like a familiar refrain. Functional programming vilifies these two things as its raison dêtre. What if, rather than fighting state and side effects, we use these as the very definition of a service? Monads are designed to encapsulate state. Could the same be said of a microservice? What about side effects? Any time in a system that there is a state boundary, is that a microservice? What about a specific side effect?

I was going to philosophize on containers and connectivity here too, but I think I’ll save it for next time.