Low friction development environments

While setting up a sample project from an unnamed large vendor the other week I was disappointed by having to read large amounts of documentation and run various bits of script to install dependencies and set up infrastructure. We live in a world that has tools old (Make) and new (Docker) that can be combined to make onboarding engineers low or zero friction.

Wobbly onboarding

At San Digital we keep in mind some of the original intent of the Agile software movement. Documentation is great, but a working tool is even better. Particularly when you have a problem where code and document are quite similar, like setting up and performing actions on a software project.

We have all been there, you arrive at your new job, sit down at the computer and check out the code you will be working on. The directory is littered with various READMEs and dangerous looking shell scripts, you ask one of your colleagues how to build it and they send you a link to a wiki page last updated 3 years ago. Eventually you crack and ask to pair with a tech lead, who begins hammering away in a terminal saying things like "oh I remember now". Eventually you get given a few nasty looking shell snippets to build, test and deploy the project and you lean on your .zsh_history like a true professional.

Things have improved a little over the years, even as our toolchains have got more complicated. Continuous integration pretty much guarantees that at least something can build the project from scratch, and preferably not on a My Machine. It's not just build and test however, there are many other housekeeping operations, test data generation, setting up local development infrastructure associated with a modern software project.

I don't want to have to trawl a wiki and read a load of probably out of date TLDR; to get a project up and running. Nor do I want to waste the time of my colleagues asking exactly which version of frumpy.js needs installing globally to stop the build screaming at me in red.

There's no need for this, at all. And in typical 'Draw the rest of the owl' style I will present a simple but illustrative example - the build system for this website.

Docker

Pretty much everyone uses docker now, unless you have been living under a rock in a space like embedded software development. Package the exact versions of a tool and OS you want, spin it up. Really useful. You should try it if you haven't already.

The tool used to build this website is a rust static site generator called Zola (I like Rust, it's beautifully designed from build system to language and has a remarkably talented and well governed community). To dockerise an instance of Zola we use the following Dockerfile, which is simple enough to just leave in the root of our site's git repository.

FROM bitnami/minideb AS builder
ENV ZOLA_VERSION v0.12.2
RUN mkdir -p /workdir
WORKDIR /workdir
RUN curl -L https://github.com/getzola/zola/releases/download/$ZOLA_VERSION/zola-$ZOLA_VERSION-x86_64-unknown-linux-gnu.tar.gz | tar xz
RUN mv zola /usr/bin

This uses a minimalist debian base, downloads a release of Zola from github, untars it and stuffs it in /usr/bin. A terrible thing to do to your development machine but you can be mean to docker images.

So now we should be able to run a dockerised Zola on all sane operating systems, with no system dependency other than docker.

We can spin it up and use it to build and serve our website with:

> docker build . -t san-zola
> docker run -p 1111:1111 -v `pwd`:/work -w /work -it san-zola zola serve

Now all we need is a nice way of running it, rather than pasting that snippet into slack.

The re-entrant makefile trick

The only other thing we are assuming is installed is some version of make, which is a good bet. Make has some nice affordances, even when you are not really using it to ..make things. It is a terse, but pretty readable declarative syntax that can manage dependencies and integrates well with most shells. You can usually type make, then tab out the options.

We have two things that we need to do for this smooth owl sized projects - statically compile a version of the size. And do the same, but while watching for changes and serving content.

.PHONY: docker
docker: 
	docker build . -t san-zola

docker-%: docker
	docker run -v $(PWD):/work -w /work -it san-zola make $*

.PHONY: build
build:
	zola build

.PHONY: serve
serve:
	zola serve

Our first target 'docker' unsurprisingly builds the dockerfile into an image. .PHONY is a way of telling make to not behave like make, without the PHONY it will start checking for a file called 'docker' which is probably not what you want when it happens.

Our second target is the re-entrant trick. The docker-% target passes whatever is on the other side of the hypen to as the $* variable. Allowing us to call make inside the docker container with the other target. So when we call make docker-serve this happens:

  1. Make sees the docker-% target is matched
  2. docker-% has a dependency of docker so make calls the docker target
  3. Make calls docker build . -t san_zola, ensuring we always have an up to date docker image
  4. Make calls docker run -v $(PWD):/work -w /work -it san-zola make zola serve
  5. Zola spins up and does what we want

This is an obviously simple example. But on projects San Digital lead and work on, we will ensure that all required operations for day to day developer activity are made and kept this frictionless.

Jumping into the FHIR - type systems and objects

We have been doing a deep-dive on FHIR implementations and tooling following our initial FHIR investigation. A critical area of investigation for any system, particularly a large distributed system with many clients and peers that need longevity and guided evolution is its type system. Use of a strict type system can have many benefits - interoperability, governance, data quality, security, performance, developer experience, reduced component complexity and the ability to evolve services with confidence

Integrating with Events

The San Digital team has worked with numerous organisations in both the public and private sectors to transform their applications architecture into a flexible and business-focused model. Working with events at scale is key to maintaining individual teams' agility.

The process of building a mobile app

The team at San Digital has extensive experience developing apps for mobile devices, smartwatches, and smart TVs; using native and hybrid technologies (and everything in-between!) including using Rust for complex comms.

Low friction development environments

While setting up a sample project from an unnamed large vendor the other week I was disappointed by having to read large amounts of documentation and run various bits of script to install dependencies and set up infrastructure. We live in a world that has tools old (Make) and new (Docker) that can be combined to make onboarding engineers low or zero friction.

Cloud-native FHIR platforms

Continuing our series of posts on web protocols, we have been investigating more specialist protocols, in this case, "FHIR". We have produced a document based on our research, investigations and experience.

Team Structures

Multiple team structures can work to deliver software projects. There is no real one size fits all, however, there are common components that can be seen across different structures. At San Digital we believe that Engineer-led teams deliver great results for short duration high-impact projects.

Rules of the Road

This is called rules of the road but they aren't rules they're more guidelines, so they're rules until there is a good reason to ignore them.

Estimating and delivering defined outcomes

Recently there has been a shift away from time and materials projects towards defined outcomes, driven by various legislative changes, specifically IR35, but also cost control in the procurement function of larger organisations.

The San Digital Stack

San Digital has been designed as a remote first business from inception, on the assumption that it's easier to add offices later if they are necessary in an agile way. To work in collaborative way completely remotely takes a carefully thought out set of tools. Some of the ones that we use are really standard and some are a little more interesting.

Test driven design, or planning driven development

Design processes in most business software development resemble peer review or crowd-sourcing. A putative design is presented to peers, who will do their best to find problems while the originator of the design defends it against these challenges. Ideally, where they are demonstrated incorrect or incomplete the process will iterate and an updated design produced and defended.

A human view of computer vision at scale

Computers analysing and acting on what they see is not science fiction or even a new concept, it has been a reality of humankind's drive towards hyper-efficiency since around the time I was born.

Building scalable frontends

Scaling frontends is hard, actually scaling all codebases is hard, frontends just happen to be particularly visible and have a tighter feedback loop and a higher rate of change. As with all codebases, it is in principle possible to scale development through standards and integration processes, but these are a poor substitute for communication. Once development moves beyond the scope of a single team, either progress slows to take into account of different processes or implementations drift away from each other over time. Teams need to find a way to operate independently towards a goal.

Cross platform native mobile development with Rust

San Digital have extensive experience of mobile development and the use of Android as an embedded operating system. We treated android as a deployment target target for Rust firmware as well as writing our intricate real time communications component for both iOS and Android. This approach has advantages, you can maintain a single code base for a complicated communications layer, while also taking advantage of the full native capabilities of each platform

The evolution of web service protocols pt 2

At San Digital, some of us have been building things for people since the dawn of the web. Our historical perspective helps inform us about technological culture and trends today, almost compensating for the creaking knees.

The evolution of web service protocols pt 1

At San Digital, some of us have been building things for people since the dawn of the web. Our historical perspective helps inform us about technological culture and trends today, almost compensating for the creaking knees.