Rules of the Road

This is called rules of the road but they aren't rules they're more guidelines, so they're rules until there is a good reason to ignore them.

General Architecture

Do the work on write/update wherever you can.

Flat file sites are FAST.

Caches are bad. If you must cache then let http do it for you. Then the data source retains control of its data and can make a load consistency trade off.

Avoid using client side JavaScript without a very good reason - don't do something with JS that could be done another way.

If you must use JS then it should only be for personalisation and client side composition.

Micro services move the complexity from compile time to deployment time. Are you smarter than a compiler? There are probably ways around this but they involve building a compiler for hardware.

State

Where is the state? If you’re doing anything of use there must be persistence somewhere. Understand the scope of the persistence. It’s actually what defines the boundaries of the service.

Queue’s count as state. (In a way persistence is queues, or at least order)

Communication

Make communication explicit. If two systems communicate make sure it is understood and not implicit through a database or some other storage medium.

Communication should be decoupled wherever possible, contracts.

If things are very closely coupled or even really chatty then they should probably be in the same process and any complexities controlled by a compiler.

Bandwidth Delay/Latency Product: You can have all the bandwidth in the world but latency will kill you most of the time. See notes.

Brewers CAP Theorem (Eric Brewer)

  • Consistency (all nodes see the same data at the same time)
  • Availability (a guarantee that every request receives a response about whether it succeeded or failed)
  • Partition tolerance (the system continues to operate despite arbitrary partitioning due to network failures)

Pick any two. Its more complicated than that though. (CAP is uncool now)

Security

Don’t do your own cryptography, ever.

Careful with the keys.

Security is very very hard don’t reinvent the wheel.

Consider the difference between Authentication and Authorisation.

If you need an audit trail that isn’t trivial then consider an event store. If that’s too complex then consider a command model, every request is a command that gets logged as run or failed. If you've built a good API you might get this for free in the logs. Depends on the purpose of the audit trail.

Non Functional Requirements

Do you have non functional requirements? Trick question you always do. Can you test that they are being met?

What are the dimensions: Request, Storage, Response Time, Transactions etc. Lots of users isn’t a dimension.

If they doubled what would happen, what about an order of magnitude. You don't have to build this, jsut be able to answer the question.

Always prototype with real infrastructure.

Monitoring

Start with errors and work up from there.

With something that looks like REST you can get a very useful log just from the HTTP log. Consider the fact that if it’s not useful you might not be as RESTful as you think.

Aggregate information somewhere and start graphing and alerting.

Extra points for doing clever correlation stuff with the messages.

Automating the response to the above is talked about a great deal. If someone at some point does something other than send a text message it will make me very happy.

Code

Management of complexity is everything. Really it is.

Consider 6 dice as a system, 6^6 combinations, untestable. 1 die only has six states, much easier to test. This rule applies at every level of abstraction.

Every problem can be soloved with more layers of abstraction - except too many layers.

Side effects will kill you. Every. Damn. Time.

  • Strings are bad
  • Side effects are bad
  • Reusability is overrated
  • So is inheritance
  • Lean on the compiler
  • The more dynamic the language the more you need tests to exercise the code - at an extreme you are building your own compiler
  • Strings are bad, repeated for effect.
  • Use languages from the 70s, not the 60s
  • Small systems do not have to mean micro-services. Can be code modules, libraries etc.

Latency Comparison Numbers

Some of these numbers will change over time because of performace improvements - some of these won't because of the speed of light. Knowing the difference is fairly important.

  • L1 cache reference 0.5 ns
  • Branch mis-predict 5 ns
  • L2 cache reference 7 ns 14x L1 cache
  • Mutex lock/unlock 25 ns
  • Main memory reference 100 ns 20x L2 cache, 200x L1 cache
  • Compress 1K bytes with Zippy 3,000 ns 3 us
  • Send 1K bytes over 1 Gbps network 10,000 ns 10 us
  • Read 4K randomly from SSD* 150,000 ns 150 us ~1GB/sec SSD
  • Read 1 MB sequentially from memory 250,000 ns 250 us
  • Round trip within same datacenter 500,000 ns 500 us
  • Read 1 MB sequentially from SSD* 1,000,000 ns 1,000 us 1 ms ~1GB/sec SSD, 4X memory
  • Disk seek 10,000,000 ns 10,000 us 10 ms 20x datacenter roundtrip
  • Read 1 MB sequentially from disk 20,000,000 ns 20,000 us 20 ms 80x memory, 20X SSD
  • Send packet CA->Netherlands->CA 150,000,000 ns 150,000 us 150 ms

Credit By Jeff Dean:http://research.google.com/people/jeff/ Originally by Peter Norvig: http://norvig.com/21-days.html#answers https://gist.github.com/hellerbarde/2843375

Sustainability - engineering a sustainable future

At San Digital, we believe the work we do today should positively impact the environment and not compromise the ability of future generations. It seems obvious, but large organisations continue to think sustainability is a box-ticking exercise and an inconvenience.

Agile Businesses

There are more than enough articles about agile software development; this post is about agility elsewhere.

Jumping into the FHIR - type systems and objects

We have been doing a deep-dive on FHIR implementations and tooling following our initial FHIR investigation. A critical area of investigation for any system, particularly a large distributed system with many clients and peers that need longevity and guided evolution is its type system. Use of a strict type system can have many benefits - interoperability, governance, data quality, security, performance, developer experience, reduced component complexity and the ability to evolve services with confidence

Integrating with Events

The San Digital team has worked with numerous organisations in both the public and private sectors to transform their applications architecture into a flexible and business-focused model. Working with events at scale is key to maintaining individual teams' agility.

The process of building a mobile app

The team at San Digital has extensive experience developing apps for mobile devices, smartwatches, and smart TVs; using native and hybrid technologies (and everything in-between!) including using Rust for complex comms.

Low friction development environments

While setting up a sample project from an unnamed large vendor the other week I was disappointed by having to read large amounts of documentation and run various bits of script to install dependencies and set up infrastructure. We live in a world that has tools old (Make) and new (Docker) that can be combined to make onboarding engineers low or zero friction.

Cloud-native FHIR platforms

Continuing our series of posts on web protocols, we have been investigating more specialist protocols, in this case, "FHIR". We have produced a document based on our research, investigations and experience.

Team Structures

Multiple team structures can work to deliver software projects. There is no real one size fits all, however, there are common components that can be seen across different structures. At San Digital we believe that Engineer-led teams deliver great results for short duration high-impact projects.

Rules of the Road

This is called rules of the road but they aren't rules they're more guidelines, so they're rules until there is a good reason to ignore them.

Estimating and delivering defined outcomes

Recently there has been a shift away from time and materials projects towards defined outcomes, driven by various legislative changes, specifically IR35, but also cost control in the procurement function of larger organisations.

The San Digital Stack

San Digital has been designed as a remote first business from inception, on the assumption that it's easier to add offices later if they are necessary in an agile way. To work in collaborative way completely remotely takes a carefully thought out set of tools. Some of the ones that we use are really standard and some are a little more interesting.

Test driven design, or planning driven development

Design processes in most business software development resemble peer review or crowd-sourcing. A putative design is presented to peers, who will do their best to find problems while the originator of the design defends it against these challenges. Ideally, where they are demonstrated incorrect or incomplete the process will iterate and an updated design produced and defended.

A human view of computer vision at scale

Computers analysing and acting on what they see is not science fiction or even a new concept, it has been a reality of humankind's drive towards hyper-efficiency since around the time I was born.

Building scalable frontends

Scaling frontends is hard, actually scaling all codebases is hard, frontends just happen to be particularly visible and have a tighter feedback loop and a higher rate of change. As with all codebases, it is in principle possible to scale development through standards and integration processes, but these are a poor substitute for communication. Once development moves beyond the scope of a single team, either progress slows to take into account of different processes or implementations drift away from each other over time. Teams need to find a way to operate independently towards a goal.

Cross platform native mobile development with Rust

San Digital have extensive experience of mobile development and the use of Android as an embedded operating system. We treated android as a deployment target target for Rust firmware as well as writing our intricate real time communications component for both iOS and Android. This approach has advantages, you can maintain a single code base for a complicated communications layer, while also taking advantage of the full native capabilities of each platform

The evolution of web service protocols pt 2

At San Digital, some of us have been building things for people since the dawn of the web. Our historical perspective helps inform us about technological culture and trends today, almost compensating for the creaking knees.

The evolution of web service protocols pt 1

At San Digital, some of us have been building things for people since the dawn of the web. Our historical perspective helps inform us about technological culture and trends today, almost compensating for the creaking knees.