On-premise or not?

Azure Local was mentioned in a discussion with a few other engineers, the other month. I’d never heard of it, but I could guess it had something to do with running an on-premise instance of Azure. Apparently they’re calling it ’edge computing'.

There’s a list of things we don’t want migrated to Azure, when we think about it, such as security camera streaming and recording, database for the security cards, software-defined networking, internal hostname resolution, that sort of thing.

According to Microsoft:

We’re also introducing Sovereign Private Cloud […], which brings together our productivity server software into an Azure Local environment that can run entirely in a customer’s own datacenter, with full control on security, compliance and governance. 

I did wonder if this was related, in some way, to the fuss being made around ‘cloud repatriation’ a few months ago, which I think is itself part of an industry cycle that repeats every decade or so. A couple of the guys made references to blog posts by David Heinemeier Hansson on his company’s efforts. Apparently Basecam/HEY migrated all their stuff back to on-premise servers, and apparently they saved a small fortune in doing so. They re-invested in hardware resources that are more capable than cloud-based VMs.

I used to be very opinionated about this being the way to go. These days I’m not so sure Services can indeed be more resilient, more cost effective and easier to maintain when they’re based on core technologies: .NET DLLs running on IIS boxes, SQL Server jobs and even PowerShell scripts that transfer files between shared directories, etc. Of course, there’d be the overhead of replicating the critical stuff across multiple sites, but that’s a solveable problem. Less complexity is better, and the core technologies themselves rarely break.

The problem is there’s a point where that becomes more costly and less manageable.

For a start, HEY’s infrastructure is very different to ours. HEY could hypothetically operate their services on a cluster of physical LAMMP servers and half-decent networking kit, and it could easily be managed by a handful of sysadmins. Our organisation is in a very different situation. We do far more than HEY and Basecamp combined, maintaining around thirty interdependent Web applications, roughly 120 integrations and God knows how many APIs. Our main database server, just to give an idea of the scale involved, is effectively a supercomputer. Before we had a unified integrations platform, we needed to maintain integrations that were developed over the course of decades, using different frameworks. The costs of maintaining the infrastructure, and the workload of addressing the technical debt, were escalating.

And, unlike HEY and BaseCamp, our organisation has to be compliant with all the requirements that come with storing and processing highly sensitive data. That’s a considerable amount of ongoing work for a team of software engineers, systems architects and sysadmins. The software applications alone would need to be continually scanned for vulnerabilities, patched, re-tested and deployed within a given period after vulnerabilities were detected. Why do all that, when the compliance and liability could be handled by Microsoft?

Sam Hilsman provided a few good counter-arguments to Hansson’s - in fact one of them is roughly the point I just made. Where I disagree is with the alternatives he mentions - things like NoSQL and ‘microservices’, I’d argue, just add complexity for the sake of it.

I found one of Hilsman’s statements a bit puzzling, though:

‘On-premise systems are inherently more specialized, more custom-built, and more arcane. They often rely heavily on undocumented tribal knowledge. New hires face longer ramp-up periods. Critical knowledge gets lost when key personnel leave.’ 

I’d argue the opposite is actually the case. On-premise stuff is typically built on core technologies, and the ’tribal knowledge’ is… well, frankly, stuff every computer engineering professional would have at least a passing familiarity with. A developer could have a general understanding of what a .NET or Java application does, just from a quick read through its source code. If we run into coding-related problems, were guaranteed to find something highly relevant on Stack Overflow that could be repurposed and engineered. If we’re trying to configure a server (of which there are many types) in the Linux command line, there’s thirty-odd years of documentation readily available. I can think of half a dozen colleagues who know more about SQL Server than I do. Vendor-specific abstractions, on the other hand, require vendor-specific knowledge to work with, and often vendor-specific expertise. With some products, that knowledge literally isn’t documented anywhere beyond the vendor’s ‘community’ forum.