Is Agile really to blame for the lack of innovation?

One of the central claims made by Moxie Marlinspike, during his discussion at the Black Hat conference appears nonsensical, because he referred to two entirely different meanings of ‘abstraction’, and was thus comparing two entirely different things:

‘We spent the past 20 years onboarding people into software by putting them into black box abstraction layers, and then putting them into organizations composed of black box abstraction layers, […] Marlinspike posited that security researchers, who routinely probe beneath surface-level abstractions, are better positioned to drive innovation in software development.’

I am entirely in agreement with Marlinspike on the first point about ‘Agile’ being a major hindrance in the software development industry, because it’s manifestly true. I think it’s unfair to say there haven’t been advances made in recent years, though. There are half a dozen software delivery methods, just for .NET, that didn’t exist a decade ago. CI/CD pipelines are a lot more complex, for better or worse. Software engineering practices are being more widely encouraged, and that’s important in itself. Without these advances, I reckon there’d be far more data breaches than there are now. On the second point, security researchers aren’t as directly relevant to the world of corporate software development as Marlispike believes they are, so it’s hard to see how they could drive innovation beyond maybe contributing to open source projets that gain traction.

When people discuss Agile, they’re typically referring to Scrum, as I think Marlinspike was doing here. Agile itself is pretty useful, as a guideline for having a product that works after each iteration, and as a project planning tool. Scrum, on the other hand, is a corrosive and unsustainable project management philosophy, prescribing multiple things that undermine teams, mostly through overhead and disruption. My management philosophy, as a lead engineer, most definitely isn’t downstream or derivative of that. My management philosophy is that an IT Services department should be a community with a shared understanding of everything that enables our services to operate - this is what makes us highly adaptable, responsive and resilient.

As Marlinspike pointed out, Scrum (which he calls ‘Agile’) generally forces software developers to work as closed groups, where some ‘product owner’, or ‘Scrum Master’, or somesuch person with a Scrum-conferred title, mediates interactions between them and everyone outside the group. I should point out the project management overhead increases the more a development team is decoupled from other business functions.

‘Instead of allowing developers to operate from the bottom up in a way that lets them combine engineering expertise with the vision to see new capabilities in existing technology, agile teams end up siloed, working separately from each other, and without much visibility into what other teams are doing, he argued.’

Of course. One of the implicit aims of Scrum is to turn software developers into replaceble units, completing atomised work items in a never-ending cycle of ‘sprint’ after ‘sprint’. This is what keeps such teams fixated on abstractions, and on how the high-level code itself should be engineered, though it could be argued that a software engineer’s job is to be an expert specifically in that.

How could security researchers possibly solve this issue? Marlinspike apparently doesn’t have any concrete answers:

There is magic in software development, Marlinspike maintained, saying that understanding how it works is analogous to mastering wizardry in the world of Harry Potter, where the talented can change the world with nothing but knowledge and a wand.

Unfortunately, if a security researcher had anything to do with corporate software development, it would typically be doing what most developers do, focussing almost exclusively on code patterns and the frameworks du jour. There just aren’t enough hours in the day to re-engineer something that could simply be repurposed. Sure, we’d all love a career working on the really interesting low-level stuff (I briefly did digital forensics, with the idea of getting into malware analysis), but those kind of jobs are few and far between. The industry is actually going the opposite direction, with organisations outsourcing outsourcing their infrastructure, security and engineering capabilities to a decreasing number of vendors.

My answer would be for security researchers to continue doing what they’re doing. Collectively, they have already influenced the software development industry in four major ways I can think of:

• Software developers are generally more conscious of Web application security these days. We’re very familiar with all the common application vulnerabilities, how they can be exploited, and why they happen.
• The underlying frameworks are considerably more secure these days, with native features and code patterns encouraging more secure design.
• We have automated package scanning, which compares project manifests with whatever CVE database, though I do find it rather annoying that so many ‘security researchers’ appear to be running vulnerability scanners against packages for the sake of publishing arbitrary CVE scores. We don’t have the resources to patch 30-odd applications, and fix breaking changes, every two weeks.
• Sometimes security researchers do publish actionable intelligence following a high-profile security incident involving a commonly-used platform or dependency. This intelligence enables the rest of us to act quickly.