Is agile even less relevant
in a scaled data and AI world?
I’ve gone full circle, back again, and to a place of complete ambivalence with agile – perhaps like many of you have. I was a very early advocate of agile, pivoting a consulting business model and its operations to an agile state when the world was still deeply entrenched in waterfall, and I’ve been supportive of #NoEstimates and #NoBacklogs movements at various points – and yes, I fondly remember having to write hundreds of pages of functional specifications – pre-agile – to make sure we didn’t get caught out by change control. Ouch!
entirely new thinking is needed that leverages the best things about product leadership, user centred design, and modern, highly secure engineering practices, for example, and layers them onto the challenges and opportunities of our time
I’d never want to go back there but whether I like it or not agile just doesn’t feel as relevant anymore. Agile was after all created during a time when waterfall was failing, the web was really taking off and IT teams lacked diversity and true specialists. Agile became a cottage industry that was packaged up into complex frameworks, elaborate processes, training pathways/certifications, and more executive friendly (think waterfall!) frameworks. The principles and ethos of agile became diluted, forgotten, and were more than a means to an end – as opposed to what I truly care about, which is delivering early and frequent business outcomes, working directly and closely with customers (with high quality and security levels) as part of happy, self-aware, and diverse teams that are bound by a sense of purpose to do good work (either on a project, in an organisation, for our industry or for our world).
This perspective of agile as a cottage industry, when you layer on the emergence of data and AI capabilities and career professions, makes agile seem even less relevant – as we need to deal with heightened security issues (Cyber attacks are increasing significantly), moral/ethical issues brought about by AI, and the existential threat to coding as we know it (I would guess that mostly people in the business of writing code designed the agile manifesto). I appreciate that a lot of agile software professionals consider themselves as data experts, the reason for that is obvious but when thinking about data engineering, data analytics, and data science (never mind Generative AI) the profession is comparatively young and there are nuances and differences (and opportunities) that I see first hand when working with data and AI teams on a regular basis. There’s a valid reason some data teams prefer to work in siloes, on the edges, and to engage in ways that seem more waterfall – it’s because agile approaches don’t always fit the domain or ways of working, in some cases.
We missed the opportunity to eradicate the inflexible systems that indirectly cause users the real pain and anguish
My colleague James Herbert recently used the phrase “process over delivery”, which I do like when thinking about the current state of the industry. There seems to be a real delivery paralysis in the public sector where agile is being adopted in name only and isn’t being embraced with the same enthusiasm or energy as it was before. I wonder if this is because of the same sentiment I share about it is taking hold or perhaps the issues are more systemic as the bureaucracies that were ‘worked around’ pre-Covid have set in again amidst the growing worry and concern about AI – hence why most AI projects are still “experimental”. There is clearly a lot of “let’s wait and see what happens” behind the lack of endeavour to deliver transformative AI. I personally felt we did a lot of superficial work during Covid – a shiny new user interface, wonderfully designed but layered on top of legacy systems. We missed the opportunity to eradicate the inflexible systems that indirectly cause users the real pain and anguish – through the lack of data interoperability, data quality issues, and cumbersome release cycles, for example.
So, what is the answer and what is the opportunity? There’s been too much “new agile”, too much re-marketing of old concepts, and dilution of core principles to sell frameworks to outdated executives – a lot of the agile community is quite rightly fed up with that. It’s also easy to say that data teams lack agile maturity but we need to look at this from a deeper level in terms of whether agile fits the problem space. I feel that entirely new thinking is needed that leverages the best things about product leadership, user centred design, and modern, highly secure engineering practices, for example, and layers them onto the challenges and opportunities of our time – security and ethical concerns, how to do data and AI at scale, data quality issues, and an increasingly autonomous development lifecycle. The legal and financial sectors are highly regulated, and rightly so, but software has always been overlooked in comparison and that needs a serious re-think as its becoming a wild west at a time when we need to become more professional. AI-enabled development tools can be dangerous in the wrong hands.
In Pivotl, we’ve been re-evaluating what delivery means to us and our clients, where agile is in that mix and what it takes to deliver fast yet safe business outcomes with Cloud, Data, and AI. More on that in a subsequent post.
