From good to great #1: Be agile, not fragile.

Part 1 - Be agile, not fragile.

Let’s kick this first article off by talking about agile. I strongly believe that the agile movement, or maybe the agile revolution is a better word, has brought huge benefits to the business as a whole. That being said, I also believe that agile in general is missing some vital business related ingredients. An important part of the purpose with agile has always been to bring business and tech closer together. I believe that this business focus is rarely emphasized enough. My experience is that agile methodologies like Scrum, Kanban and others, are still very tech- and user-driven, resulting in fragile software solutions which do not align well with the business. Involving the business in the development process for real has the potential to take agile software development to another level, allowing us to be agile but not fragile.

Mind the gap - Expectations vs. delivery

A few years ago, the agile movement totally changed how we thought about, planned, developed and released software. We went from working in framed, closed phases where each phase took quite a long time and had to be fully completed before the next phase started. As you all probably know this is what we today talk about, usually not in particularly flattering terms, as waterfall processes. All requirements were investigated and analyzed before we even started building the actual software and most often the people working with requirements weren’t the same people that developed the software.

When the requirements phase was finished the requirements were wrapped, ribboned and handed over to the development team. This huge pile of requirements were from there on seen as the sole truth and the software was built from start to end based on these requirements, often without any significant communication with the customer. After a few months, or even years, the end product was delivered to the customer and of course most of the time we completely missed the target. The customer got a terribly expensive piece of software which actually did more or less exactly what the requirements said it should do but despite this, the customer was dissatisfied since it was not at all what they had expected.

This gap between the expected and the delivered was mostly due to two problems. One that has been taken care of by agile methods and one that actually still remains in most software development projects. The first problem was that requirements and expectations tend to change quite a lot over time. When releasing the software, the business had changed, making the initial requirements established in the early requirements phase obsolete. In addition to this, the true expectations of the customer, which was probably never even noted, had also changed but since we were so occupied by coding instead of communicating with the customer, those expectations passed us by.

The agile movement started to catch speed in the early 2000s with the release of the agile manifesto, co-authored by a bunch of extremely influential people, including Kent Beck, Martin Fowler and Robert "Uncle Bob" Martin. These guys were, and still are, more or less considered rock stars of software development. Agile was a counter-reaction to the fact that static, phase-based development processes led to such a high degree of failed software projects. Instead of working in closed phases in a waterfall manner, agile introduced the mindset of working in short iterations in close dialogue with the customer. In each iteration, a smaller piece of related requirements is analyzed and refined together with the customer. An isolated and limited chunk of functionality is then developed based on the newly refined requirements. Thus, agile brought the ability to react and handle changes quickly as well as a continuous soundness check from the customer, ensuring that we are actually on the right track, building the right stuff.

Halfway to greatness

No doubt, agile has proven to be highly beneficial and has brought a huge leap forward for the software development industry. Although we can treat ourselves to sit back and laugh about the jackasses we were back in the days, I will argue that agile in itself is no silver bullet and that it has only taken software development projects halfway. I strongly believe that agile, as well as most other software development methodologies used today, totally lacks some vital ingredients, resulting in us still often missing the target, if not as badly as before.

Before rumbling on, let's complement my thoughts with some hard facts. Every other year the Standish Group releases a study called The CHAOS Report. This study researches the success rate of IT projects based on agile and waterfall methodologies and is often considered an authority when it comes to the success rate of IT projects.

 

The investigated IT projects are divided into the three categories succeeded, challenged (i.e. partly failed) and failed. What all CHAOS reports released in the last decade (the most recent being from 2019) boils down to is that agile processes have greatly improved the rate of succeeded projects and have also decreased the rate of failed projects accordingly. This is great of course, but one really interesting thing, which raises some questions and supports my somewhat cocky assertion that agile might not on its own be the best thing since sliced bread, is that the rate of challenged projects still remains about the same for agile and waterfall projects.

Don’t ask the user

So, let’s take a look at the million dollar question. How come that agile projects still partly fail no less than half of the time!? This shouldn’t be, should it!? I mean, come on, this can’t be right. We are working in highly skilled, self organizing development teams, continuously refining the requirements and delivering software features regularly every other couple of weeks for the customer to feedback upon, talking back and forth to the end users about what they want and what they need.

Now I’ll stick out my precious neck again, risking being cut down and mocked by every software developer and project manager, not to mention UX-specialist, with an ounce of self-respect.

For starters, I believe that the high amount of challenged agile projects is caused by an unfortunate gap between requirements, delivery and true business needs. Well that wasn’t that controversial, was it? So, let me ask these developers, project managers and UX-specialists a fairly simple question. How should we find out what the software we are going to develop is required to do? Now, since this is maybe the most sacred truth within the software development community, being taught in each and every software development class and education all over the world, I would say that there’s a fairly high probability that your reply will be “Ask the user”.

Well, “ask the user” sounds like a wise and reasonable answer I guess. After all, the users are the people that are going to use our new shiny, expensive software and we want to keep them happy, nothing strange about that. So, here it comes. What if I say that asking the user is not a good idea? What if I say that most users don't have a clue about what the business actually needs.

Let’s take a step back. Before you get the impression that I believe that users are incompetent morons, I’ll say that I definitely do not believe that. On the contrary, the users are the very best at what they do and for the most part they are highly competent and skilled. That being said, I believe that one very promising way to overcome the gap between requirements, delivery and true business needs is to embrace a concept known as business driven development. As the name implies, the key is to let the customer’s business needs, business goals and business problems be the things driving our software development processes. The whole purpose of the software we are building is to support and improve the way the customer is doing business. Otherwise the customer wouldn’t be investing in the project in the first place. In reality this means that we must have a business perspective in everything we do and for each and every software feature we develop we must have an understanding of why it is being developed and what value it brings to the business.

The problem with agile methods is that they still tend to be highly user-centric.

Going back to the users as the source of requirements, I actually believe that we should ask them about their preferences and needs but we should not start there and they should not be our only source of requirements. Users in general know a great deal about the low level details of their particular work. The problem though is that users in general have a limited understanding of the broader high level business issues. By focusing on the users when formulating requirements we start building our software solutions from the low level details in a bottom-up approach and thereby risk missing the bigger picture. Let’s consider an artist like Da Vinci, working on his masterpiece Mona Lisa.

I’ll bet he didn’t start by painting the eyelashes, rather with broader brush strokes, setting the overall perspective for the painting before continuing with the details. We should do the same when building software, working in a top-down approach. To maximize our chances of building what the customer really needs and expects we have to understand the business needs, goals and problems and how each single feature or component we build fits into that business. It is only then we can build software that actually fits the business needs and can evolve, grow and change together with the business in the long term, without deteriorating into a smelling pile of mud.

The problem with agile methods is that they still tend to be highly user-centric. If nothing else, it becomes very clear if you think about the most central work item in agile methodologies. All work is centered around the user story which, as the name implies, is totally user-focused. That means that every piece of software is built based on user requirements alone. By working with a user based bottom-up approach like this, it’s very easy to get entangled in the details. Making it even harder, these details are often conflicting since they originate from different users with different perspectives. This results in us missing the important business overview of things, which will then undoubtedly end up with the software getting messy and unstructured and thereby hard to change and maintain.

To sum things up, being stuck in the old user-centric way of thinking agile has only taken us halfway in our strive for excellence. The advantageous part of this is that there is still a lot of room for improvement. Software development companies, as for ourselves in the e-commerce business and for every other business as well, actually have a golden opportunity. By adapting agile to be more business driven we can differentiate ourselves from our competitors and gain invaluable trust from our customers. Even better, since thinking in terms of business driven development is still uncommon we’ll be able to improve ourselves greatly with simple means. Time will tell who won't be settling with being good but takes advantage of the opportunity with the ambition of being great.

The second part in this from good to great series is now released as well. In the second part we dig deeper into the importance of business driven development for business agility and for building sustainable software.

Stay tuned!