Agile began as a complementary methodology for developing software in situations where what will work is not previously known or well understood and needs to be discovered or created through experimentation.
Agile does not replace the traditional approach, known as the Waterfall methodology. In situations where what works is well known, the Waterfall methodology is very effective, and efficient and should be used.
For example, in constructing a skyscraper, which is a very complex process, the knowledge of how to build such a structure is well understood by civil engineers. Similarly, building an airplane, even a new type of airplane is well understood by mechanical, electrical, and aerodynamic engineers. The Waterfall approach with its methodical and sequential steps consists of
1. developing a requirements document that details all the specific requirements,
2. designing a product that meets those requirements, (often a model is created at this stage),
3. building the product,
4. testing and making final touches,
5. delivering the final product
These sequential steps work extremely well in ensuring that the construction is done according to the pre-agreed design within budget and on time.
In summary, when the issue is more about executing a well-known process, the Waterfall is the best approach.
But what do you do when the process for building the product is not well known or there is a lot of uncertainty about what works and what doesn’t? The only effective approach in this situation is to experiment, sometimes referred to as trial and error, and use those experiments to discover what works. There is a highly systemic proven methodology to efficiently experiment and iterate one’s way to what works. This methodology is known as Agile. Agile began as a better way to develop new never before never-before-created software and technology. The requirements were often guesses, or fast evolving, and the reliant technology was unproven. Using Waterfall ignored these crucial realities and proceeded as if everything was known. As a result, the delivered software didn’t meet expectations, was not delivered on time, and was way over budget. In short, a defective product was delivered way late and costed much more. Everyone, the business managers, the client, and the developers were all frustrated. There had to be a better way. These problems all arose because the inherent uncertainty was ignored. Agile was born out of embracing uncertainty rather than ignoring it.
Agile has evolved beyond software development and has been successfully applied to many varied areas such as creating startups, managing transformational innovation, and many other situations where there is a high degree of uncertainty.
Before we delve into how to successfully implement Agile, we need to clear some pernicious misconceptions.
1. Agile is the modern way to do product development and replaces Waterfall. No, both Agile and Waterfall have their uses. Using Agile when the product development process is well known is a monumental waste of time. Use Waterfall. However, if the process is unknown, using Waterfall with lead to costly through away work and resources since by the testing stage it is too late and too expensive to make major changes or pivot. In these uncertain situations use Agile. The two approaches complement each other.
2. Many business leaders will embark on becoming an Agile organization. We will do Agile from now on, is their rallying cry. What they really mean is we want to do things faster, easier, and cheaper. There is nothing Agile about this goal. What they really mean is that they want their business to become more efficient. Don’t confuse the two. As stated above, applying Agile in situations where the processes are well-known is disastrously inefficient. In these situations, the goal should be to efficiently execute the Waterfall approach. Implement Agile only in situations where what works needs to be discovered.
3. Being Agile is the latest corporate fad. Well, it isn’t a fad it is a survival necessity for areas of a business where both the competitive and economic environments are rapidly changing. In areas where the environment is relatively stable, and those do exist, implementing Agile is a fad.
If someone were to ask you “What is Agile?”, how would you answer it? Pause, think, and formulate an answer before you read on.
Compare your answer to this one.
Agile is a proven structured methodology for rapidly problem-solving within highly uncertain situations by starting an end state, creating an informed guessed solution, then testing, iterating, and repeating until a good enough solution is created.
This answer is not to be pedestalized as the epitome of the only and best answer. There is no such thing and there are many equally useful variations. Nevertheless, there are also many bad rambling unclear, and in the end, not useful answers.
Note, that this answer packs a lot of information using a MECE structure in a way that makes it clear and memorable. In other words, it follows the effective Elevator Pitch MECE structure of describing who/what it is for, what it is, and how it works all in one short sentence. That is the power of an Elevator Pitch (Click to learn more about Elevator Pitches)
So, let’s unpack it.
Who/What is it for?
It is for someone who wants to solve problems within a highly uncertain situation. What makes a situation highly uncertain can itself be unpacked in terms of problem ambiguity, no proven existing solutions, and no clear starting point.
What is it?
It is a proven structured rapid problem-solving methodology. The structured methodology can be unpacked into its sequential steps.
How it works?
A brief description of what needs to be done. This consists of starting with an end state, creating an informed guess for a solution, testing, iterating, and repeating until a good enough solution is reached. Each of these components can then be unpacked. Which is what we will do next.
The Agile Methodology
Start with the End State
Create a description of what the target end state looks like.
Make the description clear with specific must-have outcomes of End State features, not the features themselves.
Test for a common understanding of the end state amongst the solution team and other important stakeholders and iterate until there is a common understanding.
The description needs to be good enough (fit for purpose) not perfect.
Search for what exists.
Research existing solutions that show some potential for achieving some aspect of the End State. Select the most intuitively promising solution.
Create the Starting Solution
Remix and repurpose the selected solution and make other modifications to achieve the End State.
Test the Starting Solution against the End State.
If the Starting Solution tests to be good enough against the End State, then stop. You have found the solution.
If not, fix what is not working with the Starting Solution. This fixed solution becomes the new Starting Solution.
Iterate.
Repeat Steps 3 to 5
Pivot
At some point, there may be no good enough solution. In this case, you need to pivot. That is, create a new End State and start the entire process again.
When you first use Agile, you will be tempted to get each step right before you move on to the next step. After all, that is what you have been taught and rewarded for up to now. It is what Waterfall demands. Agile emphasizes rapidly but thoughtfully completing each step over taking time to do each step exactly right. At each step, roughly right is good enough. Agile emphasizes rapidity over rightness because what is exactly right is not known in advance. If it were known you would use Agile. You would use Waterfall. Rightness evolves through iterations. That is the power of using Agile when solving problems within a highly uncertain environment.
What strategies can be used to balance the need for a clear end state in Agile with the reality that the desired outcome might evolve as new information is discovered?
The Agile methodology works well when there is no existing known process to come to the end stage. However, are there any situations where combining the Agile methodology with the Waterfall methodology creates benefits? For example, there is a known process that needs to be refined and modified. In that case, is there any way to combine these two methodologies?
Questions on Agile application: How can we measure the effectiveness of Agile when the end state or solution is still evolving through iterations? What specific factors indicate that it's time to pivot rather than continue iterating on the current solution? In terms of rapid iteration, what techniques can be used to foster stakeholder buy-in for "good enough" solutions in an Agile process, especially when they expect perfection?
In a real life situation where you may not have all the details upfront, how do you decide what the end state should look like?
When there is no good solution after using Agile and we are forced to pivot to a new end state. how do we know if this new end state will satisfy our initial goal enough to be worth doing? Is it ever more beneficial to accept a solution that may or may not be 'good enough' rather than pivoting to a much lesser end state?