Software development teams faced with challenges related to developing software systems have switched from traditional software methodologies like Waterfall towards holistic, Agile methodologies such as Scrum. The latter is a project management tool that effectively deals with the complexity of modern software systems, the rapidly changing world and increasing levels of unpredictability.
Whether one currently uses Scrum or is considering its use, one cannot deny its impact on the industry. But why is that?
It is sad but true: our cognitive faculties lack the ability to deal with overly complex problems. Generally speaking, software development projects are tend to be complex. In reality, approaches based on a fully rational analysis, e.g., Waterfall, are very likely to fail. Approaches based on more empirical (inspect-adapt) approaches, e.g., Scrum, are more suitable to deal with complex projects because they are more attuned to our cognitive abilities.
This two-part post examines not only how Scrum contributes to high success rates among projects, but also how Scrum can contribute to overall job satisfaction.
The traditional way to build software – one that is used by companies big and small – is known as the Waterfall methodology. Waterfall has many variations, but they all have similar fundamentals. Waterfall’s great strength is that it is extremely logical. It dictates a clear process of thinking before building. As such, Waterfall typically begins with a detailed planning phase, where the end product is carefully thought through, designed, and documented in great detail. It necessitates writing down all the key data: requirements, design, expectations, etc. Key factors include following a plan and staying as organized as possible.
The problem is that Waterfall is rational, but humans are not. Humans typically are unable to generate all the good ideas up front. Moreover, most humans tend to avoid reading highly detailed 100-page documents. Additionally, humans have a poor ability to predict the future. As a result, the Waterfall environment can be viewed as being better suited for robots working on an assembly line than for software developers. As a result, Waterfall products fall short of expressing the creativity, skill and passion of their creators.
The Scrum framework, on the other hand, excels at eliminating risks associated with wasted time, wasted money, and poor up-front planning. Instead, Scrum practitioners focus on rapidly delivering a working, tested, business-valuable features. This is achieved by implementing an iterative, incremental process that effectively breaks down requirements into small, short “sprints” of development effort. Scrum’s empirical approach is based on the principles of inspection, adaptation and transparency. It encourages an environment which is based on individual involvement, commitment and empowerment.
The details of how Scrum works are beyond the scope of this post, but feel free to learn more about the framework by reading all about it at http://www.scrumguides.org/.
Although a marvel of evolution, our brain has yet to evolve to the point that we can excel at every aspect of knowledge work done around the world. ‘Knowledge work’ is defined as tasks that take place mostly in our mind: predicting scope, planning, estimating, considering consequences of changes, defining test cases, linking information, analyzing data, etc. So why is it that our brain can’t excel at portions of knowledge work?
Let’s have a look at some of the reasons:
- Memory – on average, our mind is capable of simultaneously maintaining seven objects (ideas, numbers, images, consequences, etc.) for up to 20-30 seconds (Miller’s Law). This limitation impedes our ability to carefully consider all the variables of a complex project.
- Attention – the ability to concentrate on one aspect while ignoring others. Humans tend to underperform while multitasking (think about the last time you answered your phone while codingJ). Switching attention between two or more tasks tends to double the time needed for individual tasks and will likely increase the number of errors. Since attention to detail is required in complex projects, this limitation impedes our ability to consider all options, estimations and predictions.
- Decision making – due to the aforementioned limitations of our memory and attention, our brains are generally not very good at making decisions. Instead of basing our decisions on a full rational analysis of all available data (which is almost never available), our brains have evolved to make quick decisions based on the immediate environment. We use simple rules like ‘what has been similar in the past?’, ‘what is my gut feeling?’, ‘what do others think?’ etc., to make decisions.
- Thinking – have you ever asked yourself why, while thinking, you find yourself writing down notes on paper? This is because our minds require interaction with a real environment to help it think. The act of writing down and pondering our thoughts helps us think and come up with new ideas. We use the environment as an extended mind to offload both our memory and our thought processes into a lower energy-consuming ‘device’. This frees our limited minds to consider new actions and ideas.
As mentioned, the Waterfall approach might work if humans could ‘think’ like computers: making predictions based on complete data and following step-by-step procedures produced at an early stage. The first problem is that ‘all the data’ is usually never present. The second problem is that our predictions are almost never accurate. The result? Project overruns, developers putting in overtime, lost money and widespread frustration.
In the next post, we will examine how the Scrum framework is better suited for the manner in which humans operate in completing complex projects in the software development industry.
See you there!