Using Scrum to manage complex projects: A cognitive perspective (and why it is better suited for our brains)

Posted by on Comments (1)

I recently received an interesting scientific article from Gunther Verheyen titled "Getting Things Done: The Science Behind Stress-Free Productivity" (Heylighen & Vidal, 2007). The article discusses possible scientific explanations for the success of a personal productivity approach called "Getting Things Done" (GTD; Allen, 2001). The authors apply insights from cognitive psychology and cybernetics to better understand why this approach is so effective. Although GTD is focussed on individual time management, the authors mention the potential for collaborative work. Although the authors are probably not aware of Scrum (a framework for collaborative software development), their paper provides useful insights into why Scrum might be so effective when it comes to managing complex software projects. Their article inspired me to apply the same insights to Scrum and to extend it with my own. For the sake of length and because most readers will be aware of Scrum, I would like to refer readers unfamiliar to Scrum to Scrum.org.

In this post I will argue that our cognitive faculties are limited. This limits our ability to deal with complex problems. Software development projects are (generally speaking) very complex. Using the aforementioned cognitive limitations, I will argue that approaches based on a fully rational analysis (e.g. waterfalls) are very likely to fail. Approaches based on a more empirical (inspect-adapt) approach (e.g.  Scrum/Agile) are more suiteable to deal with complex projects because they are more attuned to our cognitive abilities.

This is the fourth entry into my 'Science of Scrum' series (see previous entries on learning organizations, scientific management and self-organization). I highly recommend reading the original paper by Heyligen & Vidal (in press) as well. It is a bit more scientific and takes more time to explore the various perspectives, but applied to GTD. 

About complexity

Scrum.org defines Scrum as:

... a simple framework for effective team collaboration on complex projects. Scrum provides a small set of rules that create just enough structure for teams to be able to focus their innovation on solving what might otherwise be an insurmountable challenge.

Although Scrum.org does not clarify what it considers a 'complex project'. I would define a complex project as having at least some of the following traits:

  • Unclear outcome: A project is complex when its outcome is not fully known at the start of the project. This implies a creative and innovative process. The project attempts to solve a problem for which no existing solution is known;
  • Unpredictable process: A project is complex when the steps required to solve the problem are not fully known at the start of the project;
  • Chaotic system: A project is complex when it involves a lot of variables that influence the outcome and/or the process and introduce noise. Potential variables are the people involved, understanding of the problem, quality of communication, motives, politics, external factors and technologies. I.e. projects 'suffer' because it exists within chaotic systems;
  • Duration: The longer a project takes, the more complicated it becomes. The simple fact that it takes more time causes the aforementioned factors to have a more pronounced effect on the projects's success. Also, duration is not only a cause, but also a consequence of complexity;
Akin to mathematical complexity, a project becomes more and more complex as the number of variables that influence the success of the project increase. In software engineering, we can safely consider most projects to be at least moderately complex. Even a simple website can be complex. Although the technical steps to build the website may be straightforward, translating the customer's intent into a working website often is not.

This inherent complexity has led to an overwhelming number of theories, methods, perspectives and approaches to make complex projects manageable. After all, complex projects tend to have the undesirable side-effect that they fail to deliver the desired result on time (and waste money, damage the relation with the customer, etc). Broadly speaking, there are two approaches:
  • A rational approach (Upfront plan and design): One approach is to (try to) predict the entire process upfront. This involves working out detailed specifications, plannings and charts (e.g. most waterfall approaches). This approach attempts to reduce the influence of the aforementioned variables by controlling all or most of the variables;
  • An empirical approach (Inspect and adapt): Another approach is to apply a 'trial-and-error' approach. Instead of spending a lot of time on predicting the process and the result, a 'good enough' solution is devised for (a part of) the problem and tested in practice. Based on feedback, the solution is adjusted, expanded or thrown away. This cycle of inspection and adaptation continues until the project is completed. Scrum is a good example of this approach;
The first approach might work for uncomplicated projects where controlling complexity through rational effort (planning, prediction) is reasonably possible. But this tends to fall apart when complexity increases. As the number of variables (and thus complexity) increase, the accuracy of prediction and planning drop. Although there are many reasons for the drop in accuracy, I will argue below that the human mind itself is limited in it's ability to comprehend (increasing) complexity. In more complex scenarios, a more empirical approach is superior because it does not require a full rational analysis and does not require top-down control through planning and prediction.

Why our brain is not built for software engineering

Although our brain is a marvel of evolution, it has not evolved to excel at the kind of knowledge work (Davenport, 2005) that we actually do most of the time. 'Knowledge work' refers to tasks that take place mostly in our minds, like predicting scope, setting up plannings, considering consequences of changes to complex systems, working out test cases and estimating complexity. It involves activities like linking information, prioritizing, conceptualizing, analyzing data, etc. Below are some of the limitations of our brain that make it hard to complete these tasks accurately:

Limitations of our working memory

Miller (1956) identified a cognitive limitation that inhibits our ability to comprehend complex problems. Our mind is, on average, capable of maintaining seven objects in our 'working memory' (plus or minus two). We can usually keep track of up to seven 'objects' (ideas, numbers, consequences, etc) at the same time for up to 20 / 30 seconds. This is called Miller's Law. This law is relevant because it imposes a cognitive limit to the amount of complexity we can keep in our 'working memory' at any one time. There are certainly mnemonics that can extend this capacity somewhat, like chunking. But even then, our capacity is limited to about 4 chunks. Of course this limitation impedes our ability to carefully consider all the variables that make a project complex, because there are so many variables to keep in mind. This will affect all estimates and predictions.

Limitations of attention

According to wikipedia, attention is the process of selectively concentrating on one aspect while ignoring other aspects. Attention is a mental process and a limited mental resource (Ashcraft, 2002). When performing difficult tasks, attention is required. This implies that other tasks cannot be given attention. Humans are not good at multitasking at all. In fact, switching attention between two (or more) tasks tends to double the time needed for individual tasks and increases the number of errors (Rogers & Monsell, 1995). So, answering your mail and writing some code at the same time is going to decrease productiveness. Of course this limitation impedes our ability to carefully consider all the options that make a project complex, because a lot of attention to detail is required in this case. This will affect all estimates and predictions.

Limitations of our decision-making abilities

The limited capacity of our working memory and our attention are some of the reasons why most economists and psychologists consider our brains as generally not very good at making accurate predictions and decisions. Instead of basing our decisions on a full rational analysis of all the available data (rational choice theory, Becker, 1976), we make sub-optimal choices. Our minds are bound by cognitive and other limitations (bounded rationality, Herbert, 1957). In a practical sense, this is why we are not good at detailing scope for a project upfront, consider consequences of changes and estimate the time it will take to complete tasks. This might all be possible when our minds would work like a computer (and has full information). But our brains have not evolved for this kind of work. Instead, our brain has evolved to make quick, knee-jerk decisions based on the immediate environment. Instead of applying full rational analysis, we use heuristics (simple rules) to make decisions. Examples of these heuristics are 'what do other people think?', 'what is my gut estimate?' or 'what do similar experiences tell me?'. In fact, simple heuristics can actually lead to better, quicker decisions than theoretical optimal procedures (Gigerenzer, 2002).

Limitations of thinking

Despite what you might think, humans are not great thinkers. Most of us find it very hard to concentrate on the train-of-thought necessary for continuous thought. Nevertheless, the motto of software development is often 'first solve the problem, then write the code'. This implies an active thinking phase, which is subject to (among others) the limitations mentioned above. But there is another limitation; thinking itself.

Have you ever wondered why, when thinking, you find yourself scribbling notes down on a paper to help you think? According to cognitive psychologists, this is because our minds require interaction with a real environment to help it think. This is called situated cognition (Clark, 1997). According to this theory, we think not by manipulating symbols in an entirely internal process in our mind, but by iteratively performing small actions in our environment which, in turn, lead to new thoughts and actions. Simply speaking; the act of writing your thoughts down and perceiving them again helps us think and come up with new ideas. In this case, we use our environment as an extended mind and offload a lot of the memorizing and thinking to a more reliable and less energy-consuming external memory. This frees our limited minds to consider new actions and thoughts.

How to handle our limited brains?

We have identified some important cognitive limitations of our brain. Despite it's evolutionairy marvel, our brains are not optimized for heavy-duty 'knowledge work', where the primary effort of our work lies in thinking, analyzing, estimating, prioritizing and planning. Even so, we still have to deal with complex problems. So, when dealing with complex projects, what should we keep in mind?:

  • Optimize for heuristics ('Just do it' mentality): because we are not capable of full rational analysis (not accurately, at least), we should not try to analyse, predict and plan everything upfront. This will not work. Instead, we should optimize our work for the rapid application of heuristics (Gigerenzer, 2002). It's better to hit the ground running and inspect frequently than spending valuable time trying to overthink everything;
  • Continuous feedback: because we use heuristics, our process should be optimized to provide us with continuous feedback to evaluate the success and accuracy of those heuristics. This allows us to adapt, prevent (inevitable) mistakes quickly and apply new heuristics or improve existing ones;
  • External memory: to accomodate our limited memory, we should offload as much knowledge as possible to more reliable, less memory-consuming external memory stores, like noteblocks, documents, whiteboards, Scrum Boards or the 'mind' of the team;

Together, these three considerations accomodate a process called self-organization. But before explaining this concept in more detail, I would first like to apply the cognitive limitations we've identified to traditional software development.

Cognitive limitations, applied to traditional software development

In traditional approaches (e.g. waterfalls) the defining assumption is that a purely rational approach is capable of dealing with the complexity of a project. This assumption becomes apparent in the following:

  • Scope can be identified upfront: Generally, in waterfall-based approaches, analists first clarify the problem with the customer and devise a solution to solve this problem. This requires analists and customers to be capable of comprehending all possible scenarios, solutions and potential problems (technology, politics, business, etc) upfront. The solution is translated into a full set of requirements that are signed off and implemented. In other words; the entire solution and scope are designed upfront by the customer and the analist, without any way to evaluate the completeness or accuracy;
  • Hours can be estimated upfront: Based on a list of specifications, analists try to calculate the number of hours required to implement functionalities. This requires analists to comprehend the full complexity of the project, all the steps required and take into account any possible future deviations in the process. They really have to consider all the variables. Unless they multiply their estimates by pi of course, which happens more than you think :);
  • Documents can be used to transfer knowledge: In waterfall-based approaches, documents are the primary artifacts by which knowledge is transferred. Analists devise functional specifications, customers sign off on those, developers implement them and testers test them. This is a very rational approach because it assumes that everyone will interpret the documents the same and that knowledge can be captured in words and charts alone;
  • Top down control: In traditional software development, control of the process is implemented top-down. This means that control is done through planning and prediction (by a project manager or analist). The control is then executed by project managers, analists and lead developers; 
A traditional approach may work very well if our minds work like computers, and can make accurate predictions based on full data. The first problem is that we almost never have full data. And the second problem (more relevant to this post) is that our predictions are not accurate at all. So what happens is that the project overruns, developers have to put in a lot of extra hours and money is lost? I believe that Scrum is a superior approach because it provides a framework that is more attuned to our cognitive abilities. It also provides a very different way of controlling complexity, through self-organization. Which I will explain next.

Back to science; feedback, stigmergy and self-organization

Feedback-control

A few paragraphs back I mentioned that we use our environment to leave 'traces' to further our own thinking. This is where the cybernetic concept of feedback control is useful (Heylighen & Vidal, in press). According to Wikipedia, feedback control is:

Feedback is a process in which information about the past or the present influences the same phenomenon in the present or future.

In cybernetics, a feedback system is always trying to minimize the difference between a desired and an actual state. Consider a thermostate. If an apartement thermostate is set to 21 degrees celcius, it will begin heating the apartement when temperature falls below the intended temperature. It will periodically measure temperature and turn itself off when the intended temperature has been reached. 

And what does this have to do with offloading human thinking to our environment? Heylighen & Vidal (in press) argue that the feedback control loop that is prevalent in situated cognition helps us think about complex problems by constantly performing small actions and evaluating their results (trial-and-error). If we have not reached our intended goal (the solution), we will attempt further actions that will probably decrease the distance between the intended and the actual result. In other words, our thinking is one big feedback control loop where we perform actions in the world, evaluate them and determine our next action. Writing this very post is actually a good example of this. Instead of thinking up the whole post beforehand (which is quite complex, I tell you), I wrote it during a few evenings. I used the text editor in my blog to write paragraphs, sections and jot down notes and reminders on how to link everything together. The act of writing triggered new ideas and thoughts.

Stigmergy

So, with our situated cognition, we use our environment as an external memory and to aid our thinking process (Heylighen & Vidal, in press). In the terminology of cognitive psychologists, we leave traces in our environment that we or others can pick up on to stimulate next actions. In the context of writing this post, the traces I left consist of the text I've already written and the notes I've jotted down. In the context of a team, it might be a Scrum Board with the description of user stories or a hefty specification document.

A powerful example of the above can be witnessed in termite colonies (Camazine, 2003). Although individual termites have very limited cognitive abilities, the behavior of the termite colony as a whole appears to be quite intelligent. Together, termites construct complicated nests, with arched corridors and a variety of specialized rooms. Termites achieve this by leaving small traces of pheromones for other termites. In turn, this stimulates these termites to perform next actions. This is a form of swarm intelligence; although individual agents have limited abilities, the simple act of following a simple set of local rules allows them to solve very complicated problems. Without strong top-down (or rational) control.

Stigmergy can also be witnessed in human organizations. Good examples are wikipedia and open-source projects (Heylighen, 2007). The individual agents (people) perform small tasks and leave traces (commits, ideas, bug reports) that are picked up by other agents. Together, they are capable of building free encyclopedia's, kernel's or frameworks. All examples of very complex projects. This, again, is achieved without a (strong) top-down control structure.

Stigmergic actions

Of course, the quality of the traces left behind by individual agents predicts the quality of the next action (taken by the same or another agent) and the quality of the stigmergy as a whole. Within the context of me writing this post; the quality of my notes determine if I will pick up on them later or ignore them. Maybe I failed to capture a novel idea in a note, which propably means I will fail to see the point of the note in the future and skip it. 

A trace is considered of quality when it is 'actionable' (Heylighen & Vidal, in press). That is; the trace is specific enough to basically necessitate the next action (a stigmergic action). So, again, in the context of this post, a good trace would be 'Write conclusion by tying together insights with Scrum' instead of 'Finish post'. Although the former specifies which section I have to write, the latter specifies what I have to write about. A stigmergic action is so self-explanatory that another agent can pick it up and execute it, resulting in potential new stigmergic actions.

Cognitive paradigms, applied to Scrum / Agile

Scrum presents a very different paradigm to managing complex projects. Instead of top-down control (through planning and upfront designs), Scrum allows control through self-organization and very frequent feedback control loops. 

When a Scrum Team starts a sprint (e.g. 2 weeks), they start out by setting up a Product Backlog with the Product Owner. The Product Backlog is essentially a prioritized list of functionalities that are assumed to generate business value for the Product Owner. Instead of a full rational analysis of the desired functionalities, Scrum Teams capture the essence of the functionality in terms of simple user stories and overarching epics. User stories are short, preferably only one sentence. Instead of using the Product Backlog as a form of documentation and knowledge transfer, it is used to facilitate communication within the Scrum Team. As such, it can be considered an external memory that contains traces (actionable items) that aid a Scrum Team in their own thinking process but does not attempt to replace it.

Scrum Teams generally estimate the Product Backlog (or at least the top of the backlog) through a process of relative estimation. Instead of full rational analysis, teams sit together to estimate the effort or complexity of every item on the Product Backlog relative to other items on the same backlog. They do this in relative points, not in hours (e.g. a 3 point item will take about 3 times the effort of a 1 point item). This is an intentionally heuristical approach. It allows teams to quickly discuss and estimate functionality and populate a Sprint Backlog. Although the estimates are rough, the sprints allow teams to very frequently evaluate progress and adjust the team process. 

Based on a roughly estimated velocity, Scrum Teams determine how many points of functionality can be implemented in a sprint. This, again, is a very heuristical approach to manage flow. Instead of estimating developer capacity, teams start out with a 'gut estimate' of their velocity. As sprints progress, teams refine their velocity based on the number of completed points in previous sprints. This is a good example of the empirical approach. Instead of spending a lot of 'Brain Time' upfront, teams simply start and use their actual experience to (empirically) adjust the process. This essentially reduces the amount of 'Brain Time' and increases 'Do Time'.

During the sprint, teams communicate progress and synchronize their work mostly through the Scrum Board. The board contains all the items that are in the sprint. Because the items are simple (usually only one sentence), and with only a few details (acceptance criteria and specific examples), they can be considered as actionable traces. In a sense, the Scrum Board represents the external memory of a team during the sprint. It also helps the team think during the sprint. The simple act of pushing items in the sprint through the (Scrum) workflow of 'todo', 'in progress', 'test' and 'done' causes new actions to be initiated ('ok, this is done, so it has to be tested now' or 'this item is too big, let's split it up into tasks'). The Scrum Board is also a locus of stigmergy; as members leave traces of their work (new items, changes in status), other members can pick up on those traces. In efficient Scrum Teams, members can simply look at the board and pick up the tasks that are suiteable for them, instead of someone assigning the items to them. This is a good example of self-organization and stigmergy. Teams distribute their own work and, through the synchronized actions of individuals, complex problems are solved.

During the sprints, Scrum Teams periodically evaluate the progress. They do this at the end of a sprint, during the Sprint Review and the Retrospective, and every day, during the Daily Scrum. These meetings serve as important feedback control loops. Because there is no top-down control and no detailed planning, teams constantly evaluate the progress together. This allows them to (safely) experiment with new functionality and determine if the end goal is still reacheable or if adjustments need to be made.

The biggest difference with traditional approaches, I think, is that Scrum implements a process that allows a team to think by sprinting instead of thinking upfront. This offloads a lot of thinking and memory capacity to the process itself. The sprints, the Product Backlog, and the Scrum Board all act as aids in the situated cognition of the team. This in-process-thinking stimulates the creation of a 'Team Mind' that allows sharing of knowledge, instead of big design documents that attempt (but fail) to achieve the same goal. As such, the stigmergy that is apparent in Scrum's iterative approach is a more efficient means to manage complexity. Instead of trying to comprehend everything upfront, teams continuously re-organize, learn, think and adapt to get the work done and improve their own process. As such, Scrum is a much better fit with our limited cognitive capabilities when it comes to comprehending and managing complexity.

Conclusion

In this post, I applied insights gleaned from cognitive psychology, cybernetics and human factors to explain why Scrum is a superior approach for dealing with complexity compared to traditional approaches. The bottom line is that our cognitive abilities are limited. We cannot fully comprehend the full complexity of a given IT project. However, traditional approaches to software development assume that we can. This is a fallacy. It has resulted and will continue to result in many failed projects. Scrum provides a more useful alternative because it accepts these cognitive limitations. Instead, it promotes the use of heuristics to make quick decisions. But it also implements strong feedback control loops to inspect and adapt the process and correct errors. By following a few simple rules, Scrum allows for self-organization (or stigmergy) to happen in teams. Self-organization is apparent all around us (from termite colonies to wikipedia and open source projects) and shows an alternative, more natural way, for controlling complexity and solving complex problems.

In more than one way, Scrum is a superior approach for dealing with complex software development projects. And I don't think it has to stop at software development, at all.

So, what do you think? Do you agree with the reasoning and the arguments? Maybe you don't agree at all. Let me know! Despite my attempt to write a scientific post, it's still an amalgamation of sources and my interpretation of those sources. So i'm really looking forward to hear your opinion! A fruitful discussion would be awesome,

References

  • Ashcraft, M. H. (2002). Cognition, New Jersey: Prentice Hall;
  • Becker, G.S. (1976). The Economic Approach to Human Behavior. Chicago: University of Chicago Press;
  • Clark, A. (1997). Being there. Cambridge, MA: MIT Press;
  • Davenport, Thomas H. (2005) Thinking for a living. Boston: Harvard Business Press;
  • Gershenson, C. and Heylighen, F. (2004) How can we think the complex? Retrieved from http://cogprints.org/3439 on December 17, 2012;
  • Gigerenzer, G. and Selten, R. (2002). Bounded Rationality. Cambridge: MIT Press;
  • Heylighen, F. and Vidal, C. (2007): Getting Things Done: The Science behind Stress-Free Productivity. Retrieved from http://cogprints.org/6289 on December 17, 2012;
  • Heylighen, F. (2007). Why is Open Access Development so Successful? Stigmergic organization and the economics of information. In B. Lutterbeck, M. Bärwolff & R. A. Gehring (eds.), Open Source Jahrbuch (2007). Retrieved from http://www.opensourcejahrbuch.de/download/jb2007/osjb2007-02-04-en-heylighen.pdf on December 17, 2012;
  • Heylighen F. & Joslyn C. (2001): Cybernetics and Second Order Cybernetics, in: R.A. Meyers (ed.), Encyclopedia of Physical Science & Technology, Vol. 4 (3rd ed.), (Academic Press, New York), p. 155-170;
  • Rogers, R. & Monsell, S. (1995). The costs of a predictable switch between simple cognitive tasks. Journal of Experimental Psychology: General, 124, 207-231;
  • Simon, Herbert (1957). A Behavioral Model of Rational Choice, in Models of Man, Social and Rational: Mathematical Essays on Rational Human Behavior in a Social Setting. New York: Wiley;
  • Camazine S., Deneubourg, J, Franks, N. R., Sneyd, J. (2003). Self-Organization in Biological Systems (2003) New Jersey: Princeton University Press;

Comments (1) -

Illya
Illya said:
Christiaan, great article!
I have to say that I've found your blog after typing "Miller's Law" in google. I was looking for the answer why the idea Scrum team size is 7+-2 and it's about the Miller's Law.
Just about the Complex Problems - I think it's quite obvious. Ken many times mentioned that and that's the definition from the Cynefin framework.
Again, nice article! Thanks, I'll keep reading.

Add comment

biuquote
Loading