Keywords

1 Introduction

Agile values, principles, and practices can be used to support the sustainability sector. However, supporting this sector also requires learning about its values, principles, and practices. An approach that offers these connections between Agile and the sustainability sector values, principles, and practices is the so-called “Theory of Change”. This approach is frequently used in non-profit organizations, non-governmental organizations (NGOs), and other non-commercial entities which make up a significant part of the sustainability sector. A quick review of the literature on Theory of Change makes it clear that an effective Theory of Change must be responsive within a complex adaptive system because the sustainability sector is dealing with complex issues [5, 27]. The same review indicates that the implementation of Theory of Change often takes a more linear approach, indulging in what the eXtreme Programming community might call “big design up front”.

In this paper, we introduce Theory of Change to agile practitioners and show how Agile and Theory of Change (or rather the sustainability sector) can relate and mutually benefit each other. In the next sections, we provide an overview of Theory of Change, before we connect Theory of Change with Agile. We will provide some examples, followed by a discussion and conclusion section, and we will finalize the paper with the references.

2 Theory of Change

The Aspen Institute Roundtable on Community Change developed Theory of Change as an approach for evaluating community-based change programs for “a free, just, and equitable society” [29].

The benevolent organization Comic Relief in the UK commissioned an early study on Theory of Change and how it is used by non-profit organizations. That study shows how Theory of Change can be used to evaluate change as well as to plan for it and guide it. Based on this understanding, the Center for Theory of Change gives this definition: “Theory of Change is essentially a comprehensive description and illustration of how and why a desired change is expected to happen in a particular context.” [5].

“Outlining a Theory of Change involves at its most basic making explicit a set of assumptions in relation to a given change process.” [27]. Thus, a common question asked in the non-profit sector is “What is your theory of change?” The question, in fact, asks for the underlying assumptions of a change program. These assumptions are both a way to communicate program goals to funders and can also serve to promote internal learning on program strategy, according to Valters [27]. Since the aid and development sector is operating in a complex space, it needs to be understood that a Theory of Change will need to change over time as more is learned during a change initiative.

The core of a Theory of Change is often “if-then” statements with high- and low-order goals, that are followed by a roadmap that typically includes vision, mission, and the program structure [10]. The first part of the statement, introduced by “if” declares the main assumption versus the second part, introduced by “then” which describes the assumed outcome. This is accompanied by metrics for evaluating each outcome.

From an agile perspective, this resonates with Behavioral Driven Development, which is following the “given - when - then” Syntax. In that case, “given” introduces the actual situation (which might be an assumption), “when” declares the actions that will lead to the assumed outcomes that are defined by “then” [24]. In this sense, you can associate Theory of Change with a (behavioral) test-driven approach.

Theory of Change typically uses a forward-thinking approach to change. As Rogers points out, “[t]he future perspective is used to create an optimistic framework to tackle difficult and complex problems […]” [23]. Instead of looking at past data to create immediate changes (as it is practiced in Agile in retrospectives, [7]), Theory of Change builds a model from the future. It takes what planners imagine will have happened in the future and builds backward on that. While Agile retrospectives aim to improve the future by learning from history, there is also an Agile approach known as the Futurespective, where you imagine the future and look for insights and actions that will secure that future vision [15].

3 Connecting Theory of Change and Agile

Valters lists four core principles for developing or refining Theory of Change. These simple rules provide focus and deal with the complexity of change: Focus on the process, Prioritize learning, Be locally-led, and Think compass, not map [27].

3.1 Focus on the Process

This principle acknowledges that as important as it is to set off with a Theory of Change, it is more important to continuously uncover assumptions because many assumptions will not be detected at the start of the change [27]. This is clearly meant to validate the process and therefore aligns with the first and last value statement of the Agile Manifesto, “Individuals and interactions over processes and tools.” and “Responding to change over following a plan” [2].

To find the mutual benefits, we first look at the connections we can make. Focus on the process is related to agile planning and even more so to Agile Chartering [17]. In both cases, it is acknowledged that the act of the process (the planning and chartering) is more important than the outcome (the plan or the charter). To be effective in a complex system an Agile Charter and Theory of Change must be continuously revisited, and are always “good enough for now” [17, 28]. There are many ways to create an Agile Charter. Liftoff is a framework for chartering that identifies 3 focus areas of an Agile Charter (Purpose, Alignment, and Context). Each focus area also has three components [17]:

  • Chartering for purpose means creating a definition of the work to be done that is both inspiring and testable.

  • Chartering for context means creating a shared awareness of the larger complex adaptive system in which a team or project group operates. One place Agile Chartering with Liftoff shines is when it helps the team visualize the dynamically changing relationships and interplay of stakeholders and the team.

  • Chartering for Alignment focuses on a group of people (a team or a project group) establishing or deepening their alliance so they can work towards the purpose within the greater context.

3.2 Prioritize Learning

It is important to understand what can be learned from every planned activity. Theory of Change is grounded on assumptions, which means learning has to be at the core to understand if these assumptions have to change based on what is really happening. The following questions prioritize learning [27]:

  • Learning for what? This first question focuses on the purpose of the learning and thus makes this purpose of the change transparent.

  • Learning for whom? This second question explains who will mainly benefit from the learning.

  • What kind of learning? The last question ensures that there will be a critical reflection on the assumptions. Thus, this question asks explicitly for double-loop and not single-loop learning, to understand the reasoning for what happened [3].

Similarly, Agile features continuous learning. The last principle of the Agile Manifesto is stated, “At regular intervals, the team reflects on how to become more effective, then tunes and adjusts its behavior accordingly.” It is understood as asking agile teams to learn continuously. This learning is practiced in Agile retrospectives. A team “adjusting its behavior” refers to single-loop learning, yet, as Eckstein stated: “[…] if the team uses the retrospectives additionally for examining the support and effect of (organizational) norms on its own behavior also double-loop learning is happening.” [9] Moreover, in a retrospective, the team decides on actions, assuming that these actions will change the situation for the better. Using a Theory-of-Change-lens would mean validating these assumptions in the next retrospective.

The three questions (learning for what?, whom?, and what kind?) also relate to the concept of user stories, where the focus is on what problem needs to be solved, who will benefit from it, and what is the purpose/reason for the story. Or as Jeff Patton expressed it: “Good story conversations are about who and why not just what” [21]. Looking at stories from a Theory of Change perspective shows that stories are in fact also assumptions of the fulfillment of customers’ needs. The assumptions can only be confirmed by observing the outcomes of delivering the story.

3.3 Be Locally-Led

Particularly in the aid and development sector, it is important not to impose change on people. People from developed countries should never pretend to know more than someone from a developing country about their own needs and necessary changes. Thus, supporting people in developing countries always means that the change is locally led by the people in those countries [27].

There are several examples of this principle of local leadership in Agile. In general, self-organization is a core agile concept that ensures the people who are involved are also the ones who act. The Agile Manifesto points this out in various ways [2]. For example, the first value statement, “Individuals and interactions over processes and tools” emphasizes that it is valuable for the people doing the work to decide how they work. Or the eleventh principle, “The best architectures, requirements, and designs emerge from self-organizing teams.” clarifies the value of self-organization in achieving great results. Or as Woody Zuill says, “This is an important concept for us: The team doing the work can best determine how to do that work.” [31].

Moreover, Open Space Technology (OST) is both a facilitation technique and an approach for organizational development, problem-solving, and change that relies on self-organization as a driver for dealing with complexity and complex issues in an open complex adaptive system [20]. OST is applied by agile practitioners for agile transitions, company-wide agility, and for addressing many other complex challenges [8, 18, 19]. OST approaches complexity by inviting everyone with a stake in a big question or initiative to gather and offer questions and topics building a flexible agenda for a working meeting. Then, everyone gathered can choose how to organize to tackle all of the most important questions and topics [20]. This contrasts with an approach where leaders and/or managers decide what and how something needs to be done by the staff.

Finally, Extreme Programming suggests having a customer on-site as a team member, to avoid imposing features on the customer or working with invalidated assumptions [4]. Agile teams aim to work closely with the customer for ensuring they are not forcing solutions on the customer. This is also the reason why the fourth principle of the Agile Manifesto reminds everyone: “Business people and developers must work together daily throughout the project” [2].

3.4 Think Compass, Not Map

This is the understanding that although a roadmap would help, in a complex environment it is more important to have a compass for navigation because any kind of plan will change during the journey [27].

In general, the values and principles of the Agile Manifesto also serve more like a compass than a map. Not only does the Agile Manifesto explicitly value “Responding to change over following a plan,” the second principle also says “Welcome changing requirements, even late in development. Agile processes harness change for the customer’s competitive advantage” [2]. This acknowledges that change will happen frequently and will require (continuous) re-planning. Agile is known for addressing large and complex problems in an iterative manner. The third principle of the Agile Manifesto states, “Deliver working software frequently, from a couple of weeks to a couple of months, with a preference to the shorter timescale” [2]. With Agile, frequent delivery enables learning and re-calibrates the focus on what is requested next.

For addressing complexity, Agile practitioners apply various models like Cynefin, the work of Ralph Stacey, or Human Systems Dynamics [13, 16, 25]. Stacey’s recent work emphasizes the importance of reflexive inquiry, which is reflecting on how we are thinking. Adaptive Action, as part of Human Systems Dynamics (HSD), is an approach to planning that addresses any complex issue [1, 11]. The approach guides in repeatedly asking the following three questions:

  • What? This question is asked to understand the current situation. In this sense, it gathers data about what is happening and what problems need to be addressed.

  • So what? The second question is asked to learn what the data just gathered really means. This is about generating insights.

  • Now what? The final question is to decide what could be done as the next step to influence change in the system. The next action is decided by answering this question.

Having answered all three questions and having taken the action that was decided, the next round of asking and answering the three questions begins. Since every action taken within a Complex Adaptive System has the potential to shift the situation in unpredictable–but observable–ways, new data must be gathered from that actual state of affairs. This is the idea underlying the Agile practice of frequent deliveries. Developers can quickly correct the course of development with frequent deliveries in uncertain changing situations. The Adaptive Action cycle is also built into Agile retrospectives. A retrospective opens by setting the stage, before gathering data (that is asking “What?”), followed by generating insights (that is asking “So what?”), and finally deciding what to do (that is asking “Now what?”) before closing the retrospective till the next iteration [7]. In this sense, every retrospective is an Adaptive Action cycle.

3.5 Examples

Much of the literature on creating Theory of Change focuses on bottom-up, step-by-step approaches. This can obscure the dynamic nature of change in Complex Adaptive Systems, and lead people to treat Theory of Change as a map, not a compass [12]. There is no single way to create a Theory of Change, or rather there are multiple ways to create one. Organizations often begin to work with Theory of Change using a triangle template that was developed by funders for evaluating funded programs (see Fig. 1) [6].

Fig. 1.
figure 1

CES Planning Triangle© presentation of a Theory of Change (for a supported housing project) [6]

Full size image

“The triangle represents hierarchy” [12] and can also imply judgment that can be detrimental to achieving the hoped-for change. Cara Turner, CEO of Project codeX, emphasized in a private conversation that “when the triangle model is used as a basis for evaluation, the current state forms the ‘base’ of the triangle and, it implies that a steady linear progression upwards is needed to reach the single most desirable and seemingly superior outcome. […] This could be especially detrimental when equity and social equality are at stake as part of the Theory of Change.”

Another way of developing and representing a Theory of Change uses a “Logic Model” [30], Yet, by themselves, both the triangle and the Logic Model fail to account for new learning about underlying assumptions. They “[a]re not good at showing the dynamic features of a project. They create the impression that inputs and activities happen first, followed neatly by outcomes; in most projects, different aspects occur at different points in time” [12]. Instead of a bottom-up view, Turner extended the CES Planning Triangle by drawing a continuous loop cycle to refine and represent the codeX Theory of Change (see Fig. 2) [22]. Implementing this continuous loop takes into account that, “[m]ost logic models show ‘one pass’ through the intervention, but many interventions depend on activating a ‘virtuous circle’ where an initial success creates the conditions for further success. This means that evaluation needs to get early evidence of these small changes, and track changes throughout implementation.” [23].

Fig. 2.
figure 2

Presenting the Project codeX Theory of change with a continuous loop cycle [22]

Full size image

4 Discussion and Conclusion

The Innovation Network regularly surveys non-profit organizations [14]. In their last report, they discovered that 58% of non-profit organizations have a Theory of Change [26]. Unfortunately, from this report, it is unclear how many organizations only created their Theory of Change once versus how many are using it and revising it throughout implementation. Certainly, a Theory of Change will not enable transformation if it’s only used at the beginning of an initiative to secure funding and launch. Nor will it enable transformation if it is only consulted, without active adaptation, to evaluate the initiative’s success or failure at potential endpoints [27]. Like an Agile Charter, Theory of Change is about (a) developing a joint understanding of an endeavor, and it is about (b) continuing to adapt and respond to change as more is learned. Or as Valters puts it, “[p]erhaps the greatest contribution of Theory of Change will lie in helping carve out a space for genuine critical reflection […]” [27].

Further research is warranted to learn about additional beneficial connections between Agile practice and the creation and continuous refinement of Theories of Change. In this paper, we have explained the Theory of Change, one of the fundamental approaches frequently used for planning, evaluating, and guiding change in the sustainability sector. By examining Valters’ four principles of Theory of Change -Focus on the process, Prioritize learning, Be locally-led, and Think compass, not map -we created a connection to agile values, principles, and practices. For better understanding, we provided some examples of Theory of Change.

As Theory of Change is fundamental to many organizations and groups working for Sustainable Development goals, Agilists need to understand it, if they will be supporting these entities in the sustainability sector. This paper has connected Theory of Change with agile mindsets and methods (values, principles, and practices) so Theory of Change can also serve as a foundation for agilists especially when practicing sustainability by Agile in the sustainability sector.

We hope with this new understanding that agile practitioners will be able to more effectively promote sustainability by Agile. We encourage practitioners to offer support to NGOs and other organizations to continue learning about the connections between Agile and Theory of Change in working for a sustainable future.