Assessing Co-creation in Relation to Context for RRI Operationalisation

Abstract

Pilot projects and experimentations, especially when conducted in restricted contexts, require assessment activities in order to determine not only their success or failure, but also to identify potential for replication, best practices and obstacles to be tackled in the future. In addition to this, monitoring and assessment have been a pressing issue both in the landscapes of co-creation and RRI, the two main fields that SISCODE operates within.

Pilot projects and experimentations, especially when conducted in restricted contexts, require assessment activities in order to determine not only their success or failure, but also to identify potential for replication, best practices and obstacles to be tackled in the future. In addition to this, monitoring and assessment have been a pressing issue both in the landscapes of co-creation and RRI, the two main fields that SISCODE operates within. Especially in the field of RRI this issue can be traced back to a gap between the theoretical concepts underlying RRI and their effective transition into practice [4, 17]. The scope of this chapter is reporting on the SISCODE approach to assess the project pilots within the context of co-creation. In the evaluation process, all aspects of the experimentation were considered in order to effectively derive considerations from theory to practice and vice versa.

1 Monitoring and Assessing Co-creation

Like many other participatory activities, co-creation involves a great variety of different actors and stakeholders following a non-linear process [1, 10, 12]. A process that may not have one final result, but rather a variety of less specific, broader directions and future indications as a main outcome [6]. This feature turns its measurement, comparison and assessment into a highly complex procedure, where a variety of elements needs to be taken into account.

Co-creation has been widely discussed as an approach that provides access to new and to date unused resources to co-create value for both business and society [3, 6, 14].

Nevertheless, a lack of directions has been identified regarding the set up of a specific strategy to embed co-creation; analogously, there are missing indications on how to effectively apply co-creation for business purposes [6, 14]. This lack can be partly traced back to the not well-defined characteristics, techniques and methodologies that shape the specific value of co-creation [6, 17].

The need to situate co-creation in an explicit scheme and frame in order to be able to assess its success later on [17] has been addressed in SISCODE by conducting 10 experimentations which were informed according to a specific definition and framework for co-creation while furthermore addressing the aforementioned common issue: trying to close the gap between theory and practice [18].

Moreover, the entire reasoning on the assessment of co-creation goes beyond its use for proving efficacy and evaluating the overall activity. It has also suggested that the activities of assessment and evaluation contribute in building awareness and knowledge. The importance to integrate them into the activities of co-design and co-production derives from this assumption, since they can eventually lead to an improvement of the created solutions, increase motivation among the participants and lead to perceived additional value [5].

The additional dimension of the creation of long-term value in the shape of organisational capabilities and new strategies [6] that may be triggered by the introduction of co-creation exploring a broader level of impact is to be addressed specifically in this chapter.

2 The Issue in Assessing RRI Initiatives

RRI has been identified as an opportunity to tackle global societal challenges by ‘anticipating and assessing potential implications and societal expectations with regard to research and innovation, with the aim to foster the design of inclusive and sustainable research and innovation’.¹ Even though it has been widely discussed in theory, there is still a lack of its translation into practice, especially regarding evidence of impact in empirical settings [2, 7,8,9, 16]. This lack of application in real settings could be traced back to missing proof of impact and benefits leading to hesitation in adopting the novel approach despite its promising prospect [7, 8, 16].

Especially the MoRRI project² (Monitoring the evolution and benefits of Responsible Research and Innovation in Europe) addressed the issue of monitoring the development and evaluate the benefits starting from the five dimensions of RRI (Gender equality, Public engagement, Science literacy and science education, Open access, Ethics, and Governance as an overarching dimension) to develop, following an extensive research, a set of core indicators as well as a number of key insights on which needs to be considered and addressed when assessing RRI initiatives [11].

Some of these key insights to be considered to succeed in the task of monitoring and assessing RRI are not only crucial for defining synthetic indicators. They are also relevant dimensions at stake in shaping different and all-embracing ways of collecting data and defining benefits and KPI’s including the perspectives of the involved stakeholders. This implies considering indeed the different stakeholders’ point of view as well as the relatively long timespan required to be able to evaluate real change, being aware that the latter requires a certain amount of time reaching beyond the timeframe of most projects [11].

Dealing specifically with the investigation of the potential of co-creation in RRI, one of the main aims of SISCODE is to identify a model of co-creation ecosystems that includes the monitoring, evaluation and prediction of impacts. The inclusion of stakeholders and actors external to the organisation leads to a broader perspective on factors to be considered throughout the process forming an entire ecosystem around the initiative.

3 The Role of the Assessment Framework in the SISCODE Project

The importance of the relations among the single pilots and their role within the general elements addressed in SISCODE came to light relatively early in the experimentation. In particular, it emerged fundamental to include some additional points that were missing in the initially planned task and with the potential to provide precious insights on both the overall impact of the co-creation process (within the organisation, the ecosystem and policy context in which the organisation operates, and the project), and future possibilities.

Furthermore, some other aspects to be taken in consideration emerged.

Firstly, the need and opportunity to use this assessment to evaluate the broader impact of the entire initiative, and not only the single pilots, provided an additional layer of insights, connections and possible overall contributions to the project (Fig. 1).

Fig. 1

From scaling up to scaling out

Secondly, the general lack of evaluation and impact assessment tools in RRI initiatives was detected. It was noted that a number of other projects were tackling the challenge of impact assessment in (RRI) projects developing and testing indicators, processes and tools. The ‘Monitoring the evolution and benefits of Responsible Research and Innovation’ (MoRRI)³ project SUPERMoRRI,⁴ and a network of other projects dealing with monitoring and assessing shed light on the relevance of an issue affecting the entire field of RRI and even beyond.

These findings and the resulting shared need significantly impacted on the development of SISCODE assessment framework, as well as on the consideration to scale it out, going beyond the project-level expanding the initially planned task.

As anticipated, the assessment framework was originally planned to be limited to the pilot experimentation. However, the relevance of the topic of assessment within the RRI and STI community that are experimenting with co-creation led to its extension to a broader level, going beyond the project scale (see Chap. 2). The development of an assessment framework considering the different elements and fields that RRI and co-creation imply, and their combination have been essential in order to be able to assess the full dimension of the project’s goals.

Co-creation and RRI are the two main pillars that shaped the framework and its tools, together with additional, project-specific indicators derived from studies aimed at providing directions and means for monitoring its impact in different fields among co-creation, RRI, social innovation and design [15].

3.1 Parameters for Assessment

In the light of this, in the following the development of the SISCODE assessment framework, its rationale and underlying concepts are detailed.

A first exploration analysed the existing assessment tools and indicators in the RRI field. The research was intended to build a robust foundation to define a rationale of the general framework based on indicators fundamental for RRI.

The investigation led to one of the few specific assessment researches in the field conducted within the MoRRI project that resulted in the definition of a series of indicators to assess RRI initiatives. Because of their rationale and scope, the indicators developed for MoRRI represent one of the two main elements behind the assessment framework of SISCODE. However, operating on a national level, numerous indicators needed to be adapted or downscaled to fit the needs of SISCODE. For example, among the MoRRI indicators, some refer to data from large statistical European datasets, as the Eurobarometer, and a series of complementary studies not applicable to small-scale initiatives. On the topic, deriving from MoRRI, the follow up project SUPER MoRRI⁵ project has taken on the task of developing a framework for monitoring the evolution and impact of RRI on a project level. However, the project is still ongoing and its results were not yet available at the time when the SISCODE assessment framework was developed.

Also, the MoRRI indicators focus exclusively on the field of RRI, while the area of investigation of SISCODE includes other disciplines, like co-creation. The extension to other areas implied to enlarge the set of indicators so as to be able to appropriately measure the observed phenomenon. That said, the rationale behind the definition of a set of indicators started from the existing set of MoRRI indicators, which have been selected, reviewed, and, when possible, adapted and scaled to be combined in a new series of project-specific indicators developed individually for SISCODE.

As anticipated, one of the main implications when investigating the indicators developed in MoRRI has been the necessity to review, select and appropriate by downscaling the assessment from the national scale for which it has been elaborated. Considering that SISCODE is operating on a projectual and institutional level, addressing a considerably different dimension (Fig. 2), some of the aspects could not be addressed in their original shape.

Fig. 2

Scaling down the MoRRI indicators to the organisational level

3.2 SISCODE Specific Indicators

SISCODE indicators for assessment have been defined starting from the overall goal of the project and reaching out to the ambition of the experimentation; each specific objective has then been associated to one or more data sources for the evaluation as well as to corresponding indicator(s).

The development of the assessment framework and the definition of the data to gather through it took into account both theoretical and practical concerns, requiring to consider the gap of measurable data which can be obtained just through collection of primary data. Therefore, the assessment framework has been shaped targeting specific areas of interest.

3.3 Areas of Interest

Three main areas of interest have been identified for SISCODE, namely:

3.3.1 Stakeholder Engagement

This first area of interest addresses all matters related to the engagement of stakeholders on different levels. From the documentation of the types of stakeholders involved in the single project to general, organisational strategies for the identification and involvement of stakeholders. It is aimed to examine the quantity and variety of stakeholders involved as well as organisational practices and organisational change in relation to involvement practices.

3.3.2 Co-creation, Its Tools and Methodologies

Co-creation as a field of investigation relates directly to methodologies and tools used in SISCODE. Co-creation is to be investigated from a variety of different angles taking it into consideration as a practice itself when applied in the specific project with a structured methodology and specific tools. Furthermore, it is to be investigated from a broader perspective taking into account the transformation its application may trigger at a project level, on an organisational level and in the entire ecosystem of operation. It is therefore examined both as a practice focusing on its elements, methodologies and tools, as well as a practice that may evolve according to the context of application and the changes it can trigger in this context.

3.3.3 Dissemination

The dissemination of results is to be considered on one hand in direct relation to RRI, exploring the practices of provision of open access to results, and on the other hand as the capacities and practices of effectively communicating results to single stakeholders, communities and policymakers investigating the potential of future developments and impacts, exploiting a variety of channels and tools.

3.4 Transversal Topics

There are specific topics that can be considered as high-level categories, and therefore relevant for the overall project. Their nature associates them with multiple indicators. They underline the transversal aspects and the interconnection among the areas of interest, indicators and their means of analysis. In consequence, it is important to note that the same data can feed more than one indicator, since it can be re-aggregated according to the relation to the topic observed. Hence, the analysis of transversal topics entails to consider more indicators.

The main topics addressed in the investigation are:

3.4.1 (Influence on) Policymaking

The topic addresses a fundamental dimension of SISCODE. Considering the small scale of the experimentation that has been conducted, it is investigated to what extent and with which areas of policymaking these bottom-up initiatives established a dialogue with and were able to make an influence on.

3.4.2 Ecosystem Transformation

Especially when relating to multi-stakeholder involvement and the introduction of co-creation practices, it was investigated to which extent the entire ecosystem surrounding the pilot is influenced.

3.4.3 Organisational Capacities

The organisational capacities of the single labs were investigated starting from (1) the capabilities and knowledge present in the beginning of the project, (2) the ones that have been acquired throughout, (3) the transformative processes that might have been triggered during the project, and (4) the ones that are ongoing beyond its conclusion.

4 SISCODE Assessment Activities

The assessment activities planned, developed and conducted throughout SISCODE can be divided in three levels of evaluation.

4.1 Prototype Scale

At the first level, the assessment within the process of the single co-creation journeys is to be addressed by assessing the prototypes produced. This activity is considered a part of the assessment activity, with a focus more on the process and tools of the assessment rather than its results. For this assessment, the labs have been provided with a set of tools and instructions for application to monitor, evaluate and improve the single prototypes without requesting documentation or quantitative data. The knowledge and eventual capacities acquired in this process are then assessed as part of the levels 2 and 3.

4.2 Pilot and Experimentation Scale

The second and the third levels of analysis are the ones directly connected to the list of indicators. The second level focuses on the achievements of the single pilots. Considering the diversity in size of organisations, available resources, and field of work, the individual accomplishments have been analysed from a qualitative point of view. Pilots achievements have been assessed in relation to the prototype, organisational learnings and new knowledge acquired and finally, transformations triggered in the ecosystem in which the lab is operating. The third level of investigation is taking a broader view on the entire experimentation evaluating insights, opportunities, pitfalls, best practices, and learnings in relation to the set of indicators of SISCODE.

The research on the ground of the framework and its levels of analysis led to the definition and development of different tools concurring to the assessment (Fig. 3). The tools are: the labs’ journey spreadsheet, the self-assessment questionnaire, and future scenarios envisioning the long-term impacts of the solutions co-created (Fig. 3).

Fig. 3

SISCODE assessment tools in relation to the rationale of the framework

Because of their nature and scope, such tools are to be considered as partly transversal to the previously described dimensions of (i) stakeholder engagement, (ii) co-creation, its tools and methodologies, and (iii) dissemination, and they are meant to gather and evaluate as much data as possible.

The labs’ journey spreadsheet consists of an online shared excel file documenting objectively inputs and outputs and anticipating few outcomes that can be expected to be reached as a result of the concluded activity.

The self-assessment questionnaire focuses on the reflection on the outcomes and mid-term results of the experimentation. They can for instance manifest in new strategies or practices within the organisation, going beyond the single activities and the pilot itself. It aims to trigger also an initial reflection on longer-term impacts that will then be elaborated further in the scenarios. When dealing with complex and unstructured problems, the process itself can lead organisations to re-define, re-learn and unlearn previous knowledge triggered by questioning and reflecting on current practices [13]. In this context, the questionnaire aims to trigger such reflections, and both investigate and nurture organisational learning at the same time [17].

The scenarios are exclusively considering potential impacts on a long-term. Going beyond the time frame of the project, they envision future possibilities (future scenarios), offering an outlook on potential outcomes not yet achieved, but plausible. Such scenarios can have different shapes, from narratives, to moodboards and videos.

Among them, we opted for videos, considering this format an engaging way to expose possibilities, opportunities and new connections. As such, they serve both as a trigger for further reflections and considerations on the concluded experimentation, as well as to disseminate the pilots’ results with a future vision, illustrating what the prototype could become in the future.

All tools described above have been allocated to the series of SISCODE indicators. This allocation is based on the nature and typology of data collected through the single tools. Table 1 shows the specific indicators identified for each area of interest, its means of analysis, and the typology of data collected (qualitative or quantitative data).

Table 1 SISCODE indicators and allocated tools of assessment

The results of the assessment are considered on different levels:

• 1. On an internal, project-specific level to assess the pure functionality of the prototype itself;

• 2. On an organisational level to measure eventual changes and transformations that the experimentation might have triggered as well as reflecting on potential future impact; and lastly,

• 3. On investigating the regulatory and policy context to capture transformations and trace them back to actions, activities and strategies put in place throughout the project.

5 Results of the SISCODE Assessment Activities

The next sections present the results of the assessment activities, structuring them according to the three levels introduced above: prototype, organisational, and relation to the ecosystem.

5.1 Evaluation Results of the Prototypes

The assessment of the prototypes as a co-creation practice has been considered as a source of data in the indicators (see Table 1), and as such they provided valuable insights for their potentialities in terms of improvement, scaling and replication. Thus, at the prototype level, the focus of the assessment has been on the previous and developing capacities related to prototyping and its assessment, and its impact within the labs.

The most important ones are detailed in the following.

5.1.1 Validation of the Concept

The main scope of the prototyping activities planned from the beginning was the validation of the concept developed in a context where multiple stakeholders and policymakers participated in validating those solutions. The data collected showed that the co-creation activity nurtured in-depth of needs and encouraged further discussions with all the stakeholders involved in the development of the solution. Also, the importance of including policymakers as part of this validation has been stressed: their broader-scale perspective on the complex ecosystem in which the solution is situated often provided a realistic and holistic view on the concept and on what its insertion in a real-life context may lead to.

5.1.2 Testing of Specific Aspects of the Prototype

Especially when considering the testing of complex prototypes and concepts, the testing of their essential aspects in a separate way has been proven effective. Key elements could be tested and verified without having to simulate the entire concept. For instance, in the case of FabLab Barcelona that developed an entire system to fight food waste, it has been crucial to be able to test the single aspects separately since the set-up and testing of the entire ecosystem would have been on one hand not feasible and on the other hand being planned as a set of elements to be composed to build a system it provided precious insights on the single elements and how they could be implemented apart from one another as well.

5.1.3 Reflections on Future Developments

Particularly relevant has been the opening of a constant dialogue with the participants on the current state of the prototype that eventually transformed into a broader reflection both on future developments of the prototype, and the activities of the lab within its ecosystem. Engaging different stakeholders and actors from within the organisation in the process provided support in the development of a sustainability strategy inclusive and conscious of external voices, opinions and considerations.

One unexpected element is then the theoretical reflection on the background of the pilot, which has been traced back to its origins in theory. Thanks to the close collaboration with researchers who participated in the discussion of the prototype and its underlying concepts, the reflection extended to a different, theoretical level, bringing another valuable point of view in a receptive moment of the development. This condition led to bridge the gap between theory and practice, creating a fertile space of constructive discussion.

5.1.4 Experimentation of New Tools

For the monitoring, data collection and assessment of the prototypes, new methods and tools as semi-structured interviews, observation techniques and user tests have been introduced, adapted and applied. This not only produced results for the assessment itself, but also triggered new fields of application for co-creation, going beyond context analysis, ideation and prototyping by fully integrating it into the repetition of prototyping loops. Analogously, the multidisciplinary tools introduced for gathering of qualitative data encouraged an objective documentation of inputs and outputs and a reflection on their functionality and application.

5.1.5 Considerations on Scaling/Replication

Being held open to collect spontaneous and personal feedback from the participants, some users and stakeholders did not only evaluate the prototype but directly made considerations on possibilities to scale the concept or replicate it in different contexts. Taking another point of view and enriching considerations already elaborated in the labs with external voices emerged as an additional opportunity to identify hidden potential of the prototypes and reflect collectively with stakeholders and actors.

5.1.6 Novel Relations and Amplification of Network

Involving a wide variety of users and stakeholders actively in the aforementioned procedures, new connections and contacts in the ecosystem opened further possibilities for future collaborations or further development of the prototype. This awareness resulted clearly from the results of the assessment, since the labs reported on the collaborations and exchanges activated with the stakeholders involved in the co-creation and testing of their solutions.

5.1.7 Capacity Building for Feedback Collection

By providing and suggesting specific tools together with instructions for their application, a learning-by-doing process has been unleashed leading to new capacities built in relation to planning, adaptation and application of tools for assessment.

In conclusion it can be said that the assessment of the prototypes as an activity did not only contribute to the improvement of the concepts themselves, but opened up a variety of benefits and reflections beyond the sheer assessment of the developed concepts. The analysis of the data gathered showed that benefits range from the building and distribution of new capacities to the strengthening of connections with existing and novel stakeholders and eventually shedding light on undiscovered future opportunities.

5.2 Elaborating Results from the Labs

The data collected showed that the co-creation process brought several results at various levels. Figure 4 presents an overview of the main transformations that each lab experienced during SISCODE, the insights at its base have been extracted from the spreadsheet, the self-assessment questionnaire and the case studies. The different main achievements are grouped, and they are associated with those labs who experienced them, the achievements extracted are all related to the main themes of investigation as policymaking, stakeholder engagement and co-creation.

Fig. 4

Overview and synthetic representation of achievements and transformations

Table 2 provides a detailed account of all the achievements and changes experienced and reported as directly related to the prototype dimension, in terms of knowledge transfer and organisational change, as well as transformation at an ecosystem level.

Table 2 Achievements of the pilots

5.3 Results of the Overall Pilot Experimentation

The pilot experimentation as a whole has been assessed mainly in qualitative terms according to the indicators.

The results are first reported as a synthetic map of the insights (Fig. 5) obtained to be then described, grouped, and displayed within the previously defined main dimensions and topics acknowledging and pointing out their interconnection and interdependencies.

Fig. 5

Graphical representation of overall results and insights

5.3.1 Stakeholder Engagement

A crucial point in the engagement of stakeholders for the experimentation has been the transition from a less structured approach towards the embedding of strategies for stakeholder engagement into the organisation. This included also setting up initial strategies for individual initiatives that are assessed and adapted throughout the process depending on its development and unforeseen changes.

Furthermore, the variety and individuality of stakeholders have been pointed out several times relating to the need of employing different approaches of involvement within the same initiative, as well as of conducting encounters favouring exchanges and fruitful debate. Especially in relation to the perception among stakeholders, this benefitted from including a facilitator and mediator able to break schemes and allow encounters of individuals instead of established groups, involuntarily entering the discussion of biases.

It has been found necessary to keep stakeholders involved throughout the entire process to obtain the best possible solution requiring a consistency in motivation that can be achieved by transparency, creating and fostering shared values, setting common goals, and lining out balanced benefits and efforts while aligning expectation from an early stage. Not only motivation and theoretical availability, but also active involvement and efforts have to be managed shedding light on the crucial point of being transparent on efforts expected and potential benefits obtained to manage expectations and avoid misunderstandings and discrepancies.

However, it has been identified that this level and consistency in engagement can only be planned to some extent previously, but partly needs to be co-created, aligning availability and requests, and planning specific commitments without imposing involvement or contributions. In this regard, a potential supporting factor in both engagement and active involvement of stakeholders is the collaboration with similar initiatives, as well as the connection to local and regional agendas to team up to pursue common goals.

5.3.2 Policymaking

The connection to local challenges and their stakeholders is closely related to the influence and impact on policymaking. A key finding from the experimentation is the necessity to align towards common goals and activities with local policy agendas aimed at similar achievements. This can be done by tackling specific challenges addressed by local or regional agendas and/or by specifically choosing policymakers involved according to their orientation. This strategy favours the creation not only of shared objectives but of values and ideas.

Another way to increase impact on policymaking, especially in cases of smaller organisations and initiatives has been identified in exchanging practices, contacts with other organisations and initiatives. Cooperation and collaboration emerged as fundamental to increase potential impact by multiplying resources and maximise the advantages drawn from events and gatherings.

5.3.3 Dissemination

The topic of dissemination is interconnected and complementary to the one of policymaking. The definition of strategies should not only aim at disseminating results in general, but it should point at developing tailored approaches to disseminate findings and results to the different target groups identifying and exploiting their associated channels. Then, apart from the integration of practices to provide open access to results, a variety of broader reflections on the use and results of dissemination activities have emerged.

Dissemination can turn into a means of keeping stakeholders, and specifically policymakers, up to date and aligned. Dissemination itself can be strategically designed and applied as a different way of involving them, defined as ‘active dissemination’ by one of the pilots.

Also, the dissemination across a variety of channels has been identified in the possibility to share not only the process and results of an initiative, but also considerations on replicability together with instructions and material that enables others to replicate and experiment the developed solution in other contexts. This has been recognised as a different way of sharing knowledge and spreading the heritage of co-creation initiatives. It can furthermore serve as a bridging element for encouraging dialogue and exchange with similar realities. The dissemination of the co-creation practices and tools, showing their application and impact can stimulate adoption and adaptation of co-creation practices by other realities. Moreover, keeping an open and fruitful exchange encourages reflection on best practices, also favouring reflection on context dependency in the light of cultural, institutional and thematic backgrounds.

5.3.4 Organisational Capacities

The learning and exchange on co-creation practices refer mainly to the dimension of organisational capacities and their development and the deriving transformation of an organisation. In SISCODE it has been investigated especially in relation to co-creation and stakeholder engagement throughout the co-creation journey. The findings gathered are mainly associated with acknowledging that real change takes an amount of time that goes beyond the time frame of a project. However, there is first evidence on the embedding of novel organisational capacities and resulting ongoing transformations.

The major insights from this observation are that concrete projects constitute the ideal space where to initiate a learning-by-doing process that gives a tangible shape and results within the boundaries of an abstract concept as co-creation. In doing so, it facilitates the understanding and uptake of practices. This often leads to a conflict with existing practices requiring a transitioning process including a shift of mindset in order to be integrated. This has been found to be facilitated when the new practices to be adopted are also shared, discussed, familiarised with, and to a certain extent appropriated in internal meetings creating a safe space for capacity building, experimentation and discussion. Furthermore, it may lead to a more structured application of already present methodologies including them in planning and strategic activities. A series of capacities related to digital and remote working have been built due to the Covid-19 pandemic SISCODE techniques and tools were revised and adapted for being used online, and became facilitators and triggers to support the learning process and the acquisition of those capacities. The conduction of workshops online as well as the application of the revised tools and methods for online use have led to further minor adaptations intended for better meeting the needs of different user groups.

5.3.5 Co-creation

One of the main insights related to co-creation during the experimentation is its interconnection with all the other dimensions, especially the one of stakeholder engagement. This has to be considered within the frame of the context where it is applied, and the individuals involved. Hence, not only in terms of the overall concept, but as a very individual factor, that can entirely change its application depending on the context and the people involved.

The flexibility of co-creation has not only been pointed out as a positive aspect, but also as an attention point to be taken into consideration in terms of having to deal with the uncertainties of an open-end process within the organisation. In this regard, another point is related to how to manage expectations of stakeholders giving concreteness to an open and transforming process. The co-design-tools applied in SISCODE, deriving mainly from the fields of design and social innovation [12], have been found essential to contribute to this concreteness as well as to build better human interactions both while setting co-creation activities, and during their unfolding. This aspect has been pointed out in relation to a set of necessary soft skills, such as empathy, that appears fundamental in relation to the effective application of co-creation, and that can entirely change the outcomes. This aspect highlights the importance of the human factor, and the necessity to build specific capacities for co-creation beyond the application of tools and methodologies. This learning process has been fostered significantly in cases where co-creation was experimented also internally in the organisation leading to capacity building (like stated in the previous paragraph).

5.4 Discussion of the Evaluation Results Against SISCODE Theoretical Base and Findings to be Further Investigated

The findings detailed in this chapter consider the overall assessment conducted within SISCODE, reflecting on bottom-up experimentations that apply co-creation practices in RRI. A series of key insights obtained during the evaluation in relation to co-creation in RRI for policymaking are detailed in the following.

5.4.1 Extended Role of Stakeholders

Stakeholders and actors appear to be experiencing a shift of their role not only by taking an active part in co-creation activities, but starting to be involved even before the beginning of the initiative, as a part of the entire set-up. However, their active involvement as well as their contributions and benefits need to be planned and assessed apart, in order to increase consistency and arrangement. This means eventually requiring a preliminary involvement for aligning expectations and commitments. These aspects have often not been considered from the beginning, but they emerged during the co-creation process underlining the importance of the role that stakeholders play beyond their direct contribution to the ideation and development of the prototype.

5.4.2 Variability and Fluctuation of Stakeholders’ Roles in Bottom-Up Initiatives

Due to the nature of the co-creation activities as being entirely open-ended, the roles and therefore levels of engagement and involvement of stakeholders may change throughout the process. This demands for a regular check and evaluation of the initial mapping of stakeholders and their roles within the process. As part of the self-assessment, labs have been asked to upload their current stakeholder map in the beginning and the end of their journey. The request served the twofold function of providing valuable material for drawing some conclusions in terms of evolution of the stakeholder engagement through time, and also served to lab themselves as a means of reflection. Mapping the stakeholders in two different moments of the process allowed them to observe changes, transformations, and even shifts of roles. The two figures below present the two stakeholder maps from FabLab Barcelona as one example, clearly illustrating this development (Figs. 6 and 7).

Fig. 6

Stakeholder map of FabLab Barcelona in the beginning of the co-creation journey

Fig. 7

Stakeholder map of FabLab Barcelona after the conclusion of the co-creation journey

5.4.3 Integration of Novel Organisational Practices in Relation and Through Co-creation

The building of new capacities and capabilities in relation to co-creation within the field of RRI through the experimentation conducted in SISCODE was part of the initial goals and dimensions of investigation.

Figure 8 shows the assessment of relevant practices present in the organisation, traced throughout the experimentation and after its end, and displayed in form of averages. The graph represents only the sheer presence of the practice in the organisation, not taking its level or frequency of application in consideration. That said, it is noticeable the presence of a difference particularly in terms of capacities related to co-creation: a slight overall increase and alignment of capabilities has been observed, particularly in the field of co-creation closing the initially identified gap between theory and practice. This can be traced back both to the learning-by-doing effect triggered from the application of co-creation, as well as the peer-to-peer learning activities carried out during the project.

Fig. 8

Percentage of labs with relevant practices present in the organisation

By looking at Fig. 8 it is possible to observe that it occurred an acquisition of new practices and capacities where an initial lack has been identified. Then, the diminishing of the value identifying the “involvement of general public” may be associated to the discourse on how the labs changed their perspective on stakeholder engagement in consequence of the application of the strategy and tools proposed in the co-creation process.

5.4.4 Transformative Nature of Co-creation

Considering and assessing the capacity to trigger organisational change, some of the pilots have made broader reflections on the transformative capacities of co-creation taking place both within the single organisation as well as within the entire surrounding ecosystem. This not only included the potential of implementing new practices but also revising the ways in which people within the organisation and stakeholders relate to each other. This could be traced back on one hand to the aspect of co-creation to revoke current power relations valuing different kinds of knowledge and capacities. One the other hand, co-creation itself requires stakeholders, actors and users to confront each other and collaborate, opening up novel opportunities for exchange, discussion and learning (peer and beyond), eventually transforming established relationships and shaping new ones.

5.4.5 A Safe Space for Capacity Building

The complexity of capacity building in co-creation has been pointed out several times, especially in relation to the choice, adaptation, and application of its tools and methodologies. These appear to require a certain guidance or knowledge in order to be applied correctly. Moreover, if a learning-by-doing process is combined with other novel practices like novel techniques or environments for stakeholder engagement it bears the risk of being too overwhelming for the acquisition of new capacities. One potential solution for a step-by-step learning process has been identified in the creation of a safe space during internal meetings and activities: an opportunity for experimenting and discussing practices before running into their application, where to explore possibilities and possible issues ahead of time, and without the necessity to deal with the complexity of stakeholders and the development of solutions at the same time.

5.4.6 Tools for Capacity Building versus Capacities needed to apply Tools

In close relation to the previous point lies the risk of not effectively applying tools and methodologies due to the still ongoing process of familiarising with them, or even learning how to use them correctly. A risk that can cause complications in the process and eventually even hinder the building of new capacities. This could potentially trigger a vicious circle that can lead to frustration and slow down the uptake while increasing resistance to the introduction of co-creation. Acknowledging this, previous training for the use of tools has been identified as one possible solution for this building a knowledge base through specific training or application of tools inside the organisation to then expand and embed this knowledge through further application. This initial training has proven to be fundamental during the SISCODE project.

However, it is to be investigated further how this initial risk of failure and frustration can be minimised when introducing co-creation is not introduced into an organisation as part of a project providing this introductory training.

5.4.7 Complexity of Self-Assessment in Relation to Abstract Dimensions

Self-assessment has led, on one hand, to a series of reflections and insights that did not only enrich the evaluation but also did trigger some additional consideration within the pilot experimentations. On the other hand, the complexity of self-assessment has to be acknowledged. Its subjectivity and dependency on a variety of factors has been noticed especially in the self-assessment questionnaire showing inconsistencies in the patterns of self-positioning on the Likert scales. While the self-positioning in the beginning and the end of the experimentation has been relatively high, it experienced a drop in the intermediate evaluation (Fig. 8; see [15]). The hypothesis made by the researchers in relation to this fluctuation is an initial high positioning due to the sheer presence of a practice in an organisation that is then re-considered, resulting in a lower self-positioning after acknowledging the full dimension and complexity of the topic. Once the overall picture and its complexity is then understood and embedded, it leads to a reinvigoration of the investigated practices.

5.4.8 Awareness of Knowledge and Capacities

While the acquisition and transfer of novel knowledge has been mentioned several times being one of the central issues in the self-assessment questionnaire, it shed light on the related issue of awareness of existing knowledge and capacities. The introduction of novel practices did not only question the validity of current ones but also triggered reflections on how established practices are somewhat similar to the new ones, and how they could eventually integrate and complement each other. Especially some specific capacities related to co-creation like the mapping the user journeys or stakeholders are already practiced in different forms and their integration is facilitated by the recognition of those similarities.

6 Directions for Future Investigations

The assessment framework in this report is characterised by the consideration of various aspects of the co-creation process, reaching out to three scales of observations, and enabling reflections that emerge triangulating data from different sources. This nature and scope make the designed assessment framework inherently prone to get scaled out and replicated in other projects. In particular, although rooted in the specific frame of co-creation, the single objectives, areas of interests and the indicators identified can be applied to the general context of RRI, requiring minor review and adaptation.

This is possible because the process of downscaling, reviewing and adjusting was already considered and included in the development of the SISCODE framework assessment. As described in Sect. 3 of this chapter, the indicators used for assessing the 10 experimentations starting from the MoRRI indicators that were developed for monitoring and assessing the impacts of RRI initiatives at a national scale.

The process of translating the MoRRI indicators from the national scale to that of a RRI-related project produced a set of means of verification and measurement already oriented for being replicated outside of SISCODE (Fig. 9). From the very beginning, considerations on an out-scaling of the assessment framework have been indeed made to re-connect the specific framework to the field of RRI.

Fig. 9

Potential development process of a general assessment framework

Given the current state of the art of the framework, further elaborations in a scaling-out direction regard the division in general and project-specific indicators applicable to most RRI projects together with a guide to define, monitor and assess them.

Therefore, the scaling out of the SISCODE framework is currently under development in direct collaboration with other projects, as part of the work done in the SUPERMoRRI project. The identification of this issue from the RRI community has led to the activation of a series of considerations and initiatives that are aiming at the investigation and scaling of assessment frameworks in RRI.