Active Learning and Integrated Assessment: Minerva’s Approach to Teaching Future Skills

Abstract

The Minerva Project works with global higher education partners to build transformational education programs based on Future Skills. Within our learning taxonomy, we identified practical knowledge as distinct Learning Outcomes (LOs). What differentiates Minerva programs from other Future Skills-focused programs is: 1) active learning pedagogy, 2) direct feedback and assessment of Future Skills, and 3) systematic reinforcement throughout students’ learning experience. The goal in our lessons is to practice skills and broad concepts, therefore we use flipped classrooms with active learning techniques. To guide students’ improvement of Future Skills in our programs, we directly assess LOs and provide feedback on specific LO applications in both synchronous (e.g., in-class polls) and asynchronous (e.g., assignments) aspects of each course. Learning Outcomes from the learning taxonomy may be introduced in one course but are assessable in any other course in the curriculum, ensuring that skills are reinforced throughout a student’s learning experience. In this chapter, we outline principles that underlie the development of learning taxonomies and curricular approaches in Minerva programs. We employ the same principles to collaborate with partners and build custom learning taxonomies and curricula; throughout this chapter we illustrate partner adaptations using examples from our collaborations.

1 Introduction

The foundation of our learning approach at both Minerva Project and Minerva University is a learning taxonomy based on practical knowledge, rather than focusing on transmission of content as in many higher education systems. The practical knowledge in our learning taxonomy includes both skills and broad concepts that students need to adapt to a changing world. We use the same principles that underlie the development of our own learning taxonomy to collaboratively develop learning taxonomies with global partners based on their own program goals.

Minerva Project (Minerva) collaborates with a variety of higher education institutions to design and deliver comprehensive interdisciplinary educational programs focused on teaching critical skills for academic and professional success. Our programs integrate curriculum focused on practical knowledge, active learning pedagogy based on learning science, outcomes-based assessment, and systematic reinforcement and scaffolding of Learning Outcomes (LOs) throughout the curriculum. We have worked closely to help design and implement Minerva University (MU) and have collaborated with higher education partners globally to introduce transformational learning to their students.

While other Future Skills projects have also identified core sets of skills that all students should learn, what differentiates our approach are our 1) pedagogical approach, 2) direct assessment of the skills in the learning taxonomy, and 3) systematic reinforcement of these skills and concepts (i.e., LOs) across courses and other aspects of the student learning experience.

In this chapter, we outline the principles that underlie the development of learning taxonomies in Minerva programs and illustrate their adaptation using examples from our collaborations. Throughout this chapter, we include salient examples from our experience with our flagship partner, MU, and our global higher education partnerships. Minerva University is illustrative of our approach as an integrated system. We highlight the varying degrees of programmatic change in partner collaborations to illustrate the different scales of programmatic and curriculum changes that can be made to move towards skills-based teaching and learning.

2 Principles of a Minerva-Designed Learning Taxonomy

Minerva’s approach to practical knowledge is a learning taxonomy composed of four core competencies derived from empirical literature: thinking critically, thinking creatively, communicating effectively, and interacting effectively. Our core competencies were derived from literature research into the qualities of effective innovators and leaders and surveys and interviews with employers to identify qualities they value when hiring new employees (see Kosslyn, 2017). The four core competencies align with the 4 C’s concept commonly described as an effective approach to twenty-first century education: critical thinking, creativity, communication, collaboration (Kivunja, 2014).

Within each of these core competencies, sub-competencies contain sets of fundamental skills and concepts focused on in the general education curriculum that provides a common foundation for students (Case Study 1, Kosslyn & Nelson, 2017). The practical knowledge that our education system is based on should be broadly useful and aid a student to be successful in a changing world. Because information is so readily available with the internet, instead of focusing on information transfer common in traditional education, we focus on application of the skills needed to think clearly about and access the information needed for a specific task (Kosslyn, 2017). This core learning taxonomy is used at MU (see “Case Study 1: Minerva University” below). Some Minerva partners use either a subset of this taxonomy or an expanded version, depending on how many Minerva courses they adopt (see “Case Study 2: Partially Minerva-Designed Program” below). Other Minerva partners do not use this learning taxonomy at all, but instead collaborate with us to develop a custom learning taxonomy using the principles below (see “Case Study 3: Custom Collaboratively Designed Program” below).

Under the umbrella of each competency and sub-competency, we have identified practical knowledge as distinct Learning Outcomes. Each learning outcome is one of two types of practical knowledge: a habit of mind or foundational concept. Habits of mind are cognitive skills that with practice can be applied automatically. Foundational concepts are broad concepts that can be applied in multiple disciplines (Kosslyn, 2017). The LOs identified for a specific program provide focus and clear goals for the rest of the curriculum within the program. Therefore, the LO identification and selection is a key step and undertaking in the program design process. Our principles for developing LOs in core learning taxonomies include that the prospective LO should:

• 1. Be derived from one of the four core competencies (noted above);

• 2. Lead students to master practical skills or knowledge;

• 3. Be broadly applicable (i.e., be applicable to more than one discipline);

• 4. Be justified either by empirical findings, proofs, or well-established best practices; and

• 5. Lead to specific behaviors that can be evaluated systematically and reliably with rubrics (Kosslyn, 2017).

For example, in a Minerva course, a student may be scored on any LO to which they were introduced in any other course within a Minerva-designed curriculum, even if those LOs are from different competencies and sub-competencies and introduced in different disciplines (Table 22.1).

Table 22.1 Learning Outcome (LO) examples from four core competencies and example sub-competencies (Appendix A, Kosslyn & Nelson, 2017)

LOs in Minerva programs and at MU are not static. As we teach with these LOs we often find the need to redefine the scope or focus of a skill. For example, we used to teach 4 different LOs that represent different types of biases: #attentionbias, #perceptionbias, #emotionalbias, #memorybias (Gahl & Chandler, 2017). But in practice we realized that the scope of these LOs was too narrow and caused students to focus on differentiating between the type of bias, when the focus of the skill was to identify that bias was occurring and when the bias required mitigation. These skills are now clarified as #biasidentification and #biasmitigation, focusing student attention on identifying any bias present and then determining whether mitigation is appropriate (see “Case Study 1: Minerva University” for more information). To adapt LOs we collect feedback from instructors at the end of each lesson and course. We also adapt LOs in the program design phase of a partnership, when we are designing a learning taxonomy with a partner.

With partners we use the same principles for development of learning taxonomies as we do for our Minerva learning taxonomy, whether they are adapting our taxonomy or developing their own. For example, a partner business school with a focus on entrepreneurship selected different core competencies oriented towards entrepreneurial skills: Future Business Orientation, Innovation & Thinking Critically, Effective Communication, and Freedom & Self.

3 Pedagogical Approach

Learning skills and broad concepts, as opposed to the disciplinary content in most traditional education systems, depends upon similar learning processes, but supporting these different types of knowledge requires different approaches. For example, to learn a vast amount of disciplinary content, a student would ideally engage with the content in multiple ways to reinforce the content over time (spaced practice, Brown et al., 2014; Cepeda et al., 2006), such as reading, attending a lecture, doing homework, taking a test, and then working on an assignment. Similarly, for a student to learn practical skills they need to engage with the skills in multiple ways with repeated practice applying the skill. But, for a student to improve their applications of the skill, the practice requires receiving actionable, formative, and constructive feedback from the instructor or peers. For example, if we want students to learn how to problem solve, we need to give them opportunities to apply problem solving skills so that they become habits. They cannot necessarily improve their problem solving skills by learning about problem solving in a lecture or reading, though those are assuredly supplemental. If we want students to learn how to identify biases and mitigate them, we need to give them opportunities to practice identifying bias in different mediums and then mitigating bias or clearly articulating how one might mitigate bias, not just learn what biases are in a lecture or reading (though again, these are supplemental). Therefore, our pedagogical approach to teach practical knowledge in Minerva-designed courses is fully active learning, so that students are engaged at least 75 percent of the time while in class and therefore collaborate in their own learning (Fost et al., 2017).

Active learning has been shown to improve learning and retention in many different types of classrooms (Freeman et al., 2014; Kilgo et al., 2015; Prince, 2004). In part, this is because students’ focused attention span in lectures has been shown to be about 15 min (numerous studies cited in Hartley & Davies, 1978; Wankat, 2002), thus, engaging the students actively can improve retention (Prince, 2004). Improved retention from active learning is not limited to content, for example, Styers et al. (2018) demonstrated that students in flipped, active learning biology courses exhibited gains in critical thinking. The largest critical thinking gains in their study were in intermediate and upper-level courses (Styers et al., 2018), likely because of the increase in problem solving and more theoretical content of upper level science courses. Most importantly, for underrepresented students in math, technology, science and engineering, active learning has narrowed achievement gaps (Haak et al., 2011; Theobald et al., 2020).

Our pedagogical approach at Minerva is designed to nurture student’s development of practical knowledge, through the LOs that make up the learning taxonomy. We do not employ lectures in class sessions; instead, students prepare using asynchronous learning materials before class and then in-class time is spent engaging in activities designed to provide deeper processing of the material and spaced practice with the Learning Outcomes. In flipped learning, passive learning such as lectures and reading happens outside of class and in class is reserved for homework, such as problem sets (Jenkins et al., 2017). Minerva’s pedagogical approach is radically flipped, the learning outside of class includes both the reading/lectures and the homework, and in class time is reserved for active learning. We design highly structured, active learning-based lessons that allow students to apply LOs specified for the course by engaging in higher-order cognitive tasks such as problem solving, peer instruction, and analysis (Resnick, 1987; for examples of applications in an upper-level ecology class, see Gahl et al., 2021).

Using an active learning approach requires a modification of the traditional class roles of instructor, student, and classroom. The instructor in a Minerva active learning session is a facilitator and coach, guiding the group of participants through a series of activities structured for engagement. The instructor is rarely engaged in information transfer except where to provide feedback, clarify confusion, or re-focus the class. A key role of the instructor is to provide feedback to students, in real time and asynchronously, on their applications of LOs. This feedback is integral for coaching students to improve applications over time directly on the Future Skills targeted by the learning taxonomy.

In a radically flipped active learning class, the pedagogical structure of the sessions requires participation by the students. Therefore, the students must prepare for in-class sessions by completing pre-class reading and work before the session, which is typically a large departure from lecture-based classrooms, in which students can arrive to a class unprepared. In Minerva courses, we nudge students to fully prepare ahead by including preparatory assessment polls in each session and assessing students on in-class contributions. To ensure creative engagement of students throughout the lesson, course designers assess student engagement with each activity in a lesson plan by asking “what are the other students doing?”. We attempt to create a culture that embraces and celebrates this more engaged vision of a student.

Although one could teach a Minerva-designed active learning session in person, Minerva partners typically use our active learning platform, Forum™, because it was designed for active learning and teaching. Some of the classroom features necessary for active learning include space for students to collaborate, with the ability to move around (i.e., breakouts), some way for students to provide long form answers in real time and receive feedback on those responses, and a way to be looking at the same prompt (e.g., slide, text, image, blackboard, etc.). In Forum, we can do all of these easily and create a similar experience for every student taking that course. The speed at which we can make classroom changes in Forum outpaces the physical classroom. Imagine the logistics involved in telling students in an in-person class to go into small groups with pre-defined group members and instructions. Consider the amount of time it takes for students to find their peers, pick up their things, move around, receive a document that explains what they will do, and settle back into learning. With set-up beforehand, this can be done in Forum with the click of a button; students arrive in a breakout room, with instructions in front of them, with little chance for distraction or loss of focus during the transition. The instructor, meanwhile, can listen in or visit any breakout group, watch their responses being typed in the breakout workbook in real time, move students to other breakouts as needed with drag and drop, monitor who is talking more or less, chat a specific group, respond to raised hands, follow the progress of all breakout groups at a glance, pin specific groups for later debrief, and modify the breakout time as needed based on their observations of the groups’ progress.

4 Assessment in Minerva Programs

Each skill or concept that we include in our learning taxonomy is directly assessed, providing students with focused formative feedback directly on that skill or concept. Feedback is focused because each application of an LO is assessed, rather than one score on that LO for the entire work. Feedback from instructors is also formative in Minerva programs, with comments that clearly identify how the student could improve their application of a specific LO. LO statements and rubrics are clearly articulated and outline what students should know and be able to do as a result of applying a particular concept or skill. Instead of emphasizing high-stakes exams, our assessments are scaffolded, from contributions in classroom discussions to open-ended, inquiry-based projects and assignments. Instructors are trained to assign rubric scores and provide qualitative assessments to provide students targeted, actionable feedback on specific LOs. In addition, we evaluate students’ ability to transfer that skill or concept to other content areas through assessment of those same LOs across courses.

LOs in the Minerva and MU learning taxonomies are each introduced in the relevant interdisciplinary general education course, for example in Minerva’s general education Collegiate Foundations courses. Once introduced, LOs can be assessed in any other course in a student’s program. Because LOs are assessed by faculty of different disciplines in multiple courses across the curriculum, we need to ensure consistency in the feedback that we provide to students on their applications. We approach this in two key ways: 1) high levels of collaboration among faculty and 2) clear rubrics with measurable outcomes. Faculty team teach many courses at MU and at Minerva partner institutions. In our context, team teaching is a scenario of multiple faculty teaching the same course to different groups of students, and meeting regularly to discuss lesson plan implementation and corroborate grading. Team teaching in Minerva programs allows for conversations about applications and potential scores to occur both synchronously in team meetings and asynchronously in discussion threads. Faculty provide feedback and advice for each other’s assessments by both means. Rubrics are core aspects of the assessment process in Minerva-designed courses. Rubrics are developed along with the LO and reflect the aspects of LO applications that are being assessed (for example, consider the rubric for the LO #biasidentification in Table 22.2).

Table 22.2 Example rubrics for the Learning Outcome #biasidentification

Rubrics are made visible to and used by both students and faculty. For students, rubrics provide clear instructions for strong applications and can help students self-reflect on their applications. For faculty, they are instrumental tools for assessing student applications, though faculty may assess applications made quickly in a class session (i.e., in class polls and verbal comments) with a bit more leeway than applications made with time to think and plan such as those in assignments.

Forum tools integrate the use of rubrics for assessment. Instructors can provide feedback on poll responses, student in-class verbal contributions, pre-class work, and assignments all within Forum. Rubrics are available within the Forum grading tools so that instructors have them at their fingertips while grading. In their Outcome Index in Forum, students can view each LO they have been introduced to, the LO rubrics and examples. Students can also see their average scores for each LO, how their scores for that LO have changed over time, and in which courses, sessions, and assignments the LO has been assessed (Fig. 22.1).

Fig. 22.1

Example student view of a single Learning Outcome (LO) in the Outcome Index in Forum, Minerva Project’s proprietary learning platform. A student can track all assessments of a single LO across courses in the Outcome Index, can view where the LO was introduced, and see an average across all scores

5 Systematic Reinforcement of Learning Outcomes

A key aspect of the Learning Outcomes in Minerva programs is their scaffolding throughout the curriculum, leading to their reinforcement across contexts and domains, providing students with opportunities to transfer their knowledge (Dunlosky & Rawson, 2015; Hopkins et al., 2015). We track the improvement of student’s mastery of the LOs over time, as well as their facility in applying LOs under different conditions (such as different assignment types versus in class verbal comments and polls) and in different ways (such as applying a skill directly versus critique the application of a skill by someone else). LOs are integrated throughout a student’s experience, including reinforcement in experiential learning and internship opportunities. Minerva University provides the most illustrative example of the systematic reinforcement of LOs throughout the program; other global partners may implement varying degrees of integration of LOs throughout a specific program depending on their own constraints.

LOs are typically introduced in a specific course, and then may be scored in other courses. Though many LOs are revisited in subsequent courses, in these instances, the reintroduction of the LO may focus on LO applications that are more complex or highlight a different aspect of the LO (Fig. 22.2). For example, #organization is introduced in a first-year communications course and practiced in many different kinds of courses including arts and literature courses, philosophy courses, natural sciences and computational sciences and its application varies depending on the domain. This LO is scored in the course it is introduced, but is also scored in other courses where organization of written and verbal material is important and applied by students.

Fig. 22.2

Learning Outcomes (LOs) are scaffolded throughout the Minerva curriculum to promote knowledge transfer. Future Skills oriented LOs are typically introduced in the first-year courses. Students apply these LOs in upper year courses in the context of their chosen disciplines. At MU students apply these skills in fourth year capstone projects

Forum tracks all assessments that a student receives on any specific LO, and the student has visibility of each assessment in the Outcome Index (in Forum). For example, a student could view the LO #organization and see each instance of assessment they received on #organization and the context of the assessment (i.e., course, class session, assignment/poll/verbal response). This creates a cohesive view of the student’s application of each LO across courses and allows students to track their improvement on LOs over time (Fig. 22.1).

One challenge with systematic reinforcement of LOs across courses and disciplines is that LOs are then assessed by varied faculty from different disciplines. Clear rubrics are key for ensuring consistency in scores across disciplines. We also encourage faculty collaboration about grading specific LOs, either through discussion boards and collaborative meetings.

6 Case Studies

6.1 CASE STUDY 1: Full Implementation at Minerva University

Minerva University (MU) is the flagship partner of Minerva Project (Minerva) and best illustrates the full application of Minerva principles. Minerva University embodies the active learning pedagogical approach, the direct assessment of practical knowledge, and the systematic reinforcement throughout the program (Kosslyn & Nelson, 2017).

Future Skills in MU’s learning taxonomy are called Habits of Mind (H) and Foundational Concepts (C); HCs (Kosslyn, 2017). Minerva University uses a learning taxonomy of the full set of 78 HCs within the four core competencies: thinking critically, thinking creatively, communicating effectively, and interacting effectively. While the core learning taxonomy of MU and Minerva are similar, the key difference between HCs and LOs (in Minerva programs) is that LO scores are retained within the course where the LO was scored. In contrast, HC scores are always added to the first year Cornerstone course in which they were introduced, and not to the second-, third-, or fourth-year course in which they were scored (see discussion in Kosslyn & Nelson, 2017).

The pedagogy at MU reflects and implements the principles of active learning in every aspect of a student’s education. MU’s faculty design and teach structured active learning lesson plans in courses on Forum. Faculty use features of Forum in class to ensure that the class activities are engaging and equitable, including all students in the activity (e.g., TalkTime™ tool lets faculty see which students have talked less in the session and could be called on next). The strength of this pedagogy on MU students is best exemplified by two final year traditions of tutorials and Civitas at MU. Tutorials are fourth year courses that are guided by a faculty member, but are taught by students in a seminar format. At Civitas, soon to be graduates lead discussions with external experts on pressing global topics. MU students excel in leading these sessions because of their vast experience and practice with active learning over 4 years at MU.

Systematic reinforcement of Future Skills is best illustrated in MU’s implementation. Students are assessed on HCs throughout their 4 years at MU in any course they take. Students are also assessed on HCs in civic projects (collaborative experiential projects with local partner organizations), internships (students provide a self-reflection of HC applications that is scored for credit), and capstone projects (a final year project for all MU students).

6.2 CASE STUDY 2: Partially Minerva-Designed Program

Minerva Project established a partnership with a university to adapt the Minerva foundational education courses as part of a new entrepreneurship program focused on leadership and social impact. The overall program is a three-year program, with an additional year reserved for a specialization track. Students are exposed to fundamental skill sets covering management, humanities, and technology to later apply in disruptive and innovative projects in the subsequent years. The overarching learning methodology is offered via three tracks, each of which exposes students to sets of cognitive and social skills that foster the ecosystem of innovation and entrepreneurship. Students take online and offline classes and embark on project-based scenarios that focus on prototyping and problem solving throughout the entire program.

Minerva Project’s partnership is focused on the foundational education track, which offers twelve carefully curated Minerva-developed active learning structured courses and the subset of LOs that are introduced in those courses. In this partnership, the learning taxonomy is based on the Minerva/MU learning taxonomy, but customized to include the LOs that are specific to the courses to be taught by the partner. In this case, because the partner is teaching a 12-course foundation sequence, instead of MU’s typical 8-course sequence, the learning taxonomy includes additional LOs not found in the MU taxonomy that are introduced in the additional four courses that are part of this track. Assessment for the LOs are done using Minerva rubrics and Forum tools. Systematic reinforcement of LOs is primarily done within the 12 Minerva-developed courses, and not throughout the rest of the program, unlike the MU implementation.

In light of the existing program at the university that focuses on leadership, innovation, and Socratic dialogue, Minerva courses have been customized for the partner by building activities in the classroom where students engage with concepts and techniques that help create synergy between the different components of the program. This tailoring process results from a deep collaboration between Minerva and university faculty, who meet regularly to discuss the previous lesson plans and how the new lesson plans address the program’s focus, driving the choice of topics our curriculum offers.

The threading of Learning Outcomes throughout the curriculum is strategically designed to allow students to revisit priorly introduced skills and to have the opportunity to master those later. This is done by having one or even more LOs shared by different courses, which offer breadth and depth for their application scope. This repetition mechanism guarantees mastery of the skill set powered by the instructors’ regular assessment and feedback of in-class contributions and assignment work.

Students’ focus on skills in the Minerva-developed courses can then be applied to the other tracks in the program to build a solid foundation that fast-tracks students to innovate as they work on company business projects. For example, a student who has learned how to effectively communicate with an audience while using rhetorical appeals can directly apply these when prototyping a business model. By supporting their foundation years with LOs, our curriculum enhances the interdisciplinary skill set that program students develop by the end of the program.

Through this collaboration, we have developed a partnership that thrives as a model of learning and practicing. Far from a plug-and-play method, the curriculum takes shape through our continuous collaboration between Minerva and university faculty. Further, using a single focused skill-based track in the foundation years can provide a thread that will weave through the entire student experience.

6.3 CASE STUDY 3: Custom Collaboratively Designed Program

A large private university engaged Minerva Project to design and develop an entirely custom new flagship program centered around innovation, entrepreneurship, and design, with the goal that this program would blossom into a hub for educational innovation across the university. This partner set their sights high, with a vision of what was possible with creative ideas and bold changes in how we understand the ways and modes in which students learn. Together we imagined a program for students who are unconventional, driven, and eager for challenging experiences. These students would seek to develop skills from engineering to business, coding to visual arts.

Minerva collaborated with the university to develop four core structural components of the program: the academic model, the learning taxonomy, the student experience, and the depth of the major. We co-designed a three-year undergraduate degree that eliminates the large breaks that delay student learning and growth and increases the ability of students to take advantage of experiential learning activities through employer and civic engagements during a summer term. Underlying the design was understanding that learning takes place in a myriad of ways and across a variety of scales, from small moments of insight in peer discussions during class to hands-on collaborative projects to engagement with relevant problems with real-world partners.

The program’s custom learning taxonomy is focused on Future Skills for innovation and design and was collaboratively designed by Minerva and the university team. These skills are to be taught and assessed repeatedly, in diverse contexts, and on varied scales of complexity in order to enable ample opportunity for practice and to help students recognize opportunities for transfer. To achieve this solid foundation, the program requires all students to take a core set of foundational courses in their first year in which these skills are introduced. These courses are intentionally integrated and their Learning Outcomes persist over time and across program components, including design challenges. Students learn, broadly: to communicate in a compelling manner, to understand themselves, their society, and the similarities and differences among individuals and groups, to think critically and use computational methods, and to solve problems in a structured and creative manner.

The way in which these skills are woven into the fabric of the program is central to its ethos. The skills in the learning taxonomy do not simply make an appearance in a single course and then disappear. Rather, they are introduced in foundational courses, reinforced through experiential learning and design challenges, and revisited and reassessed in higher level courses. For example, the learning outcome Maker Mindset encourages students to have an approach to building and learning that embraces tinkering and learning from failure. It is organized in the taxonomy within the Adaptability and Growth sub-competency of the core competency of Self, Difference, and Power. This LO is introduced in a foundational course on designing for innovation and appears in the early design challenges. Later, it is foregrounded in a prototyping course and explicitly called out in a reflection students write as part of their final year capstone project. Importantly, Maker Mindset is always available as an outcome on which faculty can provide feedback, even if the specific activity did not explicitly call it out in its construction. This allows students to identify how the skills within the learning taxonomy appear in various intensities across contexts, and enables faculty ample opportunities to give formative feedback to encourage growth.

7 Challenges and Further Research

Each of the key aspects of the Minerva approach to learning presents challenges. We have worked with these challenges at MU and with our higher education partners to create some solutions, but there is still ample room for improvement as we continue to iterate on these programs.

Developing the learning taxonomy of practical knowledge is an ongoing process. As we teach, apply, and assess LOs, we continue to revise and hone their scope. Every academic year since MU’s inception we have revised LOs to ensure that they are of a similar scale (i.e., are not too broad or too narrow) and are assessing the skill we are targeting. It can be challenging to then ensure clarity of the new or merged LO throughout the curriculum and for instructors who will assess this skill in practice.

As effective as active learning is for student retainment of skills and knowledge, student preparation and the amount of design work needed to create lesson plans are persistent challenges. Students who are conditioned to attend lectures without having to prepare ahead often struggle to do the pre-class work necessary for an effective active learning session. We have implemented preparatory polls to assess student preparation, score pre-class work deliverables, and underscore the need for preparation in class sessions. And yet, until there is a strong culture of preparation and buy-in for active learning, this change for some students can be challenging. Additionally, to design an effectively structured active learning session takes time and expertise, much more time than instructors typically have available for class preparation. For lesson plan design, we collaborate with partners to either 1) use Minerva-designed active learning courses or 2) collaboratively design courses as a team to spread the work. We have also worked on ways to speed design efforts, using templates and instructional designers, who may be savvy in designing active learning courses but not in a specific content area.

Lastly, although rubrics help guide assessment across disciplines and faculty, some skills are not readily applied and assessed in a typical academic classroom (e.g., #teamwork, #confidence). Project-based assignments and experiential learning along with self-reflection can help to fill this gap, but this remains an area of challenge. Rubrics also help to guide assessment towards consistency, but without collaboration and discussion among instructors, consistency may be difficult to achieve.

8 Learnings and Recommendations

From our work with multiple partners implementing practical knowledge focused programs, our key learnings and recommendations are centered on alignment in both curriculum and students throughout the institution. Alignment among LOs, class activities, assignments, and assessments is key to a successful program. Effectively applying LOs is the overall goal, therefore each aspect of the overall implementation should be built to drive towards that goal. The learning taxonomy provides the blueprint for the rest of the curriculum. To focus a program on Future Skills along with active learning pedagogy and direct assessment of LOs, rather than holistic scores, is an approach that is transformational and requires a cultural and mindset shift in a higher education system. Stakeholder buy-in is important from administration, faculty, and students. The culture can be built, but it takes focused effort. When the culture is achieved, we have found that the students themselves begin to speak up to reinforce that active learning with focus on LOs continues to be implemented in their classrooms.

Having a clear learning taxonomy based on skills can be a powerful tool. Within our own organization we apply LOs in our work, in meetings, and with feedback from peers. We have also integrated active learning into our internal processes, for example, large meetings are almost always engaging, active learning sessions with time for small group collaboration and opportunities for input from all involved. In addition, we intermittently facilitate internal trainings and workshops for non-academics at Minerva Project, to improve understanding of our learning taxonomy and pedagogy. Integrating these skills internally allows us to build an effective and focused team culture and to cultivate an education organization that tries to exemplify its own philosophy in practice.

Future Skills in Practice: Our Recommendations

• For successful whole program implementation, align Learning Outcomes with class activities, assignments, and assessments throughout a program.

• Directly assess the skills that you would like students to learn, providing formative feedback at many stages throughout the program.

• Stakeholder buy-in is important across all levels of a program from administration to teachers to students.