Keywords

1 Introduction

The potential applications of AI in smart manufacturing are numerous, ranging from improving the efficiency of machinery maintenance to detecting defects in the machine or the product to preventing worker injury. AI-based systems can identify bottlenecks, optimize production schedules, and adjust settings to maximize efficiency by analyzing large amounts of data from sensors and other sources in real time.

Furthermore, AI-based software systems can provide context-specific support to machine operators. By monitoring machine performance in real time, these systems can detect potential issues, give the operators recommended actions to solve the problem, and even automate the resolution, if necessary. This support can reduce operator errors, improve machine up-time, and increase productivity.

In general, collaborative processes in smart manufacturing are characterized by alternating phases of reactive and proactive elements, with each actor supporting the other alternately [1]. AI-enabled smart manufacturing systems can be self-sensing, self-adapting, self-organizing, and self-decision [2, 3], enabling them to respond to physical changes in the production environment in a variety of ways. AI-guided interactions in the manufacturing process include stopping machines, adapting production tasks, or suggesting a change in production parameters. However, achieving effective teaming between machine operators and AI-enabled manufacturing systems requires mutual trust based primarily on the self-sensing and self-adaptation of each actor [4].

The increased situational awareness through an improved Human-AI collaboration enables operators to make informed decisions about optimizing machine settings and adjusting production schedules. This collaboration can improve product quality, reduce waste, and increase efficiency [5]. As AI continues to evolve, we can expect to see even more significant advances in smart manufacturing in the years to come.

In the frame of the international research project Teaming.AI,Footnote 1 we develop a software platform to facilitate human-AI teaming in smart manufacturing as shown if Fig. 1. We already presented reference architecture in [6]. However, in this work, we elaborate on different stakeholders’ requirements regarding the quality characteristics of AI software platforms. For this purpose, we conducted 14 structured interviews with various stakeholders of the prospective platform. They rated a set of 11 different quality characteristics and provided vital success factors that can evaluate the fulfillment of these quality characteristics during the development and operation of the platform.

Fig. 1
An infographic of a project has injection, checking and assembly, and packaging stages. An injection technician is interlinked to an injection machine that is interlinked to a teaming A I platform. The injection technician links teaming engineers who link the platform including an assembly operator.

Teaming.AI project overview

Full size image

The results of our study provide valuable insights into the different stakeholders’ expectations and remark on the importance of addressing their specific needs in the platform’s design and development. Considering these quality characteristics and critical success factors, we can ensure effective collaboration between human operators and AI systems.

The remainder of this work is structured as follows: Sect. 2 presents the related work regarding stakeholder interaction in AI-related projects. Section 3 addresses the three use cases we have faced in the context of the Teaming.AI project. Section 4 details stakeholders’ different roles in projects of this type. Section 5 discusses the pains identified when implementing such a solution. Section 6 discusses the expectations toward the technical realization. Section 7 discusses the characteristics of the high-level teaming concept. Finally, we point out the lessons we have learned in this work and some lines of future work.

2 Related Work

The field of human-AI collaboration has gained significant attention in recent years, driven by the emphasis on integrating AI technologies into collaborative work settings in Industry 5.0 [7,8,9,10,11,12]. This growing interest revolves around the exploration of how AI systems can complement human abilities rather than replace them. Numerous studies have delved into different aspects of human-AI collaboration, including the design of intelligent systems [13], the development of new interaction paradigms [14], and the evaluation of the usefulness of these approaches in real-world scenarios [15].

One area of study in human-AI collaboration strives to design AI systems that can work effectively with human counterparts. Researchers have examined various strategies for designing intelligent systems to communicate and collaborate with human users, including natural language processing, machine learning, and cognitive modeling [14]. Additionally, some studies have focused on designing new interaction paradigms that enable seamless collaboration between humans and AI systems. For example, researchers have investigated using augmented and virtual reality to create immersive environments that improve human-AI interaction.

Another focus is evaluating their usage in real-world scenarios. Several studies analyze the impact of AI systems on the performance of human workers, as well as their acceptance and adoption of these systems. These studies have explored different elements that influence the success of human-AI collaboration, such as trust, transparency, and the nature of the tasks being performed [16]. Additionally, some researchers have investigated the ethical implications of human-AI collaboration, such as the potential for bias in decision-making processes.

Within this context, Knowledge graphs (KGs) have also become a powerful tool for making production lines more efficient and flexible in manufacturing and production [17]. They provide a means of organizing and processing vast amounts of data about devices, equipment, machine models, location, usage, and other related data [18]. KGs can also help make manufacturing smarter by providing insights into the complex and competitive landscape [19]. This can enable manufacturers to identify patterns, trends, and correlations that were previously hidden, leading to more informed decision-making and improved operational efficiency. The potential benefits of KGs in manufacturing and production make them an essential technology for the future of industrial operations and highlight the importance of continued research and development in this field [20].

Especially interesting and relevant in this context are recent AI developments like ChatGPT by OpenAIFootnote 2 and Luminous by Aleph Alpha.Footnote 3 Both of them are providing natural language interfaces for their human users, so that they are able to express their problems, information needs etc. through their most natural communication means. This seems beneficial especially in situations of mental pressure or other forms of stress that workers might have to cope with in their daily routines. The TEAMING.AI sister project COALAFootnote 4 performs research on such kinds of voice-enabled digital intelligent assistants. With Luminous-Explore, Aleph Alpha points out the importance of semantic representations,Footnote 5 so that humans are no more forced to represent their thoughts and intentions in machine representations but are enabled to expressing them in a more natural way. With those kind of developments, Aleph Alpha is also focusing on industrial use cases of their technology.Footnote 6

3 Manufacturing Context

The following use cases (UC) describe concrete applications where an AI-based smart manufacturing solution could support a Human and AI collaboration in a manufacturing context. UC1 and UC2 derive from automotive suppliers and cover the process of plastic injection molding. In UC3, we investigate the ergonomic risk assessment during large-part manufacturing. Optimization focuses on the interplay between AI-controlled machine tasks and manual human labor.

3.1 UC1: Quality Inspection

The main objective of UC1 is to support the machine operator during the visual quality inspection of plastic parts produced by injection molding. The software platform shall classify products as OK or not-OK (including the type of defect), with the machine operator double-checking the latter. The software system interacts with the machine operator during the quality inspection and provides context-specific information for fault analysis and adjusting parameters to mitigate product defects. The main focus is on integrating human feedback: The machine operator should have the chance to overrule and correct the suggestions of the AI system, e.g., by manually marking defective regions if they were classified wrong.

This collaborative interaction between the machine operator and the software platform reinforces the notion of human-AI partnership, with each contributing their unique strengths to achieve the best possible outcomes. It empowers the machine operator with the authority to validate and correct AI decisions while ensuring continuous learning and improvement of the AI system by incorporating the operator’s feedback [21].

3.2 UC2: Parameter Optimization

UC2 is also concerned with injection molding. However, the produced plastic parts are bigger, and cycle times are longer. Therefore, the software platform should provide a more proactive way to reduce and prevent non-OK parts effectively (zero waste production). The software platform should predict possible process deviations and identify the likely root failure causes before they materialize in faulty parts. It should be able to explain its findings (e.g., likelihoods), present recommendations (e.g., on parameter changes), and give the machine operator the ability to provide feedback to the software platform.

To achieve this proactive approach, the software platform leverages its analytical capabilities to predict possible process deviations. It analyzes real-time data from various sensors and monitors the production parameters to identify patterns or anomalies that might indicate an impending issue. The software platform can provide the machine operator with early warnings and proactive recommendations by continuously monitoring and analyzing the production process.

3.3 UC3: Ergonomic Risk Assessment

UC3 focuses on high-precision manufacturing of significant components (e.g., gear cases for wind turbines). This manufacturing process is typically time-consuming, physically strenuous, and involves a combination of automated and manual labor.

The objective is to analyze the ergonomic risks of human workers in terms of static loads and repetitive strains, especially during workpiece setup (involving manual part positioning, clamping, unloading, etc.). Using a camera-based tracking system, the software platform can determine the location of the machine operators on the shop floor, analyze their pose for ergonomic suitability and give feedback to the operator, e.g., via alerts. In addition, the software platform should also identify manual tasks associated with milling operations (e.g., taking measurements) and collect information about the tooling used during these tasks. This way, the software platform mediates between the milling machine and the operator by combining this information with context information, such as machine data. Overall, the software system should

  1. 1.

    Improve communication between the operator and the machine

  2. 2.

    Perform a continuous ergonomic risk assessment

  3. 3.

    Allow rescheduling of similar assembly tasks to reduce repetitive strains

By incorporating these functionalities, the software platform empowers both the machine operator and the milling machine to work in harmony, prioritizing the well-being and safety of the operator. It is an intelligent assistant providing real-time insights, guidance, and risk assessments to optimize ergonomics and prevent work-related injuries. This holistic approach promotes a healthier and more productive working environment, ensuring the efficient manufacturing of large parts while prioritizing the workers’ welfare.

4 Stakeholder Roles

In the following, we describe the different stakeholder roles with their exemplary activities identified during requirements engineering.

  • Data Protection Officer (DPO) enforces the laws protecting the company and individuals’ data (e.g., the GDPR) by controlling the processing of data and properly auditing the system.

  • Software Scientist (SS) queries runtime data of the software components of the software platform, such as logging information, for evaluating and optimizing the system’s base code and behavior.

  • Data Scientist (DS) applies statistical methods to the data processed by a software platform.

  • Machine Operator (MO) performs a visual inspection of the produced parts, clamping, adjusting the workpieces, and performing manual tasks on the machine, such as obtaining measurements and making parameter adjustments.

  • Production Line Manager (PLM) monitors and optimizes the processes for producing and assembling the product or its parts on the shop floor.

The involvement of these stakeholders, each with their unique roles and activities, highlights the multi-dimensional nature of the software platform and its impact on various aspects of the manufacturing process. By incorporating these stakeholders’ expertise and responsibilities, the software platform’s development and operation can benefit from a well-rounded perspective, ensuring compliance, optimization, data analysis, production efficiency, and quality assurance.

5 Identified Pains

According to [22], pains are “bad outcomes, risks, and obstacles related to customer jobs.” In a collaborative project like Teaming.AI, the end users’ participation enables the technical side to address real market needs. To this end, two questionnaires were circulated among all use case partners to identify the existing issues and pains related to their processes and understand the potential benefits they would expect from the technologies developed. Each questionnaire is directed toward two categories of employees: (1) Managers, who can present more managerial challenges of the organization, and (2) Operators, who can more effectively depict their day-to-day challenges and are the active users of the machines that will be retrofitted.

Specifically, the questionnaire was circulated among 18 individuals who participated in the study and distributed evenly for each use case through the EUSurvey platform.Footnote 7 The personas analyzed included:

  • Injection Technician

  • Production Shift Coordinator

  • Operator

  • Engineering Director

  • Process Managers

  • R&D Manager

  • Innovation Manager

  • Production Manager

  • Head of Automotive Digital Transformation

  • Data Scientist

The first section of the questionnaire aimed to analyze the profile of each end user. Overall, all end users are more results-driven organizations. When asked about the top 2 priorities in selecting third-party collaboration for Industry 4.0 initiatives, 78% preferred parties with proven pilot cases. The next most selected priority involved the ability to ensure an easier integration of solutions by 45% of the respondents. The third and fourth most selected priorities were results-oriented and involved the capacity to promise short-term value and the market participant’s brand acknowledgment with 34% and 23%, respectively. Less prominent options also included the proximity of the technology provider and the sustainability improvement.

5.1 UC1: Quality Inspection

This particular use case involved the participation of the following roles:

  • Data scientist

  • Head of Mobile and Digital Transformation

  • R&D Manager

  • Operator

Overall, the results indicate that the end user faces pains primarily within the production department, which also affects coordinating activities. Specifically:

  1. 1.

    Setup parametrization: The most significant bottleneck in the particular production system, according to the respondents. As the manufacturing context section demonstrates, adjusting the parameters is necessary to mitigate product defects.

  2. 2.

    Scrap Generation: Causes for scrap generation can emerge from lack of quality raw materials, setup mistakes, machine issues, etc. Although scrap generation is considered financially sustainable, it impacts planning and financing.

  3. 3.

    Unexpected downtime/equipment failures: Mechanical and electrical failures accompanied by non-optimal maintenance are primary factors leading to failures.

  4. 4.

    Loss of time: Issues related to time loss refer to production delays, waste generation, lack of raw materials, non-productive processes, etc.

  5. 5.

    Not optimal production quality: The capacity to be prone to errors is affected by the level of control in the existing production system.

  6. 6.

    Increased inventory: Supply chain disruptions and unexpected failures lead to material unavailability, which leads planners to over-order to ensure that production does not remain stagnant.

  7. 7.

    Lack of Flexibility in tasks and product design: Flexibility is not limited to how operators are liberated to move between activities but also to the ability to switch between orders and in the adaptability to produce different types of products.

5.2 UC2: Parameter Optimization

The second use case of the project involved the following six roles:

  • Engineering Director

  • Injection Technician

  • Operator

  • Process Manager

  • Production shift technician

  • R&D Manager

As mentioned above, the first two use cases encounter similarities between each other contextually and, by extension, similar pains. Specifically:

  1. 1.

    Setup parameterization: The level of operator expertise influences the possibility for a defect to occur.

  2. 2.

    Unexpected downtimes/equipment failures: The main consequence of unexpected downtimes leads to production pauses and redirection of employees to other places.

  3. 3.

    Lack of human-machine interaction: Similarly with the first pain, the level of expertise has a dominant impact on production and handling impending issues.

  4. 4.

    Loss of time: The primary concern is from a managerial perspective. Identified loss of time lies in sales, dispatch, unexpected failures, and scrap generation for the organization.

  5. 5.

    Scrap generation: At a similar level with UC1. However, areas such as cost management and logistics would be the first areas to be improved by reducing scrap in production.

  6. 6.

    Increased inventory: Even though scrap generation is considered sustainable, it directly influences the purchasing and warehouse departments, which need to add to their risk management and planning activities.

  7. 7.

    Increasing Costs: Although all UCs experience increased costs due to other pains, UC2 respondents have highlighted the challenges in their cost management activities.

5.3 UC3: Ergonomic Risk Assessment

The third and final use case focused on the following roles:

  • Injection Technician

  • Innovation Manager

  • Machine Operator

  • Operator

  • Production Manager

  • R&D Manager

Following the questionnaire results, the ergonomic risk assessment use case is characterized by the following pains:

  1. 1.

    Setup parameters: In UC3, operators indicate that setup difficulties make them feel there is a lack of time.

  2. 2.

    Unexpected downtimes/equipment failures: Similarly to other use cases, it leads to high rework costs.

  3. 3.

    Waste generation: Classified as below average, there is an excess of production material waste in the current form of processes.

  4. 4.

    The system does not help meet scheduling demands: Bureaucracy leads to a lack of control over increasing equipment productivity.

  5. 5.

    Delivery delays: Inability to meet scheduling demands on time lead to delivery delays and profit reductions.

  6. 6.

    Increased inventory: Like all the use cases mentioned above, over-ordering leads to an increased inventory and profit reduction.

  7. 7.

    Not optimal planning: The current production system impedes stakeholders from conducting optimal planning activities.

  8. 8.

    Loss of time: A consequence of those mentioned above and other relevant factors reduces the production system’s time and productivity.

5.4 Total Results: Pains

To summarize, although the project involves three different use cases, which may signify different needs from different stakeholders, some common themes offer a common ground to build upon. Particularly,

  1. 1.

    Setup parameters

  2. 2.

    Unexpected downtimes/equipment failures

  3. 3.

    Loss of time

  4. 4.

    Waste/scrap generation

  5. 5.

    Increased inventory

The five pains are the primary market needs that are the groundwork to construct compelling value propositions, which is one of the building blocks of a business model.

6 Expectations Toward the Technical Realization

In previous work [23], we conducted 14 interviews with stakeholders from three industry partners and three specialized SMEs for software development of AI-based systems.

We defined candidate scenarios [24] that describe the context and the anticipated functionality from the stakeholders’ perspectives when interacting with the prospective software platform. In an interview-based case study, we assessed each of the 11 quality characteristics in terms of their importance to the overall platform from the stakeholders’ perspective. We elicited the critical success criteria related to the software platform. The quality characteristics comprised the 11 characteristics of the ISO 25010:2011 standard for software quality (SQuaRE) [25] and 3 AI-specific quality characteristics, such as trustworthiness and explicability.

At the beginning of the interviews, we explained the research context of our study (i.e., human-AI teaming in smart manufacturing) to the interviewees. Each interviewee thoroughly understood the research context since they had participated in the project for over 1 year. For the relevance assessment, we adapted the Quality Attribute Workshop format [26] and asked the interviewees to assign, in total, 100 points to the different quality characteristics according to their subjective relevance for human-AI teaming in smart manufacturing.

The interviewees rated trustworthiness, functional suitability, reliability, and security as the most important quality characteristics. In contrast, portability, compatibility, and maintainability are rated as the least important. Furthermore, the results indicate consensus regarding the relevance of the quality characteristics among interviewees with the same role. However, we also recognized that the relevance of the quality characteristics varies according to the concrete use case for the prospective software platform. In addition, we asked interviewees to discuss critical success factors related to the prospective software platform. According to the interviewees, critical success criteria for human-AI teaming in smart manufacturing are improved production cycle efficiency, fewer faulty parts and scrap, and a shorter period for detecting deviations (product or process quality). This response was unsurprising since similar pains had already been expressed earlier (see Sect. 5).

7 Team Effectiveness

As described in [27], well-designed coordination mechanisms can improve team effectiveness to ensure that relevant information is distributed throughout the team. These coordination mechanisms, which have first been described by Salas, Sims, and Burke [4] as part of their big five framework for team effectiveness, are:

  • Shared Mental Models: Shared mental models facilitate a common understanding of the environment by creating knowledge structures that promote the information exchange about state changes and team member needs. The knowledge structures need to be designed to be comprehensible by humans and AI.

  • Mutual Trust: Trust in the team setting has been defined by Webber [28] as “the shared perception ... that individuals in the team will perform particular actions important to its members and ... will recognize and protect the rights and interests of all the team members engaged in their joint endeavor.” A culture of mutual trust is essential in supporting the core components of teamwork, especially since, as [29] shows, trust critically influences how individuals within a team will interpret others’ behaviors.

  • Closed-Loop Communication: Communication between humans and AI may suffer from similar issues as communication between humans. Communication may be hindered because of misinterpretation of messages due to their perspectives and biases or because team members have become focused on their tasks rather than on how those tasks affect other team members’ tasks.

Although the original big five framework focused purely on teaming between humans, it nonetheless builds a solid foundation for human-AI teaming digitalization. By prioritizing team effectiveness as a goal rather than just performance output, the emphasis remains on human team members instead of AI, acknowledging that the interactions among team members are equally vital.

Effective communication is, therefore, essential for teams to function correctly. In the context of human-AI teams, communication can help to ensure that AI systems are correctly interpreting human input and that humans are correctly interpreting the output of AI systems. This can be particularly important in high-stakes environments where errors can have serious consequences.

8 Conclusions and Future Work

In this work, we have seen how the development of AI-based software platforms that facilitate collaboration between human operators and AI services needs the integration of the different stakeholder perspectives into a common framework. In this regard, it is vital to identify the individual relevance of different quality characteristics per stakeholder and propose key success factors related to human-AI teaming to measure fulfillment. This can help ensure that the software platform is user-friendly and practical, meeting the expectations and needs of all stakeholders involved in the collaboration. Furthermore, it can mitigate conflicts arising from differing stakeholder perspectives during the projects.

Our research has thoroughly analyzed the critical issues, challenges, and opportunities of integrating AI technologies into collaborative work environments. To do that, we have adopted a multi-stakeholder perspective, considering the perspectives of different actors involved in the human-AI collaboration process. We aim to provide insights and recommendations for designing effective human-AI collaboration systems that enhance productivity, innovation, and social welfare.

We have observed that human-AI collaboration in Industry 5.0 requires careful consideration of various factors, such as the design of intelligent systems, the development of new interaction paradigms, the evaluation of the effectiveness of these systems in real-world scenarios, and the ethical implications of human-AI collaboration [24, 30]. Moreover, we have highlighted the importance of adopting a human-centric approach to AI system design, prioritizing human users’ needs, preferences, and capabilities. Other elements (e.g., establishing trust and transparency in human-AI collaboration systems and ensuring fairness, accountability, and transparency in decision-making processes) are also essential in this manufacturing context.

In conclusion, integrating AI technologies into collaborative work environments offers immense potential for enhancing productivity, innovation, and social welfare. However, it also presents numerous challenges that require careful consideration and proactive measures. By adopting a multi-stakeholder perspective, prioritizing human-centric design, fostering interdisciplinary collaborations, and implementing responsible governance, we can pave the way for practical and ethical human-AI collaboration systems that maximize the benefits while minimizing the risks associated with this transformative technology.

As future lines of research, it is necessary to remark that as AI technologies continue to advance, it becomes increasingly essential to handle the issue of AI bias in collaborative work environments. Bias in AI systems can perpetuate existing social imbalances, support discriminatory practices, and limit opportunities for specific groups. Therefore, it is crucial to develop mechanisms that detect and mitigate bias in AI algorithms and data sets used in human-AI collaboration. In addition, integrating AI technologies into collaborative work environments necessitates ongoing training and upskilling programs for people. These programs aim to introduce individuals to AI capabilities, promote digital literacy, and provide them with the necessary skills to collaborate with intelligent systems effectively.