Keywords

1 Introduction

AI assisted learning, e.g., the use of conversational style language models like ChatGPTFootnote 1, has the potential to revolutionize the way programming courses are taught. With artificial intelligence (AI), students can receive tailored, personalized and interactive support, receive immediate feedback, and have access to a virtual tutor 24/7. AI can also analyze student’s performance and provide suggestions for improvement, making the learning experience more efficient and effective. Additionally, AI can automate tedious tasks such as grading code, freeing up instructors to focus on more important tasks such as providing meaningful feedback and engaging in interactive discussions.

The study by Adamson [1] considers the usability of artificial intelligence through different perspectives. They discuss the problems of AI systems because they have black box characteristics and therefore some checks that are consistent with philosophical scrutiny are suggested. Finally, they take an example of recent experimentation with ChatGPT and also generally highlight the challenges and expectations of AI systems.

Use of AI in university education can be seen from two different perspectives: the student’s perspective and the teacher’s. This study focuses on the use of AI from the students’ point of view. The study aims to investigate how much students already use AI to support their studies and for what kind of purposes AI tools have been used. This can help teachers to consider the pedagogical viewpoint and support the selection of pedagogical approaches that enable learning of the right things with AI tools. Thus, the research question was formed:

 

RQ::

How widely have students adopted AI tools in their studies and what are their motives for and against the use of AI?

 

To answer the research question, a survey was conducted among students enrolled in a BSc level programming course to evaluate the use of AI as a tool for supporting their study of the subject.

The rest of the paper is structured as follows: Sect. 2 clarifies the research environment with teaching, including recent AI related studies. Section 3 focuses on the research aims, context, and data collection and analysis methods. Section 4 contains results with analysis and discusses the findings. Finally, Sect. 6 summarizes the research.

2 Background

According to an international survey of over 500 students and teachers, difficulties in learning programming are common, particularly at the basic level. The survey results were analyzed to identify specific challenges and to make recommendations for improving programming education. According to the survey, students consider practical programming exercises that require actively applying participation to be the most useful for learning purposes [2].

Another study [3] reported that self-organized learning is increasingly supported by personal learning environments (PLE), where the learner is in control of their own development and learning. The PLE concept translates the principles of self-organized learning into actual practice. The PLE-driven approach puts the learner at the center and gives them control over the learning experience. In addition, PLEs aim to provide learners with tools and resources that support continuous learning. The PLE approach recognizes that learning is a lifelong process and seeks to enable learners to take control of their own learning by providing them with the necessary tools and resources [4].

Additionally, question and answer sites, such as Stack Overflow, are often used for finding help on programming related tasks. How students use Stack Overflow was studied in detail by [5]. Traditionally, search engines have been used to seek further assistance for programming tasks. However, a new trend is emerging where people are turning to AI-based solutions. These solutions may take the form of virtual assistants or chatbots, such as Siri, Alexa, or Google Assistant [6]. The increasing popularity of AI-assisted learning, utilizing chatbots as a means of providing personalized and interactive educational experiences, can be seen by taking a look at the recent publications in the IEEE Xplore database. For example, using the keywords “chatbot” and “education” on April 12th, 2023, yielded 251 results, 79% (198) of which were published in the last three years.

In the context of this research, the keyword “programming course” was used to identify relevant studies, resulting in a total of 16 publications. Of these, three studies in particular, namely [7, 8], and [9], emphasized the role of instructional guidance during the learning process.

The focus of the study by Verleger [7] was on the creation of an intelligent chatbot interface designed for an introductory computer programming course. This chatbot, named “EduBot”, was initially equipped with a limited knowledge base, as the intention was to expand it gradually based on the interactions and feedback from the students. If the question was new to the chatbot, the instructor would answer and the database was expanded. The chatbot was based on data from the Canvas Learning Management SystemFootnote 2 and Microsoft QnA MakerFootnote 3 [7].

Another study [8] introduced an AI chatbot to help students with academic issues. The chatbot was based on IBM WatsonFootnote 4 and a predefined question bank. According to the study, students found the tool useful for introductory programming [8].

A study by Carreira [9] used a chatbot (Pyo) using RasaFootnote 5 and the Python programming language. The Pyo chatbot has three primary functions: exercise assistance, error guidance, and concept definitions. The chatbot was seen as useful for students in their programming studies because it offers benefits such as immediate response and feedback, and individualized support [9].

3 Research Approach

The research aims, context, data collection, and analysis methods are presented in this section. In this study, the content analysis method was used to evaluate the answers received in the survey.

The aim of this study was to investigate students’ use of artificial intelligence in their programming studies. Tampere University published “Tampere University’s guidelines on the use of AI applications” at the end of January to clarify the rules of usage. The main idea of the guidelines is that the usage of AI applications is allowed, but if students use them, they have to mention it. The usage of AI tools in programming courses was not specifically mentioned.

In the study, the research aim was to clarify the current usage of AI tools when studying Java programming. At first, we wanted to find out how the publicity received by artificial intelligence had affected students’ behavior when studying programming. Before this study, we assumed that students might use two different AIs: the ChatGPT, which was launched in November 2022, and the Github Copilot, which was released in June 2021.

With these initial assumptions, the objective of our study was to investigate the opinions and attitudes of respondents toward AI in learning. In addition to this, we aimed to identify new and emerging challenges as well as the benefits of using AI by analyzing the survey findings. We also wanted to focus on detecting any significant differences in the underlying perceptions between respondent groups with a negative or positive attitude toward AI adoption. By achieving these goals, we hope to gain a deeper understanding of the current perceptions and attitudes toward AI and provide valuable insights that can be used to improve the adoption of this technology in education.

The research context was the programming course offered to Bachelor’s and first year Master’s level students who had already completed at least two previous courses dealing with the basics of programming. The questionnaire was arranged as part of the “Programming 3: Interfaces and Techniques (5 cr)” course at Tampere University. The course started in January and ended in May. The course had about 360 registered students from different starting years (Table 1). The data are from the Tampere University student information database, but it is notable that these data do not include knowledge about gender. Typically, students complete the course in the spring of their second year of study, but depending on the degree program there is a slight variation due to scheduling of the programming courses. Based on previous implementations of the “Programming 3” course, it is estimated that about 60–70% of those who register will complete the course. The course includes several theory and programming exercises for which students collect points. The questionnaire was defined as a theory exercise that earned points, but the quality of the answers was not evaluated. Therefore, the response percentage of 58 is an average response level for theory exercises.

The data for this study was collected through a survey administered to students enrolled in the “Programming 3” course. The survey was conducted halfway through the course. An online survey was organized based on the research questions. The survey was offered to the students using the same Plussa Learning Management System (LMS) as for returning other exercises. Plussa is based on Aalto A+ LMS [10], which was developed by Aalto University.

It should be noted that the survey was not directly related to the students’ programming studies. Two questions were used to conduct the survey, which were designed to elicit detailed responses and provide valuable information for research.

The survey was available in both Finnish and English. The first part (Q1) aimed to determine the extent of artificial intelligence usage, while the second part (Q2) aimed to investigate the manner in which students utilized artificial intelligence.

 

Q1::

Have you used AI (e.g., ChatGPT) in your studies?

Q2::

In what situations and why have you used AI? If you have not, please elaborate on your reasons for not using AI.

 

The first question (Q1) was formulated as a multiple-choice question and the possible choices were never, once or twice, occasionally, weekly, and daily. The second (Q2) was an open-ended question that allowed students to provide a detailed and personalized answer in their own words.

4 Results of the Survey

In this section, we start by answering the research question. The first multiple-choice question asked how extensively students used AI, whereas the second open-ended question aimed to understand their motivation for using or not using AI. The survey results are presented and the findings discussed in relation to teaching programming. At this point in the course, artificial intelligence tools had been briefly explained and discussed, but their use to support learning had not been covered in any way.

Multiple-choice question: Have you used AI (for example ChatGPT) in your studies?

About one third of the students had never used AI tools, such as ChatGPT, in their studies. However, a significant proportion of students had used AI tools once or twice (26%) or occasionally (19%). Only a minority of students reported using AI tools more frequently, with 17% using them on a weekly basis and 3% using them daily.

Open-ended question: In what situations and why have you used AI? If you have not, please elaborate on your reasons for not using AI.

With this question we wanted to resolve whether the respondents had utilized AI in any circumstances and if so, what were the reasons for doing so? If they had not made use of AI, we wanted them to explain the rationale behind that decision.

We received around 200 verbal answers to the open-ended question, with lengths varying from 5 to 200 words. These student answers were collected and stored in an answer database. The following steps were taken to evaluate the answers:

  1. 1.

    Data about the starting year were added to the answer database.

  2. 2.

    The answers were divided into three categories: AI Used, AI Not Used, and AI Tried. The results of steps 1 and 2 are shown in Table 1.

  3. 3.

    First, the answers were evaluated by reading them through and looking for keywords. At this stage, the keywords that appeared most frequently in the answers were selected. Finally, keywords such as “Speed up learning”, “Clarifying things”, “Code syntax clarification”, and “Learning in general” were combined into the category “Learning”. A similar process was used to define the categories “Search engine”, “Coding,” and “Debugging” as shown in Fig. 1 (“Used” answers).

  4. 4.

    Evaluating and categorizing of the “Not Used” answers gave the result shown in Fig. 2.

  5. 5.

    Evaluating the “Tried” answers. There was no clear division of reasons. Based on the answers, it is worth noting that if problems are encountered in the use of AI at the beginning, students may stop using it.

Table 1. Student Enrollment by Starting Year. The “Used” statistics are calculated from the open-ended Q2.
Full size table

According to Table 1, 60% of participants reported using AI during their studies, and the proportion was larger among students who had started their studies in 2022. It should be noted that the students who started in 2022 took the course significantly earlier than recommended. Thus, they may represent a special group.

Fig. 1.
figure 1

Categories of “Used” answers based on reasons to use AI during studies.

Full size image

Figure 1 provides information regarding the use of AI in the context of studying. The values regarding the use of AI were formed through the evaluation of verbal responses, performing searches by using key phrases (such as “to speed up learning”) to enable later categorizing of answers. The term “learning” includes aspects like speeding up learning, clarifying things, and clarifying code syntax. Several students compared the ChatGPT tool to Google and reported the faster speed of information retrieval with ChatGPT. In the “coding” option, the coding of individual parts of the assignments was frequently mentioned, but some students also reported using AI comprehensively to solve practical exercises. The term “debugging” came up frequently when trying to determine the meaning of error messages given by the program.

It is worth noting that one answer could include several of the key phrases, so the bars of Fig. 1 are compared individually to the total answers.

Fig. 2.
figure 2

Categorize of “Not Used” answers based on reasons.

Full size image

Figure 2 provides information regarding why students did not use AI in the context of studying. Unspecified reasons mostly include answers like “I didn’t feel like doing it” or “I didn’t bother to do it.” The reason “desire to learn” mostly described situations where the student’s opinion reflected the excessive ease of AI when doing the exercises and therefore they had a negative attitude toward using AI. The remainder of the students thought that they did not need to use AI, because they could handle the exercises without it.

The third category, “AI Tried,” has a clear division of aspects. The most common answer was similar to “I have tried, but I did not get any benefit from it” or “it did not work as well as I expected”.

5 Discussion

Analyzing the results of the students’ self-initiated use of artificial intelligence during their studies was carried out to answer the research question. Additionally, their motivation for and against AI usage were also taken into consideration during the analysis.

In this study, the results were based on the collected written comments from students, which were categorized according to the previously defined categories. Additional depth can be added to this analysis by featuring a few typical “AI used” comments provided by the students.

Several responses were categorized under “learning”. For instance, the comment “I use it to: - quiz me about exam topics - making the text material more clear and also asking questions about the text - if my code has a bug i ask about it” is a great example of the educational application of AI tools. The student uses the AI tool as an instructor, facilitating understanding in various ways.

One such common response when asked about the utility of ChatGPT as a search tool is, “AI is pretty useful for doing templates and getting the starting point on some questions. On the other and if you have a specific problem AI is faster in solving your problem then searching the internet yourself.” This response highlights the value and robust knowledge that AI tools can provide.

Students also utilize AI tools for coding and debugging purposes. For instance, the response “I used AI when I need more information about some concepts or how to handle data in Java” highlights the capability of AI in explaining unfamiliar programming language techniques. Conversely, the comment “I usually use AI to fix my own program or more likely to debug what is wrong with my program. But I also have tried once or twice using the AI to do a task that I don’t fully understand” falls under both the “Coding” and “Debugging” categories. This comment indicates that some students use AI tools for coding even without complete understanding. Such an approach is not ideal and students should be guided to avoid it.

Furthermore, an analysis of the “tried” responses revealed that these answers were generally longer, and the reasons provided were more diverse. Common phrases such as “It is too easy” or “I want to learn by myself” were frequently mentioned. These responses indicate that students perceive AI tools as being too powerful to use. In the future, providing instruction and guidance on how to use AI tools could help address these concerns.

The comments like “I didn’t find it useful” shows that not all students find AI tools worth their time. Though it is difficult to say whether they didn’t know how to use the tools or they simply didn’t find the tools useful. The statement “It did not work as I supposed” suggests more clearly that students had technical difficulties. Overall, when students encounter difficulties with new technology, it may be easier to set the tools aside and not utilize them. There were many answers with comments similar to either to those presented here, but students’ answers did not indicate one clear reason for the problems they encountered.

According to the results, the majority of students use AI in their studies. The results also indicate that it cannot be concluded that the students are generally against the use of artificial intelligence.

One potential problem with Q1 was discovered during the analysis phase. Students were presented with the question: “Have you used AI (e.g., ChatGPT) in your studies?” The explicit mention of ChatGPT might have directed students to consider only chat-based AI interactions, causing them to base their answers solely on using chat-based AI or even only ChatGPT itself. However, apart from chat-based AI tools, students might have also utilized other AI tools mentioned previously, such as GitHub Copilot, or other AI tools that provide code generation assistance based on students’ input, including comment lines. Additionally, students likely employed code completion tools like IntelliSense. The ubiquity of these types of AI systems in assisting coding tasks is so common that students might not consider them at all when discussing their use of AI tools.

However, from the survey results, it can be inferred that some do not want to use AI in their studies because they consider it too easy a way to complete a course. Surprisingly, some students believe that the use of AI is prohibited. Nevertheless, the university has informed students that it allows the use of AI and has also provided guidelines for its use in studying. This shows a clear need for developing guidelines for the use of AI at the university. These issues pose a challenge for teachers when considering the use of AI in developing study materials and assignments.

The results of this study should be factored in when designing material on AI usage in the programming course. We also aim to repeat this questionnaire when the course is next held. This will also provide the opportunity to compare the changes in opinions toward AI.

Further research could explore the reasons behind the variations in usage frequency and identify potential opportunities for promoting AI tool adoption among students.

6 Summary

The results of this research suggest that the use of AI is becoming more prevalent in university teaching, particularly in the teaching of programming. However, both students and teachers have difficulty understanding the role of AI in education. The research conducted a survey of a wide group of students. A study showed that students widely use AI tools as learning aids, search engines, and coding assistants. The main conclusions drawn from the results is that the goal of teaching programming should be to guide students in the responsible use of AI in their studies. This includes teaching them about AI tools and how to use them effectively, as previously mentioned, for learning, knowledge searching, and coding assistance. Teachers have an important role to play in providing this guidance. Overall, the research highlights the need for greater awareness and understanding of AI in university teaching.