Abstract
The aim of this study is to uncover how students’ practices of writing in higher education are being impacted by ChatGPT. The use of ChatGPT and other generative AI needs to be set in the context of a longer-term process of the digitisation of writing, where many tools are being employed by students to support writing because it is a complex iterative process. Generative AI appears to have had a large impact on how students write, and we propose a model of generative AI literacy to assess their capabilities in doing so. Semi-structured interviews and observation data were collected at a British University with 23 students from diverse backgrounds, including the UK, USA, China, Japan, and Saudi Arabia. The data was analysed thematically. It was found that students used ChatGPT alongside many other tools, and in rather individualistic ways often to address specific challenges they felt they had with writing. Their main concerns were around plagiarism, information inaccuracy and technology dependence. There was a relatively weak understanding or interest in the ethical issues around the exploitative and environmental impacts of generative AI. The social controversy around ChatGPT can be seen as a useful opportunity to engage students in a discussion about the digitisation of writing and promote AI literacy in this context.
Similar content being viewed by others
Explore related subjects
Discover the latest articles, news and stories from top researchers in related subjects.Introduction
The use of AI in education (AIEd) has been a discrete area of study for several decades, albeit the majority of studies have been from a technical development standpoint with less involvement of educators (Zawacki-Richter et al., 2019). Development of AIEd has tended to be concentrated on Intelligent Tutoring Systems (Guan et al., 2020). Use of such technologies in an educational context has not been without its critics (e.g. Selwyn, 2019). Meanwhile, AI has already made a relatively unheralded appearance in low-level features of much technology supporting everyday knowledge work such as search, recommendation, transcription and translation. It has also appeared increasingly within writing support tools, such as grammar checkers, as well as in plagiarism detection.
This picture of gradual change was dramatically disrupted in November 2022 by the launch of ChatGPT. Particularly in education, generative AI has created excitement but is also a considerable concern (Kasneci et al., 2023, Trust et al., 2023; Lo, 2023). The usage figures of ChatGPT show an incredibly rapid rise in popularity and the potential benefits claimed for it are wide-ranging. Much of the fear has revolved around its potential impact on academic integrity. What is lacking to date are in-depth studies that explore how ChatGPT is actually used and experienced by students. Since it is in writing text that generative AI excels, and because writing is central to many forms of learning, including assessment, this paper focuses on how generative AI is changing how students write. With the increasing use of AI in many domains of activity there is a growing interest in defining AI literacy (Long and Magerko, 2020). We build on this work to propose a model of generative AI literacy as a framework to assess student use of ChatGPT in their writing.
In this context, the present study had the aim of uncovering how postgraduate students’ practices of writing were impacted by ChatGPT, with the specific research questions for the study being:
-
(1)
How were postgraduate students using ChatGPT and other digital writing tools for writing tasks in the summer of 2023?
-
(2)
What do students consider the benefits and problems of ChatGPT’s use?
-
(3)
What are the strengths and weaknesses in student generative AI literacy?
The digitisation of writing
The impacts of generative AI tools such as ChatGPT in education, need to be understood in the context of long-term digitisation of writing. The digitisation of writing is a major shift in writing as a fundamental process of expression and learning (Strobl et al. 2019). It carries particular significance because of the cultural value and status placed on good writing, particularly in some academic disciplines. Through the introduction of word processors, then spelling, grammar and style checking, then connectivity, and now generative AI tools, this digitisation process is having profound effects on writing, albeit they remain difficult to pinpoint because they are primarily mental rather than directly visible changes (Kruse and Rapp, 2019).
There has been an acceleration of this digitisation process in the last decade with the growing number of AI-powered writing assistants that are appearing. Godwin-Jones (2022) differentiates four types of such tools:
-
Automatic writing evaluation (AWE) which provides feedback on completed work;
-
Automatic written corrective feedback (AWCF) which offers synchronous feedback on spelling, grammar and or style as text is written;
-
Translation tools;
-
Text generation tools which create bodies of text from a short stimulus.
We could add to this other tools and apps used during the research process at the beginning of writing such as for search, text summarisation (e.g., Scholarcy, iris.ai, summarisebot) and literature reviewing (e.g., ResearchRabbit, Gecko, connectedpapers). There are also well-established tools for referencing which link search and reading to producing a final reference list for a completed assignment (e.g., EndNote, RefWorks, Zotero, and Mendeley).
If writing as a process consists of the stages of “prewriting, planning, drafting, revising, and editing” (Strobl et al., 2019, p. 38), then AWE and AWCF are mostly used in the latter two stages. Our previous research revealed, however, that rephrasing tools such as Wordtune are used both in improving text at the revision stage, but also in breaking through mental blocks in the early stages of writing or even planning (Zhao et al., 2023). Similarly, Malik et al. (2023) found Indonesian students also using a wide range of AI-based tools in their writing. Translation tools might also be used at various stages, such as in processing reading and drafting text (Zhao et al., 2024). So just as writing is a complex iterative process, the use of digital writing tools is complex. Our study of Wordtune also found it being commonly being combined with other writing tools (Zhao et al., 2023). While many such tools have multiple functionalities, they tend to be used for specific tasks for which they are best known. For example, many tools will offer some support for translation, but users tend to have a preferred tool for this function.
Thus, it is important to recognise that many learners were already using multiple tools at different stages of the writing process prior to the dramatic debut of ChatGPT, so the use of generative AI appears in a pre-existing landscape of digital writing. Yet the panoply of tools to support the writing process has been little analysed, particularly from the perspective of how they are used in practice and in conjunction with each other, during writing as a complex, iterative process.
Generative AI
ChatGPT’s launch has been a dramatic, potentially paradigm-shifting intervention, influencing how writing as a central aspect of learning is performed, but also the general perception of AI in Education. It has seen an extraordinary explosion of use, with a claimed 100 million users within two months of its launch (Trust et al., 2023). At the same time, it has been deeply controversial, particularly within education, and has been linked to many of the wider debates on the ethics of AI around bias, privacy and impact on society. Whereas the widespread use in the writing of tools such as Grammarly and Google Translate seems to have been tacitly accepted with relatively little controversy, ChatGPT has drawn huge debate to the digitisation of writing (Adeshola and Adepoju, 2023; AlAfnan et al., 2023; Memarian and Doleck, 2023). This may be partly because tools such as Turnitin have at least partly given teachers the lead in the “arms race” with unfair means. From an educational point of view, it is the impossibility of detecting generative AI’s use that makes it so controversial (Uzun, 2023).
Yet it is hard to deny the power and user-friendliness of ChatGPT. While it remains essentially a form of narrow AI (as opposed to a general AI that mimics the breadth of human intelligence), it does perform a wide range of tasks across the writing process, potentially composing a complete essay, but also including the ability to: summarise readings or a topic, produce an outline for a text, draft text, rewrite text in different styles or lengths, and check grammar and spelling (UNESCO, 2023). Thus, ChatGPT has the ability to write entire texts from a prompt or support specific processes in writing. In addition, it can also write computer code, solve math problems, etc. Yet ChatGPT, at least in its early manifestation, poses many informational and ethical problems (EPIC, 2023) (Table 1).
Some of these problems are being addressed in later versions of ChatGPT or in other text generation tools such as Bard or the new Bing. Moreover, they are not inherent to large language models but rather features of systems built by BigTech. Nevertheless, given the great AI capabilities of BigTech in terms of resources including data and so their power to define the definition of AI, we anticipate that users need to be aware of such potential issues.
AI literacy
In the last 5 years, there have been growing suggestions of the need to define AI literacy, because of the increasingly pervasive presence of AI in everyday lives and work. A widely cited definition is offered by Long and Magerko (2020):
We define AI literacy as a set of competencies that enables individuals to critically evaluate AI technologies; communicate and collaborate effectively with AI; and use AI as a tool online, at home, and in the workplace.
The authors break AI literacy down under five headings, with 17 components under those headings:
-
What AI is—this is knowledge such as how to recognise AI when it is encountered and understanding distinctions between general and narrow AI.
-
What it can do—this consists of differentiating the tasks AI is good at doing from those it is not good at, and also being able to imagine future uses, reflecting the evolving nature of AI.
-
How AI works—includes ideas such as representation and has an emphasis on data literacy, emphasising learning from data and the need for critical interpretation of data.
-
How it should be used—under which ethics is placed.
-
How people perceive it.
This is useful in contrasting to more technically oriented definitions such as that of Pinski and Benlian (2023) which lack the ethical and critical dimension. Another useful definition is offered by Ridley and Pawlick-Potts (2021) when they suggest that:
Algorithmic literacy is the skill, expertise, and awareness to, understand and reason about algorithms and their processes; recognise and interpret their use in systems (whether embedded or overt); create and apply algorithmic techniques and tools to problems in a variety of domains; assess the influence and effect of algorithms in social, cultural, economic, and political contexts; position the individual as a co-constituent in algorithmic decision-making.
This is a concise expression of the key aspects, integrating notions of algorithmic literacy, which focuses on the way that AI is often encountered indirectly through functions such as filtering and personalisation on online platforms. Levels of student AI literacy have been much investigated, often using Long and Magerko’s (2020) framework (e.g. Kong et al. 2022). However, most of this work was done before the advent of ChatGPT.
The capability of ChatGPT and other generative AI to create a significant body of content from a short prompt has shifted concepts of what AI is. We suggest that this implies the need to update our notion of AI literacy. In the light of the potential and critiques of the technology (discussed above), we can suggest that generative AI literacy might be defined under five headings:
-
1.
Pragmatic understanding: The individual can use generative AI effectively and interpret the information it produces critically
-
a.
The individual can pick the right tool for the task, in the context of the proliferation of writing tools (including alternative generative AI to ChatGPT)
-
b.
The individual learns to use the chosen tool effectively for a specific task
-
i.
Deciding where in the writing process to use it, e.g. for Search, brainstorming, structuring text etc
-
ii.
Uses the tool effectively through prompt engineering), such as by
-
1.
Being CLEAR (concise, logical, explicit, adaptive and reflective) (Lo, 2023)
-
2.
Providing context for prompts posed
-
3.
Defining what sort of answer is required
-
4.
Rephrasing questions
-
5.
Asking for sources used
-
6.
Iterating and synthesising results
-
1.
-
iii.
Updating their knowledge as tools develop rapidly
-
i.
-
c.
The individual interprets generative AI outputs critically, given an understanding of how they work and their limits
-
i.
Information accuracy, currency, citeability
-
ii.
bias
-
i.
-
a.
-
2.
Safety understanding: The individual can use generative AI safely
-
a.
Is aware of privacy risks
-
a.
-
3.
Reflective understanding: The individual can assess and take action to manage the impacts of AI on their experience in the educational context
-
a.
Impacts on own skills and learning
-
b.
Impacts on social connection, including the social aspects of learning
-
a.
-
4.
Socio-ethical understanding: The individual understands the societal impacts of AI, including
-
a.
IPR issues relating to how models are trained
-
b.
Impact on information culture, misinformation and disinformation
-
c.
Social impacts such as through exploitative process of creation, and the impacts on jobs/ job enrichment
-
d.
Equity of access
-
e.
Environmental impacts
-
f.
Implications of the undue power of BigTech
-
a.
-
5.
Contextual understanding: The individual understands how to use generative AI appropriately in a particular context and make their own use explicit, as appropriate
-
a.
What is appropriate to context
-
b.
How to make use transparent and cite appropriately
-
a.
Generative AI in education
It has been education in particular that has been disrupted by the potential and risks of generative AI in 2023. Yet while much has been written about this in editorials and opinion pieces (Kasneci et al., 2023; Trust et al., 2023) usually to inform educators about how to use it, we are only at the beginning of learning its impact on student behaviour through empirical research. This is important because it seems likely that students have taken up its use far more quickly than teachers.
We do have a few early studies of use by students. A number of surveys by Best Colleges indicate that though US students had concerns about whether it was fair to use ChatGPT for assessments, they were using it and saw it as soon to become the norm (Welding, 2023). Chan and Hu (2023) found Hong Kong students positive about generative AI and willing to use it. This was partly because of its direct uses for brainstorming, individualised assistance with questions, and help with literature reviewing (such as summarisation). But their willingness to use it was partly because they saw it as representing long-term trends in technologies. They had concerns about its accuracy, the transparency of its working, the privacy of their data, the risk of becoming over-reliant, the impacts on employment and conflicts with human values. This accords well with a study by Attewell (2023) based on focus groups with UK students. This again found generative AI being used in a wide range of ways. A similar range of concerns were also expressed such as about the reliability of information from generative AI, privacy, equity of access and fears of becoming over-reliant on it. Students wanted educational institutions to have clearer policies and offer training in the use of generative AI. Interestingly, they also wanted student involvement in generating policy on AI.
If students are generally positive about ChatGPT’s use it is staff who express more concerns. Cardon et al. (2023) conducted a survey of business communication instructors. The main concern of this group of educators was that students would use it to cheat. But they saw a range of negative impacts on learning, such as
-
Less critical thinking/ creativity—itself seen as part of a wider malaise, and the crisis of creativity
-
Less writing skills
-
Less authenticity
-
Less agency because of dependence on such tools
-
Less commitment to authenticity in communication, such as valuing authorial voice and sincerity in communications
They also acknowledged that it can be helpful e.g. in the early stages of writing, and certainly enhanced the efficiency of writing. However they did believe such tools would be used in the workplace, so it was unavoidable that it had to be taught.
Methodology
To answer the research questions, we employed a qualitative methodology within the interpretivist paradigm. We used a combination of semi-structured interviews and observational techniques to gain an understanding of how students selectively employed digital tools in their writing processes and to understand their experiences and concerns regarding the use of generative AI. We recruited participants through an email invitation circulated to students asking for participants who were using “digital tools” for writing. We also asked interviewees to suggest other suitable participants. Our study included 23 participants of diverse nationalities, including students from the UK, China, India, Thailand, Japan, Greece, Malaysia, the USA, and other regions (Table 2). These students were pursuing a range of academic degrees, including postgraduate taught and postgraduate research programmes. All the participants were in the process of undertaking academic tasks, such as writing dissertations or theses. The interviews were conducted in the summer of 2023, this was before the university had issued its policy on AI use.
At the beginning of the interview, participants were asked to demonstrate their writing process for an academic essay/dissertation and explain how they use digital tools to support their writing. The second part of the interview participants were asked a series of questions, including about the tools they used during the writing process, how they had used ChatGPT, and what their concerns were about it, such as data privacy, inclusivity, accessibility, bias, ethics, and the potential impact of generative AI on education. Thematic Analysis served as our chosen method for analysing the qualitative data, enabling us to gain a nuanced understanding of students’ perceptions of digital writing and ChatGPT in particular (Braun and Clarke, 2006). The research received ethical approval from the University of Sheffield. Voluntary, informed consent was gained from participants. All the data were anonymised for the purposes of analysis and reporting.
Findings
Many tools used in the complex task of writing
One theme that emerged strongly from the data was that students were routinely using a wide range of digital tools (many with an AI component) throughout the academic writing process. The most commonly mentioned tools were grammar checkers (especially Grammarly), paraphrasing tools (Quillbot and Wordtune) and translation tools (e.g., DeepL and Google Translate). Somewhat less commonly other types of tools were in use such as for managing references (Endnote, MyBib) and plagiarism detection (Q-text).
Students who were native English speakers used more basic tools such as Word’s grammar checking or very specialist tools such as to manage references. In contrast, non-English speakers were using a wider range of tools, with considerable experience of having done so built up over time.
Such services seem to be used in quite individualistic ways and critically the impression was that their use had been learned from classmates, social media (such as Youtube or Little Red Book), and trial and error—rather than the institution and educators, although one individual was using Grammarly on supervisory advice.
Sometimes students paid a subscription for such tools; sometimes not. ChatGPT was the most frequently paid-for tool among all those mentioned.
ChatGPT: Used in many different ways
Interviewees talked about other tools as much as about ChatGPT and for these interviewees, it was early days with ChatGPT. Many had used it to only a limited extent. Nevertheless, there was evidence of ChatGPT being employed throughout the writing process. There were frequent mentions of uses to:
-
understand difficult concepts including understanding assignment briefs
-
summarise readings during the research process
-
suggest structures for writing
-
get words down on the page and break through a mental block
-
rephrase text and check grammar
Central uses that ChatGPT was uniquely good at was gaining an understanding of an assignment brief and then structuring ideas:
This is the topic and I first use ChatGPT to give me some idea about this topic. And actually, when I saw this topic, I didn’t really understand it. I use ChatGPT to give me some explanations. [9, Thailand]
So I asked it how to organise an essay about this topic. And then it gave me this structure. So I use this as a reference. [2, China]
Some also used it to check that the final text met the brief:
Then in writing or after writing, I would ask if this paragraph was in line with classwork requirements. I would send it all the requirements, and then ask it if I could write like this? [1, China]
Some used it for searching for literature.
I think the most difficult part of writing an essay is the idea you need to make your own opinions and you need to structure your essay but you need to find examples and literature to support your argument. If you just google it or search […] sometimes you can’t find much relevant information or only a few. […] You can just find less relevant information and articles but with ChatGPT, it will collect the most relevant information for you. [1, China].
For other tasks, such as summarising readings, rephrasing ideas into more academic language and grammar checking participants used other tools as much as ChatGPT. Grammarly, Quillbot, Wordtune and translation tools had heavy use in rewriting. Students tended to use ChatGPT for just one or a few of these tasks, not all of them, linked to which aspect of writing they found hardest.
Students’ individual explorations of the tool had produced quite distinct patterns of use, often linked to the areas of weakness they perceived in their own writing. For example, participant 23 emphasised using it to generate analogies to help them understand complex ideas in the context of moving to study computer science from another discipline:
If I don’t understand, repeat, repeat, repeat, repeat, repeat until I understand and then proceed. And that is how my process is. It’s a lot of repetition, a lot of reclarifying myself and always detecting all this reading and fact checking. [23, Malaysia]
Participant 17 described using in complex way to support reading:
I would copy literally like a whole article into it and then say, summarise this in bullet points. I’ll then write it on a piece of paper because that just helps me to like process it in my head as I write the words. Um, and then after that, and then I’ll read the paper like normal [17, UK]
Participant 15 described using it directly in the writing process:
Sometimes if I’ve written something, what I do is I ask for a feedback and I ask or like, you know, if I want a paraphrasing also. So what I do is I prompt it by asking that the check for flow and paraphrase and then it will kind of give it suggestions and sometimes it adds its own things. And then if I don’t want something, I can just not have it. But most of the times what happens is you get a pretty good idea of like, you know, yeah, this is something good and then you can build up from there. So it gives you that initial kind of a boost and then it becomes easier to build your arguments or build your paragraphs. (15, India)
Participant 20 stands out as potentially using it in a way to simply write ideas for him. He talked about “delegating” tasks to it, constructing himself as in control of the process:
So when I break it down, so usually when I have an academic paper, say it’s 10 pages, I have one thought for each page, like one heading for each page in my head. And once I have that set, then I just start off with ChatGPT straight up. [20, India]
Such complex uses, often involved quite interactive exchanges. Participant 23 used the term “bouncing” ideas off it.
I always have sort of like a conversation with it. [23, Malaysia]
Indeed, one interesting dimension of these interactions was the sense that the experience was something quite close to discussing with another human.
Just directly ask a question. Yes. As if I’m asking a human being. [22, China]
But always, always, always remember it is still a tool. It is not a living being. That is that’s why I keep reminding myself it’s not a living being. Always cross check your back. [23, Malaysia]
The quote implies a genuine struggle not to see ChatGPT as human. Equally, the value of ChatGPT was often linked to not needing to ask tutors or peers for help. Of course, it also gives immediate responses unlike them, but it raises the question of how ChatGPT is impacting the social dimension of learning.
ChatGPT was also being commonly used for non-academic writing, such as for job applications. Several interviewees mentioned using it for computer coding. There were also very specific uses, such as for converting text to Latex.
ChatGPT benefits and worries
The way that they described it, ChatGPT and the other digital tools they used, gave participants a sense of being more efficient and productive.
Thinking about it, basically, there is no big change, because there are still steps in writing, mainly those steps, but the efficiency has been improved. [1, China]
I think he saves me a lot of time when I’m reading like reference that’s really long. Helps me summarise it. I think it’s really efficient. [11, China]
This discourse implied that it did not really change what they did or learned, just speeded the process up and saved them time. This may be a rationalisation and disguise the loss of learning that the “efficiencies” might cause. A lot of the discourse around needing to use tools revolved around “laziness” but also a sense of a lack of confidence and anxiety. Using ChatGPT was often justified for reducing stress.
I’ll say I feel less stressful because we get to work more efficiently by using the tools, by using AI tools compared to the traditional way. [11, China]
There was a sense of ChatGPT being the future and there is an inevitability to it overtaking “traditional” learning practices.
Ten years ago, study style, study style, and the current one is very different. [23, Malaysia]
Yet, while it was being used rather extensively our participants were wary of ChatGPT for a number of reasons. Three were mentioned by most interviewees. The first was the unreliability of the information it output and the need to “fact check” it. There was less mention of specific issues such as failing to acknowledge sources or making up references. A second major concern was that its use would be detected by plagiarism detection tools (a concern they held regardless of the fact that it is generally not detectable). More positively participant 19 wanted to preserve her “voice”
I feel like it doesn’t sound like me and I do want to write in my own voice or in my own words as much as I can. [19, India]
Similarly, participant 23 thought that ChatGPT created a text that often read as “auto-generated” [23] so would not use it in many contexts, e.g. writing job applications because it would not effectively differentiate them from other candidates. Implicitly, they did not see this as a problem in the context of academic writing, suggesting that they did not see individuality as valued in that context.
There were certainly some students who did see using ChatGPT as impacting their learning:
But it sometimes it stopped me from thinking. [12, China]
It will reduce the ability to think independently, that is, reduce the motivation to think independently. Some things need to be understood by yourself. [5, China]
Thus a third major concern was a fear that the use of tools in general produced dependence. There were also a few comments that showed awareness of the risk to privacy of putting personal information into ChatGPT.
Nearly all the concerns reflected on personal impact. Disappointingly, the societal impacts of ChatGPT were not often acknowledged. One interviewee expressed concerns about the impact on low-paid workers. But even when prompted the participants showed little awareness or concern about bias, or ethical issues, e.g., sustainability or the exploitative labour relations used to create ChatGPT. Nevertheless, it seemed that the controversy around ChatGPT had filtered through to problematising its use, in ways that were expressed far less in relation to other tools such as Grammarly.
Discussion
The findings confirm our earlier work from before the release of ChatGPT, that a wide range of tools is used during the writing process (Zhao et al., 2023). Perhaps ChatGPT will displace other tools, but at the time the study was conducted it was reaching a community of users well-versed in using digital tools. It was finding a unique place in helping understand assignment questions, aligning answers to the question and structuring ideas, but was also used to perform tasks such as summarisation and proofreading that others performed with pre-existing tools. Our findings suggested that students with higher English proficiency levels tended to use AI tools less frequently.
The study confirms previous research, that students are keen to use generative AI and see it as part of a general trend in technology development (Chan and Hu, 2023). Our respondents tended to use discourses around time-saving, efficiency, and stress avoidance as justification for using the tools.
ChatGPT was used rather intensively and iteratively but in highly individualistic ways. This differential use may reflect that there was little input at this time from the institution to help students how to use and where to draw the line in terms of appropriate use. Students wanted guidance from the institution on ChatGPT, echoing Attewell’s (2023) findings.
ChatGPT has brought advanced functions to digital writing but also intensified a sense of controversy in this area. For students, the worries focussed particularly on the unreliability of information it produced, fear of being accused of plagiarism if they used it and a concern about growing dependence on technology. While often its use was claimed to be justified for its time and stress reduction this may have underestimated the overall impact on learning. Saving time on learning tasks may unintentionally remove significant opportunities to learn. Many of the deeper ethical and societal issues such as around the exploitative way ChatGPT was developed were not fully understood. Yet it was clear that how writing was done digitally had become controversial. This could be seen as a benefit of ChatGPT, in that a gradual infiltration of digital tools into writing was made more visible institutionally and the controversial dimensions of technology use in education brought to the fore.
Using the model developed above we can point to strengths and weaknesses in students’ emergent generative AI literacy (Table 3).
Overall our student interviewees showed significant generative AI literacy in most areas, particularly when considering it was early days with their use of it and almost none of them had received support from their teachers or from their institution in understanding how to use it. The weakest area of development was probably appreciation of the societal impacts of generative AI. As instructors catch up with students in understanding AI hopefully they can help students build up a more systematic understanding of pragmatic use of AI, a more reflective approach and a much more critical awareness of the social implications of AI.
Conclusion
While ChatGPT appears as a threat to longstanding practices in education, especially to some genres of academic writing such as essays, it can also be seen as productively bringing to the fore the controversial nature of AI writing technologies which were already creeping unacknowledged into common use. This context produces an opportunity for educators to actively engage with students in a discussion about how writing can be best supported. Our analysis points to areas of weakness in generative AI literacy that need to be strengthened through this process, such as the understanding of the need to select between apps, to have more sophisticated prompt engineering skills, to think more about bias in results, to be more reflective about its use’s impact on learning and have a much stronger appreciation of the societal impacts of generative AI.
The paper is one of the first to explore student use of generative AI in practice and discover in-depth their perception of its benefits and worries about its drawbacks. It has also developed a framework of generative AI literacy as a way of assessing their use. This can be used to plan both institutional policy and instructor support by identifying gaps in AI literacy that need to be filled. For example, educational developers could use the framework to facilitate discussions with teaching staff, aiding in the development of their AI literacy and enhancing their ability to teach AI literacy to students. In addition, universities could apply the dimensions of the AI framework to formulate policies and provide concrete examples that guide learning and teaching practices. The framework also has the potential in evaluating student AI literacy.
The paper has a number of limitations, pointing to where future research can build on its findings. Most participants in this study were using the free version of ChatGPT (3.5) and only a few used the paid version (4). We did not examine in detail the impact of using different versions. While ChatGPT was the main generative AI tool in use at the time of the study, there were others rapidly emerging in popularity. Future research would need to examine how choices of different apps were made and the impact of these choices on writing. Although the participants of this study come from a variety of countries, it is focused on one institution at a particular time. Given the speed of change in technology and educational policy and practice, it is likely that future research will identify rapid shifts in behaviour. But we emphasise the need to examine student writing practices in the context of significant pre-existing use of digital writing tools. Our focus was on postgraduate students. We think less experienced students, such as undergraduates may be quicker and less discriminating in adopting the technology. As generative AI evolves there will be a need to update our definition of generative AI literacy and also to integrate it with notions such as algorithmic literacy, which point to the way that AI operates in rather hidden ways within the infrastructure. This research employed interviews and observations as its main data collection methods. These offer depth of insight but have less power of generalisability. Future studies could usefully seek to validate our findings through quantitative or mixed-methods approaches, such as surveys or experimental studies. Furthermore, future research could expand the scope of this study from AI literacy to the broader concept of writing digitisation, exploring the issue from other perspectives such as psychology and second language acquisition.
Data availability
The datasets analysed during the current study are not publicly available, but are available from the corresponding author on reasonable request.
References
AlAfnan MA, Dishari S, Jovic M, Lomidze K (2023) Chatgpt as an educational tool: opportunities, challenges, and recommendations for communication, business writing, and composition courses. J Artif Intell Technol 3(2):60–68
Adeshola I, Adepoju AP (2023) The opportunities and challenges of ChatGPT in education. Interact Learn Environ 1–14 https://doi.org/10.1080/10494820.2023.2253858
Attewell S (2023) Student perceptions of generative AI. JISC National Centre for AI. https://beta.jisc.ac.uk/reports/student-perceptions-of-generative-ai
Braun V, Clarke V (2006) Using thematic analysis in psychology. Qual Res Psychol 3(2):77
Cardon P, Fleischmann C, Aritz J, Logemann M, Heidewald J (2023) The challenges and opportunities of AI-assisted writing: developing AI Literacy for the AI Age. Bus Prof Commun Q 23294906231176517
Chan CKY, Hu W (2023) Students’ voices on generative AI: perceptions, benefits, and challenges in higher education. Int J Educ Technol High Educ 20 (43). https://doi.org/10.1186/s41239-023-00411-8
Deshpande A, Murahari V, Rajpurohit T, Kalyan A, Narasimhan K (2023) Toxicity in ChatGPT: analyzing persona-assigned language models. arXiv preprint. arXiv:2304.05335
Electronic Privacy Information Center (2023) Generating Harms: Generative AI’s Impact & Paths Forward. https://epic.org/wp-content/uploads/2023/05/EPIC-Generative-AI-White-Paper-May2023.pdf
Godwin-Jones R (2022) Partnering with AI: intelligent writing assistance and instructed language learning Language. Learn Technol 26:5–24. https://doi.org/10.10125/73474
Guan C, Mou J, Jiang Z (2020) Artificial intelligence innovation in education: a twenty-year data-driven historical analysis. Int J Innov Stud 4(4):134–147
Kasneci E, Seßler K, Küchemann S, Bannert M, Dementieva D, Fischer F, Kasneci G (2023) ChatGPT for good? On opportunities and challenges of large language models for education. Learn Individ Diff 103:102274
Kong SC, Cheung WMY, Zhang G (2022) Evaluating artificial intelligence literacy courses for fostering conceptual learning, literacy and empowerment in university students: refocusing to conceptual building. Comput Hum Behav Rep. 7:100223
Kruse O, Rapp C (2019) Seamless Writing: How the Digitisation of Writing Transforms Thinking, Communication, and Student Learning. In: Looi CK, Wong LH, Glahn C, Cai S (eds) Seamless Learning. Lecture Notes in Educational Technology. Springer, Singapore, pp 191–208. https://doi.org/10.1007/978-981-13-3071-1_10
Lo CK (2023) What is the impact of ChatGPT on education? A rapid review of the literature. Educ Sci 13(4):410
Lo LS (2023) The CLEAR path: a framework for enhancing information literacy through prompt engineering. J Acad Librariansh 49(4):102720
Long D, Magerko B (2020) What is AI literacy? Competencies and design considerations. In: Bernhaupt R, Mueller F, Verweij D, Andres J (eds) Proceedings of the 2020 CHI conference on human factors in computing systems, Association for Computing Machinery, pp. 1–16
Ludvigsen K (2022) The carbon footprint of Chat GPT. https://towardsdatascience.com/the-carbon-footprint-of-chatgpt-66932314627d. Accessed 21 Dec 2022
Malik AR, Pratiwi Y, Andajani K, Numertayasa IW, Suharti S, Darwis A (2023) Exploring artificial intelligence in academic essay: higher education student’s perspective. Int J Educ Res Open 5:100296
Memarian B, Doleck T (2023) ChatGPT in education: Methods, potentials and limitations. Computers in Human Behavior. Artificial Humans: 100022
Motoki F, Pinho Neto V, Rodrigues V (2023) More human than human: measuring ChatGPT political bias. Public Choice. https://doi.org/10.1007/s11127-023-01097-2
Pinski M, Benlian A (2023) AI literacy-towards measuring human competency in artificial intelligence. In: Proceedings of the 56th Hawaii International Conference on System Sciences
Perrigo B (2023) Exclusive: OpenAI used Kenyan workers on less than $2 per hour to make ChatGPT less toxic. Time Magazine. https://time.com/6247678/openai-chatgpt-kenya-workers/ Accessed 18 Jan 2023
Rettberg J (2022) ChatGPT is multilingual but monocultural, and it’s learning your values. https://jilltxt.net/right-nowchatgpt-is-multilingual-but-monocultural-but-its-learning-your-values/ Accessed Jan 18 2023
Ridley M, Pawlick-Potts D (2021) Algorithmic literacy and the role for libraries. Inf Technol Libr 40(2) https://doi.org/10.6017/ital.v40i2.12963
Selwyn N (2019) Should robots replace teachers? AI and the future of education. John Wiley & Sons, Cambridge
Strobl C, Ailhaud E, Benetos K, Devitt A, Kruse O, Proske A, Rapp C (2019) Digital support for academic writing: a review of technologies and pedagogies. Comput Educ 131:33–48. https://doi.org/10.1016/j.compedu.2018.12.005
Trust T, Whalen J, Mouza C (2023) ChatGPT: challenges, opportunities, and implications for teacher education. Contemp Issues Technol Teach Educ 23(1):1–23
UNESCO (2023) ChatGPT and artificial intelligence in Higher Education https://www.iesalc.unesco.org/wp-content/uploads/2023/04/ChatGPT-and-Artificial-Intelligence-in-higher-education-Quick-Start-guide_EN_FINAL.pdf
Uzun L (2023) ChatGPT and academic integrity concerns: detecting artificial intelligence generated content. Lang Educ Technol 3(1):100060
Welding L (2023) Half of college students say using AI is cheating. BestColleges. https://www.bestcolleges.com/research/college-students-ai-tools-survey/. Accessed 27 Dec 2023
Zawacki-Richter O, Marín V, Bond M, Gouverneur F (2019) Systematic review of research on artificial intelligence applications in higher education—where are the educators? Int J Educ Technol High Educ 16(1):39. https://doi.org/10.1186/s41239-019-0171-0
Zhao X, Sbaffi L, Cox AM (2023) The Digitisation of Writing in Higher Education: exploring the Use of Wordtune as an AI Writing Assistant. OSF preprint. https://doi.org/10.31219/osf.io/uzwy7
Zhao X, Xu J, Cox, AM (2024) Incorporating artificial intelligence into student academic writing in higher education: the use of wordtune by Chinese international students. Paper presented at the Hawaii Systems Sciences conference, Waikiki, Hawaii, 2–6 January 2024
Author information
Authors and Affiliations
Contributions
The authors contributed equally to this work.
Corresponding author
Ethics declarations
Competing interests
The authors declare no competing interests.
Ethical approval
Approval was obtained from the ethics committee of the University of Sheffield [054920]. The procedures used in this study adhere to the tenets of the Declaration of Helsinki.
Informed consent
Participants received an information sheet prior to the interview. Informed consent was collected from all participants before the interviews were conducted. Participants were informed about anonymity and the right to withdraw.
Additional information
Publisher’s note Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Rights and permissions
Open Access This article is licensed under a Creative Commons Attribution 4.0 International License, which permits use, sharing, adaptation, distribution and reproduction in any medium or format, as long as you give appropriate credit to the original author(s) and the source, provide a link to the Creative Commons licence, and indicate if changes were made. The images or other third party material in this article are included in the article’s Creative Commons licence, unless indicated otherwise in a credit line to the material. If material is not included in the article’s Creative Commons licence and your intended use is not permitted by statutory regulation or exceeds the permitted use, you will need to obtain permission directly from the copyright holder. To view a copy of this licence, visit http://creativecommons.org/licenses/by/4.0/.
About this article
Cite this article
Zhao, X., Cox, A. & Cai, L. ChatGPT and the digitisation of writing. Humanit Soc Sci Commun 11, 482 (2024). https://doi.org/10.1057/s41599-024-02904-x
Received:
Accepted:
Published:
DOI: https://doi.org/10.1057/s41599-024-02904-x
- Springer Nature Limited