Open AI’s ChatGPT may result in educators having to rethink tests and assessments
Everyone is confronted with the breakthrough of (and the hype around) ChatGPT, a new AI tool trained by OpenAI launched at the end of last year. ChatGPT works like a written dialogue between the AI system and the person asking it questions. Open AI says the dialogue format “makes it possible for ChatGPT to answer questions, admit its mistakes, challenge incorrect premises, and reject inappropriate requests”. ChatGPT also appears to be able to pass very complex medical school exams, say researchers in the USA who have put it to the test. Kane Murdoch, the head of academic misconduct at Macquarie University, says that simpler assessments would be “a piece of cake.” Educators may have to rethink assessments as the technology opens the door to widespread cheating. Some schools in the US have banned the tool’s use and others educational institutions are thinking of going back to “pen and paper” tests and introducing stricter cheating detection measures. Academic integrity experts say these AI tools will have a bigger impact on student assessment than the pandemic. Professor Cath Ellis from UNSW, says a future syllabus that embraced the use of AI tools would probably have more oral assessments and fewer standard take-home assignments, and fewer essays.”What if we change how we assess students, and instead of reading what they have claimed to have written, we spend time talking to them about what they’ve learned?”.
Examples of ChatGPT output
These two example of “conversations” I had with ChatGPT (in order to use the tool you need to first sign up for an Open AI account) are quite telling about the technology’s disruptive potential!
Passing the US medical licensing exam
A recent article in ABC Science reports that findings from a pre-print study carried out in the US reveal that ChatGPT achieved over 50% in one of the most difficult standardised tests around: the US medical licensing exam (USMLE). For most applicants, the tests require more than a year of dedicated preparation time. The researchers fed questions from previous exams to ChatGPT and checked that the answers to those questions weren’t likely to be in the dataset accessible by the AI tool when it had been trained. They weren’t! Kane Murdoch, the head of academic misconduct at Macquarie University, quoted in the ABC Science article, said that simpler assessments would be “a piece of cake.”
If you can’t beat them join them!
Murdoch and others are pushing for universities to embrace ChatGPT, rather than banning it outright. He says that much like the advent of the calculator this is “a game changer” and that telling students that using it is forbidden will not stop usage. The Tertiary Education Quality and Standards Agency (TEQSA), which regulates higher education in Australia, agrees that it should not be banned. Educational institutions will need to develop new strategies for assessment. And, in the mean time, it is likely that ChatGPT will be widely used.
Limitations of ChatGPT
“ChatGPT is incredibly limited, but good enough at some things to create a misleading impression of greatness,” OpenAI chief Sam Altman said on Twitter in December. Mr Altman added that “it’s a mistake to be relying on it for anything important right now.” The Open AI website lists the limitations
- It sometimes writes plausible-sounding but incorrect or nonsensical answers.
- It is sensitive to tweaks to the input phrasing or attempting the same prompt multiple times.
- It is often excessively verbose and overuses certain phrases.
- Ideally, it would ask clarifying questions in case of an ambiguous query, but the current models usually guess what the user intended.
- Despite efforts to make the model refuse inappropriate requests, it will sometimes respond to harmful instructions or exhibit biased behaviour.
Open AI says it is eager to collect user feedback to aid their ongoing work to improve this system.
Looking to the future
Professor Ellis and others said student cheating was reflective of a broader issue of universities and educational institutions emphasising assessment over “genuine learning“. Kane Murdoch agrees and adds that we are currently focusing students’ minds on assessment rather than learning. “We need to better understand the pros and cons, consider how such tools fit into professional contexts, and engage in a very serious change process with regard to how and what we assess.”
Sources
“ChatGPT appears to pass medical school exams. Educators are now rethinking assessments”, James Purtill, ABC Science, January 11, 2023
“New York City school officials block access to controversial artificial intelligence ChatGPT to stop students generating essays” ABC NET, January 8, 2023