AI and Quizzes

I still think this was a sound idea, even though it didn’t produce the results that we hoped for…

Instead of paying expensive human beings to write quizzes for our www.educationquizzes website, we would employ ChatGPT for just $20.00 a month and get it to work full-time plus 40 hours overtime a week.

We would tell it what we needed and let it do the rest.

We believed that the key to success with ChatGPT was to combine minimal guidance with a highly detailed prompt. We designed a simple user interface that looked like this:

However, behind the scenes, a complex prompt provided the program with specific instructions, including:

• Determine question difficulty based on the curriculum (KS1, KS2, GCSE, etc.).
• Create a unique quiz title.
• Write relevant meta titles and descriptions.
• Compose an engaging introduction for each quiz.
• Develop ten age-appropriate questions.
• Ensure each question aligns with the curriculum.
• Write four plausible answers for each question.
• Identify the correct answer for each question.
• Add a ‘Helpful Comment’ with supplementary information for each question.
• Use all this information to fill the seventy fields in our Quiz-builder program needed to successfully launch the quiz.

As we refined these prompts, they grew increasingly complex, but after weeks of trial and error, we finally developed a prompt that worked tolerably well.

There were two significant problems:

Not Understanding Pain Points

A teacher knows the tricky spots in every curriculum. They understand that if you tell a child penicillin was discovered by ACCIDENT, it’s a fact they’ll remember forever. But who discovered it and when? That detail usually slips away for most seven-year-olds.

Teachers know where to focus time and energy to make sure the less exciting yet important facts are committed to memory.  ChatGPT doesn’t get it. ChatGPT expects that once a child is told something, there must be something wrong with it if it subsequently forgets. Facts, irrespective of their importance or insignificance are all given equal weight.

Plagiarism

During its training on tens of millions of web pages, ChatGPT likely encountered questions from the Education Quizzes website and perhaps thought, “If I’m ever asked about a school curriculum topic, I’ll use what I’ve learned here.” And indeed, the questions our teachers carefully crafted over years reappeared time and again. ChatGPT reliably reproduces what it considers the best examples based on the prompt but never truly generates anything new.

1. ChatGPT CAN write quizzes, but they are not as good as those written by teachers.
2. We could spend weeks and months refining the prompt to make ChatGPT do a better job.
3. We’ve decided to stick with the teachers!

Colin King – CEO of HR Quizzes