Google and Eedi study 1,179 pupils: tutoring gains differ across tasks and broad tests

Eedi designs incorrect answer options to signal common misconceptions. For example, answering 1/2+1/3 with 2/5 could suggest adding numerators and denominators separately. Guiding questions could explore why equal-sized pieces give 3/6+2/6=5/6. This is an illustrative, fictional example, not an actual trial question or AI response.

Random assignment divides pupils so that starting attainment or learning preferences are less likely to concentrate in one condition. Each pupil kept the same condition, with different help after a mistake.

Pupils in all four conditions could also press an optional help button to ask a human tutor. The comparison concerns support automatically triggered by a mistake, rather than the complete presence or absence of human help.

The models were called inside Eedi and updated during the trial. Reading it as a single fixed model tested in ordinary chat would leave out the classroom setup.

In the AI conditions, a maths expert had to approve each message before it reached a pupil. The expert could accept it unchanged, ask for regeneration or rewrite it. A pupil’s next reply prompted another draft and another review.

%% Review before reaching the pupil · AI conditions in the Google–Eedi trial
flowchart TD
  A["Pupil’s mistake or next question"] --> B["Gemini drafts an explanation"]
  B --> C["Expert approves, regenerates or edits"]
  C --> D["Approved message delivered<br/>Pupil tries the maths again"]

Experts approved 89,406 of 94,385 reviewed drafts unchanged, about 94.7%. The other 4,979 were regenerated or rewritten. Some generated drafts were superseded by a new pupil message before review, so total generation and reviewed drafts have different denominators.

Solving a related problem after an explanation and taking a test spanning many topics measure different breadths of learning. The researchers did not use one score to stand for them all.

The team investigated a possible trade-off between depth and coverage. A tutoring conversation used roughly four to seven minutes, reducing study units covered in fixed lesson time. Across ten weeks, median completed units were 132 with standard support, 77 with human tutoring and 99 in each AI condition. A median is the middle value when individual counts are ordered.

This is an explanation under investigation, not a guarantee that giving tutoring longer will improve broad attainment. Sample composition and other factors also matter.

The report found no consistent additional benefit from learning-history AI over question-context AI. Its implementation, however, had a bug that prevented notes from earlier tutoring chats carrying into later sessions. Study-unit attainment, pace and misconceptions were provided; the full intended conversation history was not.

This does not establish that all personalization is ineffective. A feature called “learning history” also needs evidence about which information actually reached the system before its benefit can be assessed.