How Computers Helped Indonesia Move From a Cheating Culture to a Learning Culture
This blog was originally published on the RISE website.
In Indonesia, cheating on national exams was so widespread that a mother who exposed that her child’s teacher promoted cheating was accused by other parents to be a ‘disgrace to the school’. In many schools, teachers and students shared answer sheets and helped each other during the exam. When cheating is so common, it is likely that teachers and students collude to cheat. In that case, there is little chance that they will work to mitigate it, so it becomes hard to fight. Since 2015, the Indonesian Government has aimed to prevent cheating practices with the implementation of computer-based testing (CBT) on Grade 9 and 12 national exams (the end of junior and secondary school, respectively). The CBT program draws exam items directly from a server, such that each student has a unique test version. This way, answer sheets are useless, and it is difficult for teachers and students to share answers during the exam.
In a recently published working paper, Menno Pradhan, Rahmawati, Daniel Suryadarma, Arya Swarnata and I evaluate the introduction of CBT. We find that CBT not only leads to substantially lower grades in junior secondary schools that were known to cheat on national exams, but it also reduces cheating among schools that still administer the exam on paper. Moreover, after CBT is implemented, exam scores increase again as cheating is no longer possible. This suggests that learning outcomes improve when CBT is used and that students learn less when they have the opportunity to cheat.
Cheating on the Grade 9 leaving exam was widespread
The magnitude of the cheating problem in Indonesia before the introduction of CBT becomes clear when we look at suspicious patterns on answer sheets. According to a so-called ‘integrity index’, which is generated by the Government of Indonesia based on previously developed algorithms that detect those patterns, there is substantial evidence of cheating on the national exam in 2015 for about a third of all junior secondary schools (Grades 7-9). These ‘low integrity schools’ are locally concentrated, as shown on the map below. In red districts (118 out of 514 districts), between 75 and 100 percent of the junior secondary schools, with 532,946 students or 13 percent of all exam takers, have suspicious answer patterns on their 2015 exam. This large number of cheating students and teachers point to the existence of a ‘cheating culture’ in these regions.
School average exam scores dropped substantially after the implementation of computer-based testing
Each year since 2015, schools have applied to participate in CBT once they have the required resources (i.e., computers and stable electricity). By 2019, 39,379 out of 50,124 junior secondary schools implemented CBT. Using administrative data of all junior secondary schools between 2015 and 2019, we find a drop in average exam scores as soon as schools switched to computers.
You might wonder if a decrease in exam scores is driven by students who are unfamiliar with computers. As shown in the figure below, we find that when schools switch from exams on paper to CBT, their average exam scores drop only if they had suspicious answer patterns (i.e., low integrity) before the intervention. The drop in the average exam scores of low integrity schools is almost 10 points on a scale from 0 to 100. We also find only a small difference in the effect between schools that already had a computer lab before the intervention (41 percent of the schools) and schools that did not. These findings provide supporting evidence for our assumption that the drop in exam scores is due to reducing the possibility of cheating, not due to a lack of computer skills.
When cheating is no longer possible, learning outcomes improve
It is still important for students to achieve high grades because the result determines the quality of their subsequent education. With higher grades, they can get access to higher ranked senior secondary schools. Also, local governments and schools gain prestige from high grades on the national exam, so they continue to pressure teachers to perform well. Accordingly, we find that exam scores improve after the first year of CBT implementation. In the third year of CBT implementation, the effect for schools with computer labs in 2015 was less than half the size of the effect in the first year of CBT implementation (-3.9 points compared to -8.9 points, respectively). Since cheating is virtually impossible, this improvement can only be achieved through students and teachers exerting more effort into learning outcomes.
When more schools in a district switch to computers, it also becomes more difficult for other schools to cheat
Schools within the same district switched to computers in different years. The computers not only changed the exam scores of the CBT schools; the exam scores of schools that continued their exams on paper also decreased as more schools surrounding them switched to computers. As shown in the table below, when all schools surrounding a school that takes the paper-based exam switched to computers, the average paper-based exam score of this school decreased by 9 points, and the integrity index increased by 21 points in a district with a cheating culture (dark orange and red on the map). In districts where less than 50 percent of schools had low integrity in 2015 (light orange on the map), there was no significant change in the average score or integrity index on paper-based exams as the percentage of CBT schools in the district increased.
Our results imply a norm change with respect to cheating. We have a few explanations for why cheating on paper-based exams becomes more difficult when surrounding schools cannot cheat anymore. It might be harder for them to obtain answer sheets. Teachers and students that take the computer-based exam have no reason to try to obtain the answer sheets and, therefore, they cannot forward them either. In addition, teachers from CBT schools might be stricter when proctoring the exams of schools that still take the paper-based exam, to which they are assigned by the district government. (The national exam proctoring procedure is that teachers never proctor students from the school in which they teach.) If their students cannot cheat, to ensure a fair competition, they might also prevent cheating in schools that conduct paper-based exams.
Taking our findings together, the transition from paper-based exams to computer-based exams has helped Indonesia to move from a cheating culture towards a learning culture. Cheating on national exams in Indonesia was concentrated in certain regions, pointing to the existence of a local cheating culture. In many districts, there was sufficient evidence for substantial cheating on the paper-based exams for more than 75 percent of the schools. We find that the computers induce a norm change with respect to cheating practices in these regions, while learning outcomes improve over time when cheating is no longer possible.
Emilie Berkhout
Research Associate
Specialized in impact evaluations in low- and middle-income countries.