How Much Can PISA Scores Be Hacked?
I tried computing PISA Averages (from 2012) (with the UK’s average set to 99.2) with several changes to methodology:
Computer sample only
How I would compute them: take the first general factor of the three tests, control for age using a spline, and set the results to the UK’s SD.
Same as 1, but with sampling weights
3 - same as 2, but using “senate weights” (I don't know what kind of weights these are.)
Same as 1 (no weights/composite of 3 tests), but with no age corrections.
Using the average of the three tests with no weights or age controls, and assuming an SD of 100.
Combined Computer + Paper sample:
Take the first general factor of the three tests, control for age using a spline, and set the results to the UK’s SD.
Using the average of the three tests with no weights or age controls, and assuming an SD of 100.
Conclusion:
PISA averages are extremely hard to hack, though using general factor measurements normed to the UK’s SD would increase international differences.
Full disclosure, I have no idea what computational sorcery you are doing here, but PISA cheating in Asia happens on a different level. Nations decide themselves how a representative selection of students is picked. China, for instance, picks students from elite areas of the country, and then from the best schools within that area.
Rural Chinese areas have very poor school performance. In 2010, 2/3 of Chinese children lived in rural areas (note 2/3 of the total population lived in urban areas).
PISA scores seem easy to hack, and seemingly are. Let me know if you want sources.