Quotes from Treisman's talk, on this slide:
- "You can see from this that if you control for child poverty, we're pretty much on the top of the world."
- "The PISA scores mask the fact that child poverty rates were the principal factor in performance, not the particular structures of the country's education systems."
I tried to track down his data. PISA scores are easy enough to find, and the PSID, CNEF, UNICEF stuff comes from a UNICEF report, "Measuring Child Poverty." Cool. I dug in.
I noticed something on the Treisman slide. I saw Finland and Switzerland among the top scorers, but I didn't see any of the East Asian countries that regularly appear at the top of the PISA scores. It turns out that the report doesn't have child poverty rates for all of them, but that UNICEF report does have Japan's child poverty rates. I also noticed that Treisman's slide doesn't include all of the PISA or UNICEF countries, so I thought that I would put all of the data from PISA and UNICEF into an excel spreadsheet and graph them all.
This tells a bit of a more complicated story, no?
Also:
- As far as I can tell, these are the countries missing from Tresiman's slide:
- From the UNICEF report: "Underlying weak monitoring is the lack of any robust public or political consensus on how child poverty should be defined and measured." Some people think that relative child poverty should be measured, i.e. percentage of children below 50% of the median income of the country. Others want to use absolute poverty measures. There's one called the Deprivation Index that is determined by things like "Percentage of Children with Three Meals a Day" and "Percentage of Children that have some new clothes." Which measure of poverty matters for education: relative or absolute poverty?
- There's another child poverty report out there, with data from South Korea, another PISA high-flyer. The report is from the OECD. I didn't include their data in this graph, but South Korea is above the mean, near Ireland and Slovakia, by their measures.
- We don't have good data on child poverty for some of the other PISA rockstars. That includes a lot of the East Asian data points: Hong Kong, Shanghai, Singapore, Taiwan. If you think that they're better on child poverty than similar scoring countries, then this helps Treisman's point. If you think that they're worse on child poverty than similar scoring countries, than this hurts his point.
- Hey, take my spreadsheet. Mess around with it. (If, someday in the future, this link is broken, email me and I'll send it your way.)
Comments comments comments comments below.
Do you have the raw data? It seems to me that a trend-line would be useful on that data to see how strong the correlation is between the PISA scores and the levels of poverty in each country (although this should be interpreted with a large grain of salt given how different countries measure poverty).
ReplyDeleteA link to my spreadsheet is at the bottom of the post. I added a line of best fit, and R^2 was 0.34.
DeleteOne point: it's not the countries that are measuring poverty, not exactly. UNICEF defines poverty relatively as below 50% of the median income, or something like that. The countries report median income and income distributions, which, for all I know, they collect using different sampling methods. I dunno how significant those differences are, if they exist.
Did you standardize the scores before R^2?
DeleteThank you. This is very useful information to see. I've heard good things about Japan's system (and bad), and they are very impressive on your chart, with high scores even though they have higher child poverty than Finland.
ReplyDeleteWhat that says to me is that a rational, consistent approach to math education will probably work well, whether it's similar to Finland, Japan, or something else. Both countries show lots of respect to teachers, and I think both give them extensive planning time compared to what U.S. teachers get. In other ways, I think they do things quite differently.
Japanese do lesson study, correct. They have time built in to the day to plan, observe each out, and I assume to grade.
DeleteInstead of comparing by poverty level, suppose you compared by percentile rank within the country. For example, compare the top 75% of the scorers in the US with the top 75% of the scorers in Finland, Japan, etc. I did this two or three years ago with TIMMS data. You will find that the the top 75% in the US score much lower than the top 75% in Finland, Japan, Canada, etc (525 to 575 or so). Of course some of the high US scorers will also be high poverty kids. And you could argue that their scores would be even higher if they were from more affluent families.
ReplyDeleteOn the othe hand, pulling the 25 high poverty kids out of 100 US test takers and the 5 high poverty kids out of 100 Finish test takers doesn’t make for a fair comparison either. Some of those 25 high poverty kids would be low scorers even if they were from more affluent families.
It seems to me that the reasonable approach is somewhere in between these two methods.
The spreadsheet is gone.
ReplyDeleteYour graph relates an average score (PISA) to a median based percentage (poverty). This is interesting. It might indeed be that the average school results get better if income is more leveled. The explanation could be that in countries with a high percentage of poor more children are left behind.
Yeah, sorry that the link got broken. I need to figure out a better hosting solution for my files (and my blog?).
DeleteTry this link for now.