Friday, December 6, 2013

Media Request Reaction PISA Results

Arrowhead High School
Media Request for Reaction to the Release of PISA Results

Recently Arthur Thomas, reporter for the Waukesha Freeman, emailed me asking for my reactions to the 2012 Programme for International Assessment (PISA) results, which had just been released.  PISA tested 510,000 15-year-olds, from 65 countries out off an approximate 28 million 15-year-olds globally.  The test measures students in the content areas of math, reading and science, and then ranks the participating countries based on the results.  

If you have watched the news recently or read the paper you likely have seen headlines that reported on American schools and students using terms like “failing,” “stagnant,” “mediocre,” “slipping,” or “lagging.”  As a result of responding to Mr. Thomas’ request and having shared my response with our Board of Education and administration, it was suggested I share the communication with our Arrowhead parents.

I apologize ahead of time for the length of the communication.  The issue of the PISA results are controversial, yet the reuslts are used, in part, as rational for many of the federal and state mandates, thus I thought a little more than a brief comment was warranted.  The following 3 questions were emailed by Mr. Thomas asking for my reaction:

  1. What is your general reaction to the results showing the U.S. at the international average in reading and below average in reading and math?
  2. How, if at all, do you use this kind of test to inform what you do in the district?
  3. Are these international comparisons a good idea?

//////////////////////////////////////////////////////////////////////////////////////////////////////


(Response emailed to Mr. Thomas at the Freeman)

Arthur,

I have spent very little time reviewing information relative to the PISA results but will try to answer your questions and provide my general thoughts on this issue. 

First, I would state that southeastern Wisconsin and more specifically the Freeman readership area has some of the top schools in the State of Wisconsin, which in-turn makes them some of the top schools in the nation.  With that said, we all have areas that we are working on to improve in better serving our young people.  What I have observed is that Arrowhead and the schools around us embrace the challenges of improving our schools, are working extremely hard to enhance the quality of our schools and continue to provide exceptional learning experiences for our students. 

In response to your questions:

1.  What is your general reaction to the results showing the U.S. at the international average in reading and below average in reading and math?


My "general reaction" is that I see very little value for us locally in the information provided by PISA.  I believe the PISA represents approximately 500,000 students (there are approximately 50 million, public elementary and secondary schools students in the US) tested with high numbers coming from Massachusetts, Connecticut and Florida, which expanded their testing.  I'm not sure what could be gained from this sampling that would be of benefit to our school in terms of working with individual students, groups of students or our student population as a whole.  Quite frankly, I don't see how the results have any significant meaning for southeastern Wisconsin or the state of Wisconsin.  

Again, I want to emphasis that as a school we have areas we have identified that we need to improve.  Finding little value in the PISA does not mean that we don't have continued work as a school system.  Our staff is very focused on the identified areas and are working diligently.  We use local assessments and data collected regularly to direct and influence our daily work with individual students.  The PISA, when used to rank countries would appear to have some areas of concern, just based on the little I know about the test and procedures.  A few examples (I'm sure you can find other areas of concern with a little research):

·   In China only scores from select regions allowed for release, not poorer regions (i.e. financial hub of Shanghai, one of the country's richest cities, is not representative of China's overall education level)

·   No correction for income inequality: Poverty matters, socio-economic circumstances matters and there is no factoring for this in the results reported from PISA.  The media reports proclaim US schools are failing...the myth persists that once our nation led the world on international tests, but we have some how fallen from the top position in recent years.  The U.S. has never been first in the world, nor even near the top, on international tests.

So my "general reaction" would probably be much in line with what you would hear from Alfie Kohn, author of several books on education and human behavior.  The following is from a Washington Post article titled “We’re Number Umpteenth!”: The myth of lagging U.S. schools:

By Alfie Kohn

"Beliefs that are debatable or even patently false may be repeated so often that at some point they come to be accepted as fact.  We seem to have crossed that threshold with the claim that U.S. schools are significantly worse than those in most other countries.  Sometimes the person who parrots this line will even insert a number — “We’re only ____th in the world,  you know!” — although, not surprisingly, the number changes with each retelling.

The assertion that our students compare unfavorably to those in other countries has long been heard from politicians and corporate executives whose goal is to justify various “get tough” reforms:  high-stakes testing, a nationalized curriculum (see under:Common Core “State” Standards), more homework, a longer school day or year, and so on.

But by now the premise is so widely accepted that it’s casually repeated by just about everyone — including educators, I’m sorry to say — and in the service of a wide range of prescriptions and agendas, including some that could be classified as progressive.  Recently I’ve seen it used in a documentary arguing for more thoughtful math instruction, a petition to promote teaching the “whole child,” and an article in a popular on-line magazine that calls for the abolition of grades (following a reference to “America’s long steady decline in education”).  

Unsurprisingly, this misconception has filtered out to the general public.  According to abrand-new poll, a plurality of Americans — and a majority of college graduates! — believe (incorrectly) that American 15-year-olds are at the bottom when their scores on tests of science knowledge are compared to those of students in other developed countries.[1]

A dedicated group of education experts has been challenging this canard for years, but their writings rarely appear in popular publications, and each of their efforts at debunking typically focuses on just one of the many problems with the claim.  Here, then, is the big picture:  a concise overview of the multiple responses you might offer the next time someone declares that American kids come up short.  (First, though, I’d suggest politely inquiring as to the evidence for his or her statement.  The wholly unsatisfactory reply you’re likely to receive may constitute a rebuttal in its own right.)

1.  Even taking the numbers at face value, the U.S. fares reasonably well. Results will vary depending on the age of the students being tested, the subject matter, which test is involved, and which round of results is being reported.  It’s possible to cherry-pick scores to make just about any country look especially good or bad.  U.S. performance is more impressive when the focus is on younger students, for example — so, predictably, it’s the high school numbers that are most often cited.  When someone reduces our schools to a single number, you can bet it’s the one that casts them in the worst possible light.

But even with older students, there may be less to the bad news than meets the eye.  Asan article in Scientific American noted a few years back, most countries’ science scores were actually pretty similar.[2]  That’s worth keeping in mind whenever a new batch of numbers is released.  If there’s little (or even no) statistically significant difference among, say, the nations placing third through tenth, it would be irresponsible to cite those rankings as if they were meaningful.

Overall, when a pair of researchers carefully reviewed half a dozen different international achievement surveys conducted from 1991 to 2001, they found that “U.S. students have generally performed above average in comparisons with students in other industrialized nations.”[3]  And that still seems to be the case based on the most recent data, which include math and science scores for grade 4, grade 8, and age 15, as well as reading scores for grade 4 and age 15.  Of those eight results, the U.S. scored above average in five, average in two, and below average in one.[4]  Not exactly the dire picture that’s typically painted.

2.  What do we really learn from standardized tests?  While there are differences in quality between the most commonly used exams (e.g.,
 PISA, TIMSS), the fact is that any one-shot, pencil-and-paper standardized test — particularly one whose questions are multiple-choice — offers a deeply flawed indicator of learning as compared with authentic classroom-based assessments.[5]   The former taps students’ skill at taking standardized tests, which is a skill unto itself; the latter taps what students have learned, what sense they make of it, and what they can do with it.  A standardized test produces a summary statistic labeled “student achievement,” which is very different from a narrative account of students’ achievements.  Anyone who cites the results of a test is obliged to defend the construction of the test itself, to show that the results are not only statistically valid but meaningful.  Needless to say, very few people who say something like “the U.S. is below average in math” have any idea how math proficiency has been measured.

3.  Are we comparing apples to watermelons?  Even if the tests were good measures of important intellectual proficiencies, the students being tested in different countries aren’t always comparable.  As scholars Iris Rotberg and the late Gerald Bracey have pointed out for years, some countries test groups of students who are unrepresentative with respect to age, family income, or number of years spent studying science and math.  The older, richer, and more academically selective a cohort of students in a given country, the better that country is going to look in international comparisons.[6]

4.  Rich American kids do fine; poor American kids don’t.  It’s ridiculous to offer a summary statistic for all children at a given grade level in light of the enormous variation in scores within this country.  To do so is roughly analogous to proposing an average pollution statistic for the United States that tells us the cleanliness of “American air.”  Test scores are largely a function of socioeconomic status.  Our wealthier students perform very well when compared to other countries; our poorer students do not.  And we have a lot more poor children than do other industrialized nations.  One example,supplied by Linda Darling-Hammond:  “In 2009 U.S. schools with fewer than 10 percent of students in poverty ranked first among all nations on PISA tests in reading, while those serving more than 75 percent of students in poverty scored alongside nations like Serbia, ranking about fiftieth.”[7]

5.  Why treat learning as if it were a competitive sport?  All of these results emphasize rankings more than ratings, which means the question of educational success has been framed in terms of who’s beating whom.  This is troubling for several reasons.

a)  Education ≠ economy.  If our reason for emphasizing students’ relative standing (rather than their absolute achievement) has to do with “competitiveness in the 21st-century global economy” — a phrase that issues from politicians, business people, and journalists with all the thoughtfulness of a sneeze, then we would do well to ask two questions.  The first, based on values, is whether we regard educating children as something that’s primarily justified in terms of corporate profits.

The second question, based on facts, is whether the state of a nation’s economy is meaningfully affected by the test scores of students in that nation.  Various strands of evidence have converged to suggest that the answer is no.  For individual students, school achievement is only weakly related to subsequent workplace performance.  And for nations, there’s little correlation between average test scores and economic vigor, even if you try to connect scores during one period with the economy some years later (when that cohort of students has grown up).[8]  Moreover, Yong Zhao has shown that “PISA scores in reading, math, and sciences are negatively correlated with entrepreneurship indicators in almost every category at statistically significant levels.”[9]

b)  Why is the relative relevant?  Once we’ve refuted the myth that test scores drive economic success, what reason would we have to fret about our country’s standing as measured by those scores?  What sense does it make to focus on relative performance?  After all, to say that our students are first or tenth on a list doesn’t tell us whether they’re doing well or poorly; it gives us no useful information about how much they know or how good our schools are.  If all the countries did reasonably well in absolute terms, there would be no shame in being at the bottom.  (Nor would “average” be synonymous with “mediocre.”)  If all the countries did poorly, there would be no glory in being at the top.  Exclamatory headlines about how “our” schools are doing compared to “theirs” suggest that we’re less concerned with the quality of education than with whether we can chant, “We’re Number One!”

c) Hoping foreign kids won’t learn?  To focus on rankings is not only irrational but morally offensive.  If our goal is for American kids to triumph over those who live elsewhere, then the implication is that we want children who live in other countries to fail, at least in relative terms.  We want them not to learn successfully just because they’re not Americans.  That’s built into the notion of “competitiveness” (as opposed to excellence or success), which by definition means that one individual or group can succeed only if others don’t.  This is a troubling way to look at any endeavor, but where children are concerned, it’s indefensible.  And it’s worth pointing out these implications to anyone who cites the results of an international ranking.

Moreover, rather than defending policies designed to help our graduates “compete,” I’d argue that we should make decisions on the basis of what will help them learn tocollaborate effectively.  Educators, too, ought to think in terms of working with – and learning from – their counterparts in other countries so that children everywhere will become more proficient and enthusiastic learners.  But every time we rank “our” kids against “theirs,” that outcome becomes a little less likely."



NOTES

1.  Pew Research Center for People and the Press, “Public’s Knowledge of Science and Technology,” April 22, 2013.  Available at:
 www.people-press.org/2013/04/22/publics-knowledge-of-science-and-technology/.

2.  W. Wayt Gibbs and Douglas Fox, “The False Crisis in Science Education,”  Scientific American, October 1999:  87-92.

3.  Erling E. Boe and Sujie Shin, “Is the United States Really Losing the International Horse Race in Academic Achievement?” Phi Delta Kappan, May 2005: 688-695.

4.  National Center for Economic Statistics, Average Performance of U.S. Students Relative to International Peers on the Most Recent International Assessments in Reading, Mathematics, and Science: Results from PIRLS 2006, TIMSS 2007, and
 PISA 2009, 2011.  Available at: http://nces.ed.gov/surveys/international/reports/2011-mrs.asp

5.  See, for example, Alfie Kohn, The Case Against Standardized Testing (Heinemann, 2000); or Phillip Harris et al., The Myths of Standardized Tests (Rowman & Littlefield, 2011).

6.  For example, see Iris C. Rotberg, “Interpretation of International Test Score Comparisons,” Science, May 15, 1998: 1030-31.

7.  Linda Darling-Hammond, “Redlining Our Schools,” The Nation, January 30, 2012: 12.  Also see Mel Riddile, “
PISA: It’s Poverty Not Stupid,” The Principal Difference[NASSP blog], December 15, 2010 (http://bit.ly/hiobMC); and Martin Carnoy and Richard Rothstein, “What Do International Tests Really Show About  U.S. Student Performance?”, Economic Policy Institute report, January 28, 2013 (http://www.epi.org/publication/us-student-performance-testing/).

8. Keith Baker, “High Test Scores: The Wrong Road to National Economic Success,”Kappa Delta Pi Record, Spring 2011: 116-20; Zalman Usiskin, “Do We Need National Standards with Teeth?” Educational Leadership, November 2007: 40; and Gerald W. Bracey, “Test Scores and Economic Growth,” Phi Delta Kappan, March 2007: 554-56.  “The reason is clear,” says Iris Rotberg.  “Other variables, such as outsourcing to gain access to lower-wage employees, the climate and incentives for innovation, tax rates, health-care and retirement costs, the extent of government subsidies or partnerships, protectionism, intellectual-property enforcement, natural resources, and exchange rates overwhelm mathematics and science scores in predicting economic competitiveness” (“International Test Scores, Irrelevant Policies,” Education Week, September 14, 2001: 32).

9.  Yong Zhao, “Flunking Innovation and Creativity,” Phi Delta Kappan, September 2012: 58.  Emphasis added.


2. How, if at all, do you use this kind of test to inform what you do in the district?

We do not use the PSIA test results.  The results tell us nothing about how an individual student is performing, how specific groups of students at Arrowhead are performing, how students in general at Arrowhead are performing, how students in southeastern Wisconsin are performing, or even how students in Wisconsin are performing, so the information is of little or no value in providing meaningful data that supports improving our school or supporting the individual learning needs of our students.  I would reference #4 from Kohn's article above:

"4.  Rich American kids do fine; poor American kids don’t.  It’s ridiculous to offer a summary statistic for all children at a given grade level in light of the enormous variation in scores within this country.  To do so is roughly analogous to proposing an average pollution statistic for the United States that tells us the cleanliness of “American air.”  Test scores are largely a function of socioeconomic status.  Our wealthier students perform very well when compared to other countries; our poorer students do not.  And we have a lot more poor children than do other industrialized nations.  One example,supplied by Linda Darling-Hammond:  “In 2009 U.S. schools with fewer than 10 percent of students in poverty ranked first among all nations on PSIA tests in reading, while those serving more than 75 percent of students in poverty scored alongside nations like Serbia, ranking about fiftieth.”[7]"

3.  Are these international comparisons a good idea?

I think you can glean from Kohn's article several reasons why drawing "international comparisons" might be flawed and should be of concern.

I realize you may have been looking for a few quotes to put in an article but truly I don't know how to respond to your questions without providing the kind of detail Kohn outlines for consideration on this issue and challenging you in your effort to report accurately to take a closer look into what this "ranking" means and how it was determined. 

Again, we all have areas for improvement and I truly believe the area schools are working diligently to make great things happen for kids on a daily basis.  There is much more to our schools than what is measured on a one-time, multiple choice exam.  The factors impacting learning are complex and no standardized test can even begin to provide the measures necessary to assess the quality of our schools.  If there truly are concerns with the international comparisons, it might be time to start asking questions of those responsible for implementing the reform mandates (i.e. NCLB, Race to the Top, Educator Effectiveness) in our country that have had significant influence and impact on our schools.

The more we focus on standardized tests, the more we kill creativity, ingenuity, and the ability to think differently. We have students who think and learn differently, who are very bright and yet they get lower scores. The more we focus on tests, the more we ask students to conform and to be compliant, working towards getting the right answer, the more we will stifle their learning and acquiring the skills that will truly support them in the future.
 
In 1983 a federal report, “A Nation at Risk”, stated that the country was in desperate trouble because of the poor academic performance of our students. The report referenced international test scores as a reason for the concern.  The report stated, our society is being eroded due to the increasing mediocrity in our public education system and threatened the future of our country.  I think the United States is still the world's most dominant economy.  I think if you did your research you would not find a correlation between standardized test scores of countries and the countries economy.  How is it that the U.S. scores so poorly on international tests continually over time and still is the world's leading economy?

My "general reaction" can probably be summed up by the following points: 

·    A ranking of international test scores that uses a sampling that would represent approximately 1% of the students in U.S. public schools and 2% of the 15-year-olds globally and then is used to determine how well our schools are doing and in-turn is used as the rational for blanket federal policy and one-size-fits-all solutions (NCLB & RTTT) is very concerning.  As just a couple of examples of the data that is used locally to measure student performance and to direct learning at Arrowhead in contrast to international rankings like PISA, I would share the following:

Parents are asked in selecting courses for their child as an incoming freshman, to determine the appropriateness of taking an Advanced Placement (AP) 9th grade course.  It is recommended that students have a 22 in reading on the ACT, along with other considerations, to provide the best opportunity for them to succeed.  Of the 90+ students in the 9th grade AP class the average ACT reading score is 27. Parent/student informed choice, drives course selection and provides the a measure that supports students being successful in this course.

Struggling learners in the area of reading are provided with a lexile reading score.  These students are below grade level in reading and in need of additional support and resources.  The lexile score is determined and then instruction, interventions, supports, and resources are devoted to students in supporting their learning needs.  Typically, students can expect a normal growth of around 40 to 50 lexile points (most of these students have not been experiencing normal growth).  Based on the efforts of the students and staff we are routinely seeing growth of anywhere between 75 and 150 lexile points.  

These are just a couple of examples of the many tools and assessments being used daily with staff and students to improve learning.  The value of an international ranking, irregardless of the tests validity, really pales in comparison to anything we are doing locally.

·   If the PISA scores demonstrate anything it is the failure of the past 10/15 years of public policy in the country.  The federal focus/reform movement of standardized testing and increased accountability, with billions of dollars and huge amounts of human resources expended have not raised test scores or our countries "ranking" on the international comparisons.  Federal and state government mandates such as No Child Left Behind and Race to the Top have failed to accomplish the intended goals of raising test scores.

·  Poverty and socio-economic factors matter.  Approximately 22% of our students in the U.S. live in poverty.  What is being done to impact a factor that there is substantial data to support has a significant impact on student academic achievement?

·   Standardized tests are a one-time, multiple choice test, where students are measured on their ability to pick the correct answer to a specific test question.  What we measure is what can be measured.  Skills that most would consider important in a young person's future success (creativity, imagination, work ethic, problem-solving, ingenuity, collaboration - ability to work with others, perseverance, etc.) are not measured.  My concern is the policies of the federal government (No Child Left Behind and Race to the Top) and the emphasis on accountability, measured exclusively by standardized tests are jeopardizing the qualities that have made our country the most innovative and talented nation in the world.

· We should review and question the results of a state like Florida. Florida embraced the model of testing, accountability and choice.  If we value these test scores then we should look hard at a state like Florida that has implemented these measures yet their scores in math and science are below the U.S. average and reading is average.  The data, in this instance does not show that a state who has lead in these efforts has resulted in raising the test scores.  

I would be glad to discuss this further with you.  Hopefully this information answers your questions.

Craig

//////////////////////////////////////////////////////////////////////////////////////////////////////

After having read this information, provided to the Freeman, if you have any questions please feel free to contact me directly.

Sincerely,

Craig Jefson
Arrowhead Superintendent

2 comments:

  1. Excellent piece. Thank you for sharing these thoughts and quotes.

    ReplyDelete