Arrowhead High School
Media Request for Reaction to the Release of PISA Results
Recently
Arthur Thomas, reporter for the Waukesha Freeman, emailed me asking for my
reactions to the 2012 Programme for International Assessment (PISA)
results, which had just been released.
PISA tested 510,000 15-year-olds, from 65 countries out off an
approximate 28 million 15-year-olds globally.
The test measures students in the content areas of math, reading and science, and then ranks the participating countries based on the results.
If you have watched the news recently or read the paper you likely have
seen headlines that reported on American schools and students using terms like “failing,”
“stagnant,” “mediocre,” “slipping,” or “lagging.” As a result of responding to Mr. Thomas’ request and having shared my response with our Board of Education and administration, it was suggested
I share the communication with our Arrowhead parents.
I
apologize ahead of time for the length of the communication. The issue of the PISA results are controversial, yet the reuslts are used, in part, as rational for many of the federal and state mandates, thus I thought a little more than a brief comment was warranted. The following 3
questions were emailed by Mr. Thomas asking for my reaction:
- What is your general reaction to the results showing
the U.S. at the international average in reading and below average in reading
and math?
- How, if at all, do you use this kind of test
to inform what you do in the district?
- Are these international comparisons a good
idea?
//////////////////////////////////////////////////////////////////////////////////////////////////////
(Response
emailed to Mr. Thomas at the Freeman)
Arthur,
I have spent very little time reviewing
information relative to the PISA results but will try to answer your
questions and provide my general thoughts on this issue.
First, I would state that southeastern
Wisconsin and more specifically the Freeman readership area has some of the top
schools in the State of Wisconsin, which in-turn makes them some of the top
schools in the nation. With that said, we all have areas that we are
working on to improve in better serving our young people. What I have
observed is that Arrowhead and the schools around us embrace the challenges of improving
our schools, are working extremely hard to enhance the quality of our schools
and continue to provide exceptional learning experiences for our
students.
In response to your questions:
1. What is your general reaction to the
results showing the U.S. at the international average in reading and below
average in reading and math?
My "general reaction" is that I see very little value for us locally
in the information provided by PISA. I believe the PISA represents approximately 500,000 students (there are approximately 50
million, public elementary and secondary schools students in the US) tested
with high numbers coming from Massachusetts, Connecticut and Florida, which
expanded their testing. I'm not sure what could be gained from this sampling
that would be of benefit to our school in terms of working with individual students, groups of
students or our student population as a whole. Quite frankly, I don't see
how the results have any significant meaning for southeastern Wisconsin or the
state of Wisconsin.
Again, I want to emphasis that as a school we
have areas we have identified that we need to improve. Finding little
value in the PISA does not mean that we don't have continued work as a
school system. Our staff is very focused on the identified areas and are
working diligently. We use local assessments and data collected regularly
to direct and influence our daily work with individual students. The PISA, when
used to rank countries would appear to have some areas of concern, just based
on the little I know about the test and procedures. A few examples (I'm
sure you can find other areas of concern with a little research):
· In China only scores from
select regions allowed for release, not poorer regions (i.e. financial hub of
Shanghai, one of the country's richest cities, is not representative of China's
overall education level)
· No correction for income
inequality: Poverty matters, socio-economic circumstances matters and
there is no factoring for this in the results reported from PISA.
The media reports proclaim US schools are failing...the myth persists
that once our nation led the world on international tests, but we have some how
fallen from the top position in recent years. The U.S. has never been
first in the world, nor even near the top, on international tests.
By Alfie Kohn
"Beliefs that are debatable or even patently false may be repeated so
often that at some point they come to be accepted as fact. We seem to
have crossed that threshold with the claim that U.S. schools are significantly
worse than those in most other countries. Sometimes the person who
parrots this line will even insert a number — “We’re only ____th in the world,
you know!” — although, not surprisingly, the number changes with each
retelling.
The assertion that our students compare unfavorably to those in other countries
has long been heard from politicians and corporate executives whose goal is to
justify various “get tough” reforms: high-stakes testing, a nationalized
curriculum (see under:Common Core “State” Standards), more homework, a longer
school day or year, and so on.
But by now the premise is so widely accepted that it’s casually repeated by
just about everyone — including educators, I’m sorry to say — and in the
service of a wide range of prescriptions and agendas, including some that could
be classified as progressive. Recently I’ve seen it used in a documentary
arguing for more thoughtful math instruction, a petition to promote teaching
the “whole child,” and an article in a popular on-line magazine that calls for
the abolition of grades (following a reference to “America’s long steady
decline in education”).
Unsurprisingly,
this misconception has filtered out to the general public. According to
abrand-new poll, a plurality of Americans — and a majority of college
graduates! — believe (incorrectly) that American 15-year-olds are at the bottom
when their scores on tests of science knowledge are compared to those of
students in other developed countries.[1]
A dedicated group of education experts has been challenging this canard for
years, but their writings rarely appear in popular publications, and each of
their efforts at debunking typically focuses on just one of the many problems
with the claim. Here, then, is the big picture: a concise overview
of the multiple responses you might offer the next time someone declares that
American kids come up short. (First, though, I’d suggest politely
inquiring as to the evidence for his or her statement. The wholly
unsatisfactory reply you’re likely to receive may constitute a rebuttal in its
own right.)
1. Even taking the numbers at face value, the U.S. fares reasonably well.
Results will vary depending on the age of the students being tested, the
subject matter, which test is involved, and which round of results is being
reported. It’s possible to cherry-pick scores to make just about any
country look especially good or bad. U.S. performance is more impressive
when the focus is on younger students, for example — so, predictably, it’s the
high school numbers that are most often cited. When someone reduces our
schools to a single number, you can bet it’s the one that casts them in the
worst possible light.
But even with older students, there may be less to the bad news than meets the
eye. Asan article in Scientific American noted a few years back, most
countries’ science scores were actually pretty similar.[2] That’s worth
keeping in mind whenever a new batch of numbers is released. If there’s
little (or even no) statistically significant difference among, say, the
nations placing third through tenth, it would be irresponsible to cite those
rankings as if they were meaningful.
Overall, when a pair of researchers carefully reviewed half a dozen different
international achievement surveys conducted from 1991 to 2001, they found that
“U.S. students have generally performed above average in comparisons with
students in other industrialized nations.”[3] And that still seems to be
the case based on the most recent data, which include math and science scores
for grade 4, grade 8, and age 15, as well as reading scores for grade 4 and age
15. Of those eight results, the U.S. scored above average in five,
average in two, and below average in one.[4] Not exactly the dire picture
that’s typically painted.
2. What do we really learn from standardized tests? While there are
differences in quality between the most commonly used exams (e.g., PISA, TIMSS), the
fact is that any one-shot, pencil-and-paper standardized test — particularly
one whose questions are multiple-choice — offers a deeply flawed indicator of
learning as compared with authentic classroom-based assessments.[5] The
former taps students’ skill at taking standardized tests, which is a skill unto
itself; the latter taps what students have learned, what sense they make of it,
and what they can do with it. A standardized test produces a summary
statistic labeled “student achievement,” which is very different from a
narrative account of students’ achievements. Anyone who cites the results
of a test is obliged to defend the construction of the test itself, to show
that the results are not only statistically valid but meaningful.
Needless to say, very few people who say something like “the U.S. is
below average in math” have any idea how math proficiency has been measured.
3. Are we comparing apples to watermelons? Even if the tests were
good measures of important intellectual proficiencies, the students being
tested in different countries aren’t always comparable. As scholars Iris
Rotberg and the late Gerald Bracey have pointed out for years, some countries
test groups of students who are unrepresentative with respect to age, family
income, or number of years spent studying science and math. The older,
richer, and more academically selective a cohort of students in a given
country, the better that country is going to look in international
comparisons.[6]
4. Rich American kids do fine; poor American kids don’t. It’s
ridiculous to offer a summary statistic for all children at a given grade level
in light of the enormous variation in scores within this country. To do
so is roughly analogous to proposing an average pollution statistic for the
United States that tells us the cleanliness of “American air.” Test
scores are largely a function of socioeconomic status. Our wealthier
students perform very well when compared to other countries; our poorer
students do not. And we have a lot more poor children than do other
industrialized nations. One example,supplied by Linda Darling-Hammond: “In
2009 U.S. schools with fewer than 10 percent of students in poverty ranked
first among all nations on PISA tests
in reading, while those serving more than 75 percent of students in poverty
scored alongside nations like Serbia, ranking about fiftieth.”[7]
5. Why treat learning as if it were a competitive sport? All of
these results emphasize rankings more than ratings, which means the question of
educational success has been framed in terms of who’s beating whom. This
is troubling for several reasons.
a) Education ≠ economy. If our reason for emphasizing students’
relative standing (rather than their absolute achievement) has to do with
“competitiveness in the 21st-century global economy” — a phrase that issues
from politicians, business people, and journalists with all the thoughtfulness
of a sneeze, then we would do well to ask two questions. The first, based
on values, is whether we regard educating children as something that’s
primarily justified in terms of corporate profits.
The second question, based on facts, is whether the state of a nation’s economy
is meaningfully affected by the test scores of students in that nation.
Various strands of evidence have converged to suggest that the answer is
no. For individual students, school achievement is only weakly related to
subsequent workplace performance. And for nations, there’s little
correlation between average test scores and economic vigor, even if you try to
connect scores during one period with the economy some years later (when that
cohort of students has grown up).[8] Moreover, Yong Zhao has shown that “PISA scores in reading, math, and sciences are negatively
correlated with entrepreneurship indicators in almost every category at
statistically significant levels.”[9]
b) Why is the relative relevant? Once we’ve refuted the myth that
test scores drive economic success, what reason would we have to fret about our
country’s standing as measured by those scores? What sense does it make
to focus on relative performance? After all, to say that our students are
first or tenth on a list doesn’t tell us whether they’re doing well or poorly;
it gives us no useful information about how much they know or how good our
schools are. If all the countries did reasonably well in absolute terms,
there would be no shame in being at the bottom. (Nor would “average” be
synonymous with “mediocre.”) If all the countries did poorly, there would
be no glory in being at the top. Exclamatory headlines about how “our”
schools are doing compared to “theirs” suggest that we’re less concerned with
the quality of education than with whether we can chant, “We’re Number One!”
c) Hoping foreign kids won’t learn? To focus on rankings is not only
irrational but morally offensive. If our goal is for American kids to
triumph over those who live elsewhere, then the implication is that we want
children who live in other countries to fail, at least in relative terms.
We want them not to learn successfully just because they’re not
Americans. That’s built into the notion of “competitiveness” (as opposed
to excellence or success), which by definition means that one individual or
group can succeed only if others don’t. This is a troubling way to look
at any endeavor, but where children are concerned, it’s indefensible. And
it’s worth pointing out these implications to anyone who cites the results of
an international ranking.
Moreover, rather than defending policies designed to help our graduates
“compete,” I’d argue that we should make decisions on the basis of what will
help them learn tocollaborate effectively. Educators, too, ought to think
in terms of working with – and learning from – their counterparts in other
countries so that children everywhere will become more proficient and
enthusiastic learners. But every time we rank “our” kids against
“theirs,” that outcome becomes a little less likely."
–
NOTES
1. Pew Research Center for People and the Press, “Public’s Knowledge of
Science and Technology,” April 22, 2013. Available at: www.people-press.org/2013/04/22/publics-knowledge-of-science-and-technology/.
2. W. Wayt Gibbs and Douglas Fox, “The False Crisis in Science
Education,” Scientific American, October 1999: 87-92.
3. Erling E. Boe and Sujie Shin, “Is the United States Really Losing the
International Horse Race in Academic Achievement?” Phi Delta Kappan, May 2005:
688-695.
4. National Center for Economic Statistics, Average Performance of U.S.
Students Relative to International Peers on the Most Recent International
Assessments in Reading, Mathematics, and Science: Results from PIRLS 2006,
TIMSS 2007, and PISA 2009, 2011.
Available at: http://nces.ed.gov/surveys/international/reports/2011-mrs.asp
5. See, for example, Alfie Kohn, The Case Against Standardized Testing
(Heinemann, 2000); or Phillip Harris et al., The Myths of Standardized Tests
(Rowman & Littlefield, 2011).
6. For example, see Iris C. Rotberg, “Interpretation of International
Test Score Comparisons,” Science, May 15, 1998: 1030-31.
7. Linda Darling-Hammond, “Redlining Our Schools,” The Nation, January
30, 2012: 12. Also see Mel Riddile, “PISA: It’s Poverty Not Stupid,” The Principal Difference[NASSP blog],
December 15, 2010 (http://bit.ly/hiobMC); and Martin Carnoy and
Richard Rothstein, “What Do International Tests Really Show About U.S.
Student Performance?”, Economic Policy Institute report, January 28, 2013 (http://www.epi.org/publication/us-student-performance-testing/).
8. Keith Baker, “High Test Scores: The Wrong Road to National Economic
Success,”Kappa Delta Pi Record, Spring 2011: 116-20; Zalman Usiskin, “Do We
Need National Standards with Teeth?” Educational Leadership, November 2007: 40;
and Gerald W. Bracey, “Test Scores and Economic Growth,” Phi Delta Kappan,
March 2007: 554-56. “The reason is clear,” says Iris Rotberg.
“Other variables, such as outsourcing to gain access to lower-wage employees,
the climate and incentives for innovation, tax rates, health-care and
retirement costs, the extent of government subsidies or partnerships,
protectionism, intellectual-property enforcement, natural resources, and
exchange rates overwhelm mathematics and science scores in predicting economic
competitiveness” (“International Test Scores, Irrelevant Policies,” Education
Week, September 14, 2001: 32).
9. Yong Zhao, “Flunking Innovation and Creativity,” Phi Delta Kappan,
September 2012: 58. Emphasis added.
2. How, if at all, do you use this kind of
test to inform what you do in the district?
We do not use the PSIA test results. The
results tell us nothing about how an individual student is performing, how
specific groups of students at Arrowhead are performing, how students in
general at Arrowhead are performing, how students in southeastern Wisconsin are
performing, or even how students in Wisconsin are performing, so the
information is of little or no value in providing meaningful data that supports
improving our school or supporting the individual learning needs of our
students. I would reference #4 from Kohn's article above:
"4. Rich American kids do fine; poor
American kids don’t. It’s ridiculous to offer a summary statistic for all
children at a given grade level in light of the enormous variation in scores
within this country. To do so is roughly analogous to proposing an
average pollution statistic for the United States that tells us the cleanliness
of “American air.” Test scores are largely a function of socioeconomic
status. Our wealthier students perform very well when compared to other
countries; our poorer students do not. And we have a lot more poor
children than do other industrialized nations. One example,supplied by
Linda Darling-Hammond: “In 2009 U.S. schools with fewer than 10 percent
of students in poverty ranked first among all nations on PSIA tests in reading, while those serving more than 75 percent of
students in poverty scored alongside nations like Serbia, ranking about
fiftieth.”[7]"
3. Are these international comparisons a good
idea?
I think you can glean from Kohn's article several
reasons why drawing "international comparisons" might be flawed and
should be of concern.
I realize you may have been looking for a few
quotes to put in an article but truly I don't know how to respond to your
questions without providing the kind of detail Kohn outlines for consideration
on this issue and challenging you in your effort to report accurately to take a
closer look into what this "ranking" means and how it was
determined.
Again, we all have areas for improvement and I
truly believe the area schools are working diligently to make great things
happen for kids on a daily basis. There is much more to our schools than
what is measured on a one-time, multiple choice exam. The factors
impacting learning are complex and no standardized test can even begin to
provide the measures necessary to assess the quality of our schools. If
there truly are concerns with the international comparisons, it might be time
to start asking questions of those responsible for implementing the reform
mandates (i.e. NCLB, Race to
the Top, Educator Effectiveness) in our country that have had significant
influence and impact on our schools.
The more we focus on standardized tests, the
more we kill creativity, ingenuity, and the ability to think differently. We
have students who think and learn differently, who are very bright and yet they
get lower scores. The more we focus on tests, the more we ask students to
conform and to be compliant, working towards getting the right answer, the more
we will stifle their learning and acquiring the skills that will truly support
them in the future.
In 1983 a federal report, “A Nation at Risk”, stated that the country was in
desperate trouble because of the poor academic performance of our students. The
report referenced international test scores as a reason for the concern.
The report stated, our society is being eroded due to the increasing
mediocrity in our public education system and threatened the future of our
country. I think the United States is still the world's most dominant
economy. I think if you did your research you would not find a
correlation between standardized test scores of countries and the countries
economy. How is it that the U.S. scores so poorly on international tests
continually over time and still is the world's leading economy?
My "general reaction" can probably be summed up by the following
points:
·
A ranking of
international test scores that uses a sampling that would represent approximately 1% of the
students in U.S. public schools and 2% of the 15-year-olds globally and then is used to determine how well our schools are
doing and in-turn is used as the rational for blanket federal policy and
one-size-fits-all solutions (NCLB & RTTT) is very concerning. As just
a couple of examples of the data that is used locally to measure student
performance and to direct learning at Arrowhead in contrast to international
rankings like PISA, I would share the following:
Parents are asked in selecting courses for their
child as an incoming freshman, to determine the appropriateness of taking an
Advanced Placement (AP) 9th grade course. It is recommended that students
have a 22 in reading on the ACT, along with other considerations, to provide
the best opportunity for them to succeed. Of the 90+ students in the 9th
grade AP class the average ACT reading score is 27. Parent/student
informed choice, drives course selection and provides the a measure that
supports students being successful in this course.
Struggling learners in the area of reading are
provided with a lexile reading score. These students are below grade
level in reading and in need of additional support and resources. The
lexile score is determined and then instruction, interventions, supports, and
resources are devoted to students in supporting their learning needs.
Typically, students can expect a normal growth of around 40 to 50 lexile
points (most of these students have not been experiencing normal growth).
Based on the efforts of the students and staff we are routinely seeing
growth of anywhere between 75 and 150 lexile points.
These are just a couple of examples of the
many tools and assessments being used daily with staff and students to improve
learning. The value of an international ranking, irregardless of the
tests validity, really pales in comparison to anything we are doing locally.
· If the PISA scores
demonstrate anything it is the failure of the past 10/15 years of public policy
in the country. The federal focus/reform movement of standardized testing
and increased accountability, with billions of dollars and huge amounts of
human resources expended have not raised test scores or our countries
"ranking" on the international comparisons. Federal and state
government mandates such as No Child Left Behind and Race to the Top have
failed to accomplish the intended goals of raising test scores.
· Poverty and
socio-economic factors matter. Approximately 22% of our students in the
U.S. live in poverty. What is being done to impact a factor that there is
substantial data to support has a significant impact on student academic
achievement?
· Standardized tests are
a one-time, multiple choice test, where students are measured on their ability
to pick the correct answer to a specific test question. What we measure
is what can be measured. Skills that most would consider important in a
young person's future success (creativity, imagination, work ethic,
problem-solving, ingenuity, collaboration - ability to work with others,
perseverance, etc.) are not measured. My concern is the policies of the
federal government (No Child Left Behind and Race to the Top) and the emphasis
on accountability, measured exclusively by standardized tests are jeopardizing
the qualities that have made our country the most innovative and talented
nation in the world.
· We should review and
question the results of a state like Florida. Florida embraced the model of
testing, accountability and choice. If we value these test scores then we
should look hard at a state like Florida that has implemented these measures
yet their scores in math and science are below the U.S. average and reading is
average. The data, in this instance does not show that a state who has
lead in these efforts has resulted in raising the test scores.
I would be glad to discuss this further with
you. Hopefully this information answers your questions.
Craig
//////////////////////////////////////////////////////////////////////////////////////////////////////
After having read this information, provided to the Freeman, if you have any questions please feel free to contact me directly.
Sincerely,
Craig Jefson
Arrowhead Superintendent
