School tests limited in what they can measure, experts say
Editor’s note: This is the final article in a three-part series about standardized testing in District 88.
NEW ULM — Every year each state in the U.S. assesses students via standardized tests as required by the Every Student Succeeds Act (ESSA).
The tests are purportedly used to measure individual student ability as well as the quality of the school, but as it turns out what the tests really measure is far more complicated.
“Standardized tests measure a lot of things and some of those things they are actually trying to measure and many of those things they are not trying to measure,” said Jack Schneider an assistant professor of education at the College of the Holy Cross in Worcester, Mass.
To start with, it may help to understand what the tests are intended to measure. Take the Minnesota Comprehensive Assessments (MCAs) for example.
The MCAs are designed to test how a school’s curriculum is aligned with the Minnesota state standards in English language arts, math and science, though state standards also exist for social studies and physical education, according to the Minnesota Department of Education’s (MDE) website.
“The MCAs are designed to show how a district or a school is implementing the standards,” Emily Bisek assistant director of communications of MDE said.
Those standards define what a student should be able to do by the end of a specific grade in a specified subject.
Take an example for third-grade math. A student must be able to “use geometric attributes to describe and create shapes in various contexts,” according to the MDE’s website.
Ultimately the standards should mean if a student achieves all or most of them by the time they graduate, they will be prepared for college or the working world.
“If we are going to make sure that every kid graduates from high school ready for life, than we have to have something that defines for teachers what these kids need to go at every step of the progression,” MDE Chief Academic Officer Greg Keith said.
With that in mind, even though ESSA requires states to develop individual reports for students, the MCA’s are not really aimed at measuring individual student success.
“This is not about student achievement levels, this is about a systems check — taking the temperature,” MDE Data and Recording Supervisor Kate Beattie said.
That way using comparable results from standardized tests, schools and districts can be analyzed.
There are a few caveats to that. First, standardized tests are what Schneider describes as “imperfect indicators.”
If a student gets a question on their English test correct or incorrect, that can indicate they understood the passage but does not really show what level of comprehension they are at.
More importantly, the test has no way of explaining why a student failed or succeeded on a question.
This gets into what tests inadvertently measure, a lot of which turn out to be related to socio-economic status.
“We know, for instance, from educational research that roughly two-thirds of students’ standardized test scores are shaped by out-of-school variables,” Schneider said.
In August Politico reported on a new program in Baltimore, Md. called Vision for Baltimore that is giving out free glasses to poor students. Among other results, students test scores went up.
A study published in June by researchers at the University of Wisconsin, Oshkosh and the University of South Carolina looked into how food stamps affect test scores.
The study titled “When Does it Count? The Timing of Food Stamp Receipt and Educational Performance,” used randomized dates of when families received food stamps along with administrative data on test results to find a connection between the two.
The study found that when assistance from the Supplemental Nutrition Assistance Program (SNAP) was received just days before a test date, scores were notably higher than near the end of the assistance cycle.
A handful of factors that Schneider pointed out standardized tests could not account for would be if a student comes from a wealthier background, or if their parents valued and enforced a subject such as reading, or if the child could simply be particularly good or bad at a subject.
None of those factors are within the control of a school, but influence how high a school’s quality appears, Schneider said.
“That is a part of what makes measuring school quality so different is that there are all of these invisible factors that have nothing to do with what a school is doing that really play a critical role in a young person’s success in school,” Schneider said.
Even the timing of a lesson can shift scores. If a tested subject is taught only after the test date, scores related to it will likely suffer even if the school is meeting state standards.
Where they fall flat
While standardized tests can measure a variety of factors, some unintentionally, they cannot measure everything that goes into being a good school.
“The MCAs are focused only on a discreet set of standards in a set of subject areas,” Keith said.
To begin with, they only measure performances in math, science and English language arts. The tests also cannot measure other forms of learning.
Writing a persuasive essay, putting together a research project or evaluating a piece of art were all examples of important academic work that tests miss, Schneider said.
“With these tests you cannot do verbal communication and eye contact and collaboration — how do you work together with others — those tests are not measuring that,” Superintendent Jeff Bertrang said. “they are just measuring your skill-based knowledge of math, science and English.”
While standardized tests do have shortcomings, Schneider argued they would be a fine metric if there was not so much value placed on them.
That emphasis encourages some schools to focus only on what is tested on the MCAs or equivalent assessments.
“It turns out that if you are trying to get your scores up on a standardized test, there are lots of ways to do so without actually promoting the kind learning that we want students to be engaged in,” Schneider said.
Which inverts one of the biggest purposes of standardized testing — keeping schools accountable.
“Just remember the fact that a standardized test can only measure learning in very particular ways and if you hold people accountable for that, then they will only encourage very particular kinds of learning and all other kinds of learning are therefore not valued and not promoted,” Schneider said.
Schneider, who is also the research director at the Massachusetts Consortium for Innovative Education Assessment (MCIEA), argues there is a better way to assess students.
What MCIEA wants to use is what it calls performance assessments. They include tests, quizzes, essays, projects, basically everything teachers are already using to assess their students’ learning.
The difference that MCIEA is working on, along with a few other organizations, is developing a standardized rubric.
A common rubric between schools would mean classroom level assessments could be compared across districts.
Until that becomes common place, and probably even after, it is important to remember that a test score is still just a test score.
“When you look at the full picture of the school, it is more than one test score,” Keith said. “Every school has a story and test scores are only one part of that story.”