Home   /  News


New tests hold promise, face perils

By Heather Vogell - The Atlanta Journal-Constitution

Pockets of resistance to state standardized tests — even in traditionally pro-testing states such as Texas and Florida — cropped up across the country this spring as students sat for the yearly exams.

Some advocates even urged parents to “opt out” and refuse to allow their children to participate. Repeated errors by testing companies were among the chief complaints.

And yet, largely lost in the public conversation about the country’s aggressive testing regimen is talk of how to make quality-control an ongoing priority.

The concerns come with testing in flux nationwide as most states revamp the exams to measure knowledge of national Common Core standards.

But new tests are likely to face many of the same challenges that an Atlanta Journal-Constitution investigation found bedeviled the old ones – a breakneck race to get scores returned, a lack of state personnel to provide oversight, and a high volume of near-simultaneous test-takers. The tests will face new hurdles as well, experts say, such as greater reliance on technology.

In general, advancements in testing have not only ushered in improvements, but also temporarily heightened the risk of error, said Ellen Haley, president of CTB/McGraw-Hill. “That’s when things can happen that you didn’t expect,” she said.

Already, work on the Common Core exams by two “consortia” of states has taxed state departments and the testing companies involved, and complaints have flared about cost.

“Everyone’s work level essentially has doubled over the last couple years,” said James Mason, Mississippi’s director of student assessment. “You’re trying to build a new airplane while flying your own.”

States that have bowed out of the common tests, including Georgia and Indiana, will face similar issues, whether they create new tests alone or partner with smaller groups of states. Some will work with a smaller budget than the states that belong to consortia.

Across the country, states’ testing programs have suffered for years from quality-control failures that, at times, threatened to hurt the students they were supposed to help. Thousands of pages of documents obtained by The Atlanta Journal-Constitution detailed more than 100 testing snafus and revealed flaws in questions on many exams – even as states used them to make key decisions about schooling.

Whether the new consortia of states will address such problems is an open question, said Thomas Toch, of the Carnegie Foundation for the Advancement of Teaching. Doing so would cost more money, he said.

Representatives of the groups said they plan to offer tests that are both more sophisticated and less prone to error.

“I’m not saying we have all this locked down at this point,” said Jeff Nellhaus, director of assessment policy research and design for the Partnership for Assessment of Readiness for College and Careers (PARCC) coalition.

But, he said, the group recognizes that steps such as scoring need a high level of oversight to make sure results are accurate and comparable across states. Test questions will undergo careful analysis. Audits and redundant checks will be built into the process.

Member states will still have some oversight responsibilities, he said, but they will also have PARCC’s support. “Over the course of the next two decades there might be a little problem every once in a while,” he said. “We’re going to minimize it.”

The Smarter Balanced consortium will offer states a common pool of rigorously reviewed questions and access to software proven to work, said Executive Director Joe Willhoft. States will also undergo certification that their tests met key standards from a prominent testing research center at the University of California, Los Angeles.

Scoring, however, will remain the states’ job. They will hire their own contractors and develop their own plans for oversight through measures such as reviewing analyses of question responses.

The consortium is trying to learn from the experiences of states that had a better track record with testing, Willhoft said. “There were a lot of states doing really good things,” he said. “It’s just, not all states were doing good things.”

Oversight board proposed

Industry officials and experts point to progress made over the years to reduce testing mishaps through measures such as the publication of a “best practices” guide and federal peer reviews of state programs.

Those efforts, however, don’t regularly scrutinize how well the exams actually perform in a given state. Nor do they examine testing company contracts or handle complaints about problem tests.

Such gaps have led some testing experts and observers to advocate creating an oversight board for testing.

Toch is among them. In a 2006 report, he argued for a national testing oversight agency. Such an agency could audit state testing programs and function like federal consumer-protection agencies, he wrote.

“These experts would in an unbiased way, with an independent perspective, take the measure on an ongoing basis of the profession to reduce the amount of error,” he said in a recent interview.

The goal: To make sure the standards for the tests were as high as the expectations for the students taking them.

The idea is promising, said Chris Domaleski, a senior associate at the National Center for the Improvement of Educational Assessment, as long as it preserves the flexibility states need to carry out their programs.

“I don’t hear any of my colleagues in the field objecting to more scrutiny and oversight,” he said. “Any sensible professional in the field would welcome it.”

The proposal also gets support from Karran Harper Royal, an activist and parent of two in New Orleans. Testing errors have helped erode public confidence in the exams, and she said an independent board could address some of those concerns.

“If we are going to base so much on tests,” she said, “there needs to be an independent entity that evaluates the tests to make sure they are of the highest quality.”

Such a measure would likely need to overcome some industry resistance, however.

Stuart Kahl, a co-founder of testing company Measured Progress, said talk years ago about such an external auditing agency went nowhere. “My problem is I find it very difficult to believe that any outside group would be knowledgeable enough to do a good job of it,” he said. “I just don’t think that will solve anything.”

Others also have doubts. Jon Cohen, executive vice president and director of assessment at the American Institutes for Research, said the real motivator for testing companies is competition in the industry.

“Oversight boards are not typically that effective because there is so much nitty-gritty to get into,” he said.

Fines as deterrent

There is one step states can take now to increase pressure on the industry to deliver better quality control. That is to include in new contracts potential fines for errors.

Only about 20 states used such a power during the past decade, according to states’ responses to information requests from The Atlanta Journal-Constitution. Some, such as Vermont, said they had no reason to. But others didn’t specify in their contracts the power to seek compensation for errors.

Kahl said bad publicity and damages clauses in contracts push companies to get better. “Testing companies – they don’t like penalties, they don’t like bad press,” he said.

States must negotiate such provisions with testing companies. Texas quietly dropped from its contract with Pearson last year a phrase that made it easier for the state to seek damages when the company made a mistake. Then-Education Commissioner Robert Scott and his chief-of-staff were unaware of the change.

“We would never approve that,” said Scott, who left the department later that year after criticizing testing. The deletion put the state in a precarious position, he said.

Texas education officials said that while the contract change deleted some legal language, it still allows the state to seek reduced payments to Pearson when problems arise.

‘A big mistake’

A final avenue for addressing quality-control problems on tests, some experts say, may simply be to revisit policies that force schools and districts to use test scores to make critical decisions.

“I think we’ve created an environment where we want to replace human judgment with results from tests,” said Kahl, the testing executive. “And that’s a big mistake.”

Peter Behuniak has devoted his career to testing. A former testing director for Connecticut, he’s now a measurement professor at the University of Connecticut.

And yet, his advice in recent years has taken on a cautionary tone.

“A lot of people think that the solution to whatever problem they are focused on is another test,” he said. “More is not necessarily better.”

Popular on MyAJC.com



Living Intown Magazine – a product of the Atlanta Journal-Constitution

Gain an insider’s perspective into the unique neighborhoods of in town Atlanta. Our lifestyle magazine explores all the things that make in-town living so appealing – from restaurants and attractions to the neighborhoods and their distinct personalities. Available on your desktop, tablet and smartphone and at select outlets around town.

Learn More

Read Today's Paper online

Read Today's Paper online

Still like to read the newspaper in the familiar page-by-page format? Great news! Digital versions of today's paper are available on your computer or tablet. And it's included in your subscription.

Learn moreRead