Noted Educator Discusses Evaluating the Quality of Accountability Systems
WASHINGTON, April 22, 2015 /PRNewswire-HISPANIC PR WIRE/ — As Congress updates the No Child Left Behind (NCLB) Act, Brian Gong, Executive Director of the National Center for the Improvement of Educational Assessment, yesterday outlined the challenges facing policymakers and others charged with creating accountability systems based on realistic and reachable goals. His remarks came during Educational Testing Service’s 15th Angoff Memorial Lecture at the National Press Club.
The lecture, “Evaluating the Quality of Accountability Systems: Beyond Reliability and Validity” explored the policy debate surrounding the appropriate way to hold schools, districts and teachers accountable for student achievement.
“The essential question I am interested in,” Gong said, “is, how can we tell whether an accountability system is good? Or more generally, how should we evaluate the quality of accountability systems? Evaluating them is inextricably tied to the questions we ask about the accountability systems, and the procedures and evidence we use to answer those questions.”
Gong noted that it is becoming increasingly common for student assessment results to be applied in the process of evaluating schools, districts and teachers. This raises the question that even with reliable and valid student assessments, is this method of evaluation truly valid?
According to Gong, the logic underlying accountability systems is straightforward in its basic form:
- Decide what you want students to learn and how you want students, schools, teachers to perform
- Determine how students/schools/teachers have performed
- Decide whether that performance is “good enough”
- Decide what to do if it is or is not “good enough” (i.e., what should happen)
- Implement and monitor the accountability system
“We need to evaluate the quality of each accountability system part, and the whole together, to know we can trust the results and to improve the system” he said. “There are some sound, practical things we can and should do to guide our efforts to evaluate the quality of accountability systems.”
“Many people assume that the same means of evaluating the quality of assessment instruments apply to accountability systems. I believe that valid and reliable assessments do not guarantee high-quality accountability systems. Evaluation of the quality of accountability systems should go beyond checking whether the assessments are valid and reliable.”
He explained that accountability scores and decisions have their own characteristics regarding reliability and validity, and do not automatically inherit attributes of reliability or validity from the assessments upon which they are based. For example, he said, it is possible to have accountability decisions that are quite unreliable, even though the assessments out of which they are constructed are quite reliable. Similarly, he added, it is possible to have highly invalid accountability scores and decisions built upon valid assessments.
“Second, accountability systems incorporate action,” Gong said. “Accordingly, accountability systems’ prescribed actions and their consequences should be evaluated. For these, program and policy evaluation frameworks are often useful and more appropriate for evaluating the quality of this component of accountability systems than the construct validation frameworks that have been developed for assessments.”
Other factors to study include:
- the effect on reliability of combining individual student test scores to become a group score
- making accountability decisions based on subgroups — especially subgroups with few students — may lead to less-than-desired reliabilities especially between schools with different subgroup make-ups
- attending to aspects of validation proactively, in the design, development and improvement of assessments through evidence-centered design
- the operational feasibility, cost/quality trade-offs, compliance with security and other legal requirements
“Education is vitally important to students, parents, teachers and the nation at large,” said Ida Lawrence, ETS Senior Vice President of Research and Development. “A large part of that is designing accountability systems to inform everyone how we’re doing and guiding improvement. This presentation reinforces that education systems can be improved but it takes quality assessments, good data and solid research to inform our actions.”
Gong concluded, “Accountability systems may be a particularly American educational policy tool. We have student accountability, teacher evaluation, higher education accountability and a new federal consideration of what Congress will require of states in terms of school accountability. I hope that we can — also fueled by traditional American ingenuity, humility and persistence — evaluate our accountability systems more effectively so that 20 years from now we will say our schools, our accountability systems, and the lives of our children and grandchildren are better than they are today.”
The 15th William H. Angoff Memorial Lecture will be published later this year and available for downloading free of charge at http://www.ets.org/research/policy_research_reports/angoff/
About ETS
At ETS, we advance quality and equity in education for people worldwide by creating assessments based on rigorous research. ETS serves individuals, educational institutions and government agencies by providing customized solutions for teacher certification, English language learning, and elementary, secondary and postsecondary education, and by conducting education research, analysis and policy studies. Founded as a nonprofit in 1947, ETS develops, administers and scores more than 50 million tests annually — including the TOEFL® and TOEIC® tests, the GRE® tests and The Praxis Series® assessments — in more than 180 countries, at over 9,000 locations worldwide. www.ets.org
