SAT Process Review—Answer Sheet Processing
Executive Summary
In late March of this year, the College Board's General Counsel asked the consulting firm Booz Allen Hamilton to conduct a comprehensive review of answer sheet processing with a particular emphasis on the scanning process. The request was made in order to identify specific improvements that would minimize any risk of incorrect scores being reported in the future as a result of scanning problems or other related factors. Booz Allen conducted a thorough review of answer sheet scanning, answer sheet design, completed answer sheet receiving, opening, acclimatization, matching and validation.
In its comprehensive report, dated May 26, 2006, SAT Process Review—Answer Sheet Processing, Booz Allen reaffirmed the quality of the SAT scanning process and of the additional controls that were introduced beginning this spring. The report concludes that:
"...the current process is reliable and has prudent controls in place to safeguard scoring accuracy for those marks made in accordance with test directions. The operational changes made by College Board in response to the October administration further improved process reliability by introducing scanning redundancy, more frequent manual scoring checks, an environmental acclimation period to eliminate the effects of humidity, and other safeguards."
The controls, including the recent enhancements put in place in the spring of this year, have had their desired effect and we have seen no repeat of the October scanning problem. We will continue to consider further enhancements to make sure that the SAT remains the standard for reliability in standardized testing.
We asked Booz Allen for specific areas where we could improve performance, and recommendations for accomplishing that goal. Booz Allen created an exhaustive list of potential "risks" in the processing of answer sheets, and it analyzed the likelihood of each risk and how completely the risk is being addressed. The risks Booz Allen identified range from technical problems within the scanning process to the possibility that boxes of answer sheets could be lost or damaged in shipment.
Booz Allen then divided the list into "primary risks", that is, risks which would be likely to impact multiple answer sheets in an administration, cause scanning or editing errors, and result in a scoring error, and "secondary risks" which would be less likely to occur and would have a lesser impact, but still suggest areas where further enhancements might be considered.
Booz Allen found that "...all primary risks had prudent quality controls in place to prevent an error from occurring or to observe and correct an error before it propagated into a scoring error." In addition, they found that of the 51 identified "secondary risks," 35 are either "mitigated by quality controls or are unlikely to occur and if they did occur would have a very low/no impact on scoring." As a result of their analysis, Booz Allen recommended that we focus on the remaining 16 "secondary risks" and gave recommended actions for near-term process enhancements, as well as longer-term possibilities for future consideration.
The College Board is addressing each of the 16 identified "secondary risks" as a part of its ongoing quality enhancement program and is putting action plans in place to address each one. Several of the Booz Allen recommendations have already been implemented, such as the "double scan" (scanning each answer sheet twice, on different machines on different days). In addition, several others are already planned for implementation, such as making #2 pencils available to students at SAT test centers, and including "anchor points" on the answer sheets (an additional control which adds fixed points to answer sheets to help in the identification of paper expansion, contraction or misalignment). Some of the recommendations by their nature require further analysis by the College Board in cooperation with its vendors to ensure that the risks Booz Allen has cited are effectively managed and mitigated through appropriate process enhancements.
Read the full report: SAT Process Review—Answer Sheet Processing
Return to Press Release: College Board Releases SAT Answer Sheet Processing Report
