Limits of Tests for Assessing College Readiness

Limits of Tests for Assessing College Readiness
Can any test of high school students predict college success and indicate adequate college preparation?. Probably not if the only data used for prediction is a single test like ACT or SAT.
There are inherent problems with any single test as reported by the National Research Council’s Lessons Learned About Testing enumerates. Below are some relevant direct quotes from the NRC report:

There is measurement error related to the fact that the questions on a test are only a sample of all the knowledge and skills in the subject being tested – there will always be students who would have scored higher if a particular test version had included a different sample of questions that happened to hit on topics they knew well.

Other examples of factors that contribute to measurement error are students’ lucky guesses, physical condition or state of mind, motivation, and distractions during testing, as well as scoring errors. Therefore, a test score is not a perfect reflection of student achievement or learning.
One common problem is the tendency to use what are single, inexact measures to make very important decisions about individuals.

Testing professionals advise that when making high-stakes decisions it is important to use multiple indicators of a person’s competency, which enhances the overall validity (or defensibility) of the decisions based on the measurements. It also affords the test taker different modes of demonstrating performance.

High Stakes (1999) concludes that tests should be used for important decisions about individual students only after implementing changes in teaching and curriculum that ensure that students have been taught the material on which they will be tested.

A major rebuttal to Professor Ericksson’s contention that tests are the best predictor of college success is provided by UC studies of Grade Point Average at UC (UCGPA). These studies use student transcripts over several years to find whether high school Grade Point Average (HSGPA) or admissions tests used by IC (SAT I and SAT II) are the best predictor of UCGPA. These three studies listed below find the opposite of what Professor Ericksson contends.

1. Geiser with Studley (2002). This research, published in the peer-reviewed journal Educational Assessment, focuses on the relative predictive validity of the SAT I and SAT II examinations at the University of California (UC) for the years 1996-1999. The authors found that the SAT II tends to be a better predictor of first-year UC GPA than does the SAT I. Although the evidence is mixed, the research also demonstrates that HSGPA tends to perform better than SAT I and SAT II scores at predicting UC GPA. The research does not address the issue of the relative predictive validity of HSGPA versus the combination of SAT I and SAT II when both are used simultaneously to predict UC GPA.

2. Update of Geiser/Studley results (2004). The Geiser and Studley methodology was used to analyze data for 2000-2002 that have become available since the original study was conducted. In these analyses, HSGPA is an unambiguously better predictor of first-year UC GPA than is the SAT I or the SAT II. Furthermore, the updated analysis directly compares the predictive power of HSGPA with that of a combination of SAT I and SAT II scores, and finds that HSGPA is a better predictor of college completion and college success.

3. Burton and Ramist (2001). In a research report published by the College Board, these authors review more than a dozen predictive validity studies that were conducted on different data sets by various authors. In most of the studies, the high school record was a better predictor of college success than the SAT.

Labels: College Completion, College Readiness, College Success