National Assessment Will Move Forward On Grade 12 College Preparedness Measures

I chaired a technical panel to recommend how to assess grade 12 state and national performance on college preparedness. Below is our summary from a report accepted by the National Assessment Governing Board on November 21. The recommended studies will help the Board set score scale ranges on NAEP reading and math that indicate college preparedness in the 2009 assessment. This college measure would supplement current achievement levels of basic, proficient, and advanced.



The Technical Panel on 12th Grade Preparedness Research, convened by the Governing Board, consists of 7 members with expertise in a variety of measurement and policy areas related to preparedness. The purpose of the Panel is to assist the National Assessment Governing Board in planning research and validity studies that will enable the National Assessment of Educational Progress (NAEP) to report on the preparedness of 12th graders for postsecondary education and job training after they graduate from high school. The first round of studies will be conducted before and during the 2009 NAEP 12th grade assessments of reading and mathematics, and the Board plans to begin this new type of reporting with these NAEP 2009 results, scheduled for release in 2010.

The Panel’s deliberative process engaged each Panel member’s expertise to refine ideas; gather supplementary materials; convene testing companies and partner organizations; and review the advantages and disadvantages of various sources of data. At each step in the process, the Technical Panel considered a range of alternatives and feasibility issues and then made choices to advance to the next point in deliberations.

KEY RECOMMENDATIONS
Use a variety of methodologies for NAEP preparedness studies in order to determine if mutually confirmatory evidence exists.
A multimethod approach is a sound and reasonable way to gain understanding of this complex set of issues and interrelationships. There is not a particular study that would comprehensively address the feasibility and validity issues for prospective NAEP preparedness reporting. As described above, the four recommended types of studies are:
 content alignment;
 statistical relationships with other assessments and postsecondary outcomes data;
 judgmental standard-setting; and
 national surveys.

Highlight the focus on reading and mathematics academic skills and avoid representing NAEP’s preparedness reporting as the single, authoritative definition of preparedness.
Several national conversations include capabilities beyond academics when addressing preparedness and readiness. A non-academic emphasis is not the function of NAEP, and it is important to clearly communicate the focus of NAEP preparedness research to avoid misrepresentation and overstatement.

Maximize the information produced from all studies.
In comparing NAEP with other assessment instruments used as indicators of preparedness, Panel members have noted there may be overlap and there may be non-overlap. The Panel sees equal importance in describing the characteristics of overlap and the characteristics of non-overlap. These sources of information should be used to provide context and rigor for NAEP preparedness research and reporting.

Be mindful of the evolving context of preparedness.
There has been a substantial increase in the development of policies and standards to promote preparedness of students transitioning from high school to postsecondary endeavors. The Panel recommends careful positioning with respect to this dynamic context. A contextual statement should be added to Report Cards to explain what NAEP can do and what NAEP cannot do in its reporting of 12th grade student preparedness. The statement should explain the definition that NAEP is using for preparedness. NAEP’s capabilities and definition of preparedness should be presented in the larger policy context.

Conduct preparedness validity research as an iterative process with additional studies for NAEP 2009 and beyond.
The Panel recommends additional studies be conducted to enable continued preparedness reporting beyond NAEP 2009. To build on the foundation set by the NAEP preparedness studies for NAEP 2009 Reading and Mathematics and to address the evolving national context of preparedness, the Panel has proposed specific additional studies for NAEP preparedness research. These studies represent an incremental approach, including study designs such as benchmarking studies to administer NAEP to groups of interest, studies with additional state databases, studies to examine additional occupations, and studies to develop composite college courses of the knowledge and skills needed to be prepared for entry.

RECOMMENDED STUDY DESIGNS
Content Alignment
The Technical Panel Recommends content alignment studies, as an essential step, to be conducted for each assessment used as an indicator for reporting preparedness on the 12th grade NAEP scale. In order to use other assessments as indicators of preparedness and capitalize on their preparedness research for interpretations of NAEP results related to preparedness, NAEP and the other assessments should measure similar content in a similar way. Content alignment studies will provide evidence of the extent to which the two assessments are aligned and provide a basis for interpreting the relationships of scores on the two assessments.
Statistical Relationships
The Panel recommends a series of studies aimed at statistically relating NAEP and performance on other assessments that serve as indicators of preparedness for higher education and for job training programs in the civilian and military sectors. The Technical Panel recommends that the strongest feasible form of linking should be used to establish statistical relationships between NAEP and the other assessments. It is important to note that the strongest form of linking, known as equating, will not be possible because equating involves relating scores between two tests built to the same specifications—same content, same difficulty, same reliability—which means that results are interchangeable. Because NAEP is a unique assessment with a different function and purpose, equating is not an option. Instead, statistical relationships, such as concordance or the use of equipercentile methods to establish a working relationship will be most likely between NAEP and other assessments.
Judgmental Standard Setting Studies with Subject Matter Expert Panels
The Technical Panel recommends studies involving judgments by subject matter experts (SMEs) for each type of postsecondary activity, relative to pre-existing sets of academic performance standards (or knowledge, skills, and abilities statements). For some studies, the performance standards would need to be developed if a particular occupation, for example, did not have an appropriate set available for SME panels’ use.
Survey of Postsecondary Education and Job Training Institutions
The Technical Panel recommends a survey to collect data from a nationally representative sample of two- and four-year postsecondary education institutions. The survey would collect information about the assessments used for course placement and the cut score(s) on widely used standardized tests for placement into college credit coursework, placement into remedial programs in reading and mathematics, and exemption from placement tests. The survey results will yield descriptive information related to results from other studies and provide a context for reporting NAEP preparedness research.

RECOMMENDED STUDIES FOR NAEP 12TH GRADE PREPAREDNESS REPORTING
HIGH PRIORITY
Content Alignment Studies for NAEP and Assessments of Postsecondary Preparedness
 College admissions and placement examinations (ACT, SAT, ACCUPLACER, COMPASS)
 Workplace eligibility and placement examinations (WorkKeys and ASVAB)
Statistical Relationship Studies for NAEP and Other Assessments of Postsecondary Preparedness
 Linking national NAEP scores with preparedness indicator scores from other assessments
 Linking 12th grade state NAEP samples with state longitudinal databases (Score data for college admission and course placement; transcript data; and workplace salary data)
Judgmental Studies to Set NAEP Cut Scores for Workplace Preparedness (Military and Civilian)
 Identification of 5 – 7 target jobs across all sectors
 Identification and development of eligibility criteria for target job training programs
 Set NAEP reading and mathematics job training program cut scores
National Survey of College Course Placement Assessments and Cut Scores

MEDIUM PRIORITY
Judgmental Studies to Set NAEP Cut Scores for College Preparedness
 Set NAEP reading and mathematics college preparedness cut scores using:
 ACT College Readiness Standards
 College Board Standards for College Success
 Standards Developed by subject matter experts for college course placement

Labels: College Testing