College Data on Student Preparation and Success Is Getting Worse

Dennis Jones of the National Center for Higher Education Management Systems wrote this analysis of college data weaknesses as part of the report Measuring Up: 2008 by the National Center for Higher Education and Public Policy
The Information Gap: Much Talk, Little Progress
By Dennis P. Jones


Over the past decade, states have used Measuring Up to evaluate and compare their performance in higher education. Policymakers and the public have tracked their state's progress and setbacks in preparing students for education beyond high school, enrolling them in college, trying to keep college affordable, and conferring degrees. During this time, one trend has held constant: not all the information needed by policymakers is available to them.


When first published in 2000, Measuring Up identified the key areas where comparative, objective information was not available across states. Most of the deficiencies noted at that time persist today (see table). In fact, in many areas there is less information available now. In some cases, states have not participated in national assessments that would have provided important state-level data; in other cases, national groups have not collected sufficient data from each of the states. The result is a failing grade-an F-for the nation's performance in developing data resources for state-by-state comparisons in higher education.


Preparation
There has been some improvement in assessing how well states prepare students for college. The Census Bureau's new American Communities Survey (ACS) now provides more timely and accurate data about high school completion. However, this improvement does not affect two important areas: advanced course taking and student achievement.


Advanced K-12 Course Taking. Enrollment levels in advanced courses can help to indicate preparedness for college. Since 2000, substantially fewer states participate in national surveys that indicate how many eighth graders take algebra and how many high school students enroll in advanced math and science.


Student Achievement in the 12th Grade. Most states-but not all-continue to participate in the National Assessment of Educational Progress (NAEP) for eighth graders. For high school seniors, there is a comparable national assessment but it does not provide data at the state level. Information about the "stock of learning" that students acquire in high school and carry into college continues to be missing in the states.


Participation
There has been no progress in assessing the extent to which states provide opportunities for residents to enroll in higher education.


College Enrollment Rates for Recent High School Graduates, by Income. At the national level, rates of college enrollment are available by racial group and by income. At the state level, these rates are available by racial group, but not by income. Data about student financial aid packages for college freshmen have improved, but nothing is known at the state level about the family incomes of students who do not apply for (or receive) such aid. Given the changing demographics of college students, information about the family incomes of college-eligible individuals and those who actually enroll is crucial for effective state policymaking. Its absence represents one of the most notable of all the information gaps.


Migration of Students Across States. Information about the state of origin of college freshmen continues to be available. As a result, state-to-state migration of entering students can be determined. Once students enroll, however, federal data collection does not offer a way to track their progress or geographic location. The National Student Clearinghouse (NSC) could be used for this purpose if the protocols for use could be agreed upon nationally. Matching records from multiple state-level record systems has proven possible but arduous. At a time when workforce development is particularly important to state policymakers, the inability to assess migration patterns beyond the freshman year represents a severe handicap.


Affordability
There has been some progress in tracking the affordability of higher education for students and families, but this progress has not gone nearly far enough.


Unmet Financial Need for Eligible and Qualified Students. The available data estimate unmet financial need on a national basis, but not at the state level. As a consequence, there is still no state-by-state assessment of the extent to which financial factors affect college participation.


Distribution of Student Aid. Since 2000, some progress has been made in calculating financial aid patterns, though the improvements are far from adequate. Data on the amounts of different kinds of aid distributed to freshmen is now available by campus. Still missing, however, are data about the economic circumstances of aid recipients and the extent to which aid packages change as students advance in their college careers. For example, do loans supplant grants after the freshman year in some states more than others? An oversample of 12 states by the National Postsecondary Student Aid Survey in 2004 provided this kind of in-depth information. Until this information is available for all 50 states, however, policymakers will not be able to have a clear picture of college affordability.


Undergraduate Student Loans. In 2000, data about borrowing by graduate and undergraduate students were combined, making it impossible to determine levels of undergraduate borrowing. This problem has been remedied-one of the few areas of clear progress.


Completion
Problems remain in assessing whether students are completing their educational programs in a timely manner.


Progression of Individual Students Across Systems and States. Since many students transfer among colleges, it is important to track students across institutions. Many states have data systems that allow such tracking across public institutions in-state, but not across state lines. Data from the National Student Clearinghouse have been analyzed through a pilot effort. While this resource has limitations, it has proven capable of yielding good information for most states. Not all institutions participate, although a majority in most states do. Key data elements have not been available, such as whether a student is enrolled for the first time in college. Since protocols have not been agreed upon nationally to continue the pilot analysis, it must be concluded that no lasting progress has been made in this area.


Degree Completion in Six and Ten Years. Unlike in 2000, all institutions of higher education now report information on the proportion of full-time, first-time students who complete their programs within 150% of program length (six years for bachelor's degrees). Completion rates are also provided for students after four and five years. This is clearly an improvement, but there are still major shortcomings. Six years is too short a time period for many students, particularly working adults. The data cannot track students who transfer between institutions, both in-state and out-of-state. And the data are particularly flawed for community colleges because they fail to account for students who start part-time (the majority of enrollments at many community colleges) and students who transfer to four-year institutions. This is an area where most of the data are available in many states, but not in a way that allows national comparisons. In sum, progress has been made but remains inadequate.


Benefits
There has been some improvement in tracking the benefits that accrue to states as a result of having an educated population.


Educational Attainment. Two improvements have occurred in assessing whether state residents have a bachelor's degree. First, the U.S. Census Bureau's American Community Survey (ACS) now provides much more accurate data about the educational attainment of adults. Secondly, it is now possible to calculate the percentage of college degree holders who were born in the state in which they are living. This provides a basis for comparing states in developing home-grown talent.


Civic Engagement. New information about volunteerism is now available, including comparisons of volunteerism for college graduates and for those without college degrees. Although these data have rather large sampling errors at the state level, some progress has been made.


Learning
As in 2000, there are still no common benchmarks that would permit state comparisons of the knowledge and skills of college students. There are isolated instances in which learning outcomes are assessed, such as South Dakota's mandatory exam of rising college juniors. There are assessments that cover portions of the population, such as Graduate Record Examinations (GREs), which test those pursuing graduate study. And there are assessments in selected fields, such as licensure exams in nursing or WorkKeys in selected vocational fields. But there is no nationwide approach to assessing learning that would allow state-to-state comparisons. What energy was available for state assessments in 2000 has been directed to campus-level assessments in 2008, such as the Voluntary System of Accountability. This represents a step backward, not forward.


Adult Skill Levels. In assessing adult skills in the states, there has also been a large step backward. In 1992, the National Assessment of Adult Literacy (NAAL) provided a sufficient survey base to estimate the mastery of higher-level skills among the adult populations of most states. That assessment was re-administered in 2003. In 1992, 13 states participated in an oversample; in 2003, only six states did so. And almost five years later, the data have not been released for secondary analysis. National results indicate lower literacy levels for adults in 2003, but data are unavailable for all but a limited number of states. If states are to improve workforce preparedness, it is crucial that policymakers have access to information about the skill levels of state residents.


Cost Effectiveness
Over the past decade, there has been little progress in assessing state performance in higher education relative to the resources committed to the endeavor. An approach to calculating cost effectiveness was developed by the National Center for Higher Education Management Systems (NCHEMS). However, until learning outcomes are available by state, calculating the cost effectiveness of higher education will continue to rely on proxy measures that leave much to be desired.


Conclusion
State leaders and the public need access to objective information to assess and improve higher education. No single entity is at fault for the absence of information about one of the most critical problems facing the nation today; there is plenty of blame to go around. In some areas the states-in others the nation-must provide leadership in developing the data resources for state-by-state analysis. It is time for every state-and the nation-to commit to getting the information needed to advance the educational attainment of the citizenry, and to halt the worrisome slide of the United States vis-à-vis other developed nations in this area.


http://measuringup2008.highereducation.org/commentary/jones.p

Labels: academic preparation, College Completion, data systems