Validity and Limitations of College Student Self-Report Data: New Directions for Institutional Research, Number 150
June 2011, Jossey-Bass
Volume editors Serge Herzog, director of institutional analysis at the University of Nevada, Reno, and Nicholas A. Bowman, postdoctoral research associate in the Center for Social Concerns at the University of Notre Dame, have assembled contributing authors who are leading scholars in the field of college student self-reports. Combined, the chapters draw on data from a mix of colleges and universities, capturing student growth at different stages of the undergraduate experience, and even beyond graduation.
This is the 150th volume of the Jossey-Bass quarterly report series New Directions for Institutional Research. Always timely and comprehensive, New Directions for Institutional Research provides planners and administrators in all types of academic institutions with guidelines in such areas as resource coordination, information analysis, program evaluation, and institutional management.
Serge Herzog, Nicholas A. Bowman
1. Examining Systematic Errors in Predictors of College
Student Self-Reported Gains 7
Nicholas A. Bowman
A potential concern with using self-reported gains as a proxy for longitudinal growth is that predictor variables might provide consistently different results depending on how the outcome is measured. This chapter explores that possibility using a longitudinal sample of 8,615 fi rst-year college students at forty-nine institutions; it includes selfreported gains and longitudinal measures of the same constructs. The analysis examines differences associated with student characteristics and institution type, and the chapter uses cognitive and social psychological theory to anchor the fi ndings.
2. Gauging Academic Growth of Bachelor Degree Recipients:
Longitudinal vs. Self-Reported Gains in General Education
Few institutional research practitioners enjoy access to longitudinal data that measure changes in student mastery of general education skills. This chapter shows to what degree student self-reported gains in an alumni survey correspond to objective longitudinal changes in scores on widely used high-stakes standardized tests. Controlling for student sociodemographic background and academic experience, the analyses explore the infl uence of postgraduate employment status, field of study, and interaction with faculty on perceived versus longitudinally measured learning gains.
3. Using College Students' Self-Reported Learning Outcomes in
Scholarly Research 41
Gary R. Pike
The debate over validity of student self-reported learning should be informed by established theory on the role of student personality and academic discipline, and how self-reported learning is infl uenced by these two factors, as argued in this chapter. Using Holland's personenvironment fi t theory of vocational and educational behavior, the author examines the association between four types of self-reported learning outcomes and corresponding academic disciplines.
4. The Tie That Binds: The Role of Self-Reported High School
Gains in Self-Reported College Gains 59
Tricia A. Seifert, Ashley M. Asel
This chapter analyzes data from the Research on the Iowa Student Experience study to determine the extent to which students' retrospective self-reported gains from high school are associated with college self-reported gains. Given previous research on self-reported gains, the study hypothesizes a significant link between these measures and explores whether that relationship varies between fi rst-year and senior students.
5. Measuring How College Affects Students: Social
Desirability and Other Potential Biases in College Student
Self-Reported Gains 73
Nicholas A. Bowman, Patrick L. Hill
Social desirability may be one of several potential sources of bias in student self-reports. The authors of this chapter weigh the infl uence of this and other factors that may compromise the validity of student survey responses, including satisfaction with college, narcissism, sense of personal growth, and self-esteem. The impact of these factors is measured for both fi rst-year and advanced students, taking into account retrospectively perceived gains attained prior to entering college.
6. The Validity of Student Engagement Survey Questions: Can
We Accurately Measure Academic Challenge? 87
Stephen R. Porter, Corey Rumann, Jason Pontius
Our understanding of whether students feel academically challenged typically hinges on what they report in popular surveys such as the National Survey of Student Engagement. To examine the accuracy of what students report, this study compares survey responses with actual assignments documented in course syllabi for the courses that students completed. Specifi cally, the authors determine whether self-reported assignments for the most recent semester are signifi cantly associated with assignments documented in the syllabi.
7. Clearing the AIR About the Use of Self-Reported Gains in
Institutional Research 99
Robert M. Gonyea, Angie Miller
This chapter clarifi es and conceptualizes student self-reported gains from the perspective of two researchers affi liated with the National Survey of Student Engagement. The authors present new fi ndings on the degree to which social desirability bias may (or may not) skew NSSE data. They furnish advice on proper use of self-report data, discuss efforts to improve the NSSE as a suitable instrument to gauge student impressions of their learning, and describe how that information is a valuable complement to other sources of data used by institutional researchers.
8. Reconciling (Seemingly) Discrepant Findings: Implications
for Practice and Future Research 113
Nicholas A. Bowman, Serge Herzog
Synthesizing results from the previous chapters, the authors discuss the various conditions examined in this volume that may infl uence the congruence between student self-reported and objective longitudinal measures of growth. Limitations and proper use of self-report data are discussed in the context of improving the practice of institutional research.