Changes in the modern landscape of standardized testing are reflected in a newly revised edition of The Standards for Educational and Psychological Testing, released last week in Washington, DC. The Standards are jointly published by the American Educational Research Association, the American Psychological Association (APA), and the National Council on Measurement in Education (NCME). These standards have been published since 1966 and are regularly revised to maintain their “gold standard” status for psychological, educational, and occupational testing. Changes in The Standards are of interest to NACAC members since they contain the most current best-practices for implementing and interpreting educational tests. NACAC's Statement of Principals of Good Practice encourages members and other professionals who interpret standardized test scores to be familiar with the most current research on the fair and ethical use of these instruments.
The major changes for the 2014 version of The Standards include increased attention to the fairness of tests and the ethical use of scores. Content about the growth of technology, proprietary testing, and scoring has also been expanded to reflect the increased use of proprietary tests in schools and workplaces.
Previous editions of The Standards have commented on the importance of fairness in standardized testing. Since the last edition was published in 1999, the use of educational testing has become more extensive with tests now affecting educational and career trajectories for students and teachers like never before. With greater participation in standardized testing, The Standards now account for the new diversity of the test taking population. Sections of The Standards that have traditionally pertained to fairness are now consolidated into a single chapter in the “Foundations” section to highlight fairness’s importance in the era of widespread testing since the Joint Committee now feels that Fairness has equal importance with Validity and Reliability.
The Standards interprets fairness to mean that tests are sufficiently sensitive to test taker characteristics so that the test scores will yield valid interpretations for their intended uses. Ideally tests should have the same meaning for all individual test takers and would not advantage or disadvantage any populations in a systematic way. Ultimately, tests should be designed to allow test takers to demonstrate their standing on “construct(s) that the test is intended to measure.” The Standards enumerate how fairness in tests can be achieved by minimizing various “threats” in the areas of content, context, and opportunity. Concerns about fairness have also made their way into other areas such as “consequential validity.” With the advent of high-stakes testing in schools and workplaces, The Standards recommends that evidentiary requirements should always be proportional to the importance or consequence of the test. Overall, all modifications and accommodations to the tests should be made with an eye toward ensuring the comparability of scores. The Standards recognizes that achieving test fairness often goes beyond what is required by law.
The newest edition of The Standards also gives extensive consideration to the way technology now impacts standardized testing. The Joint Committee now recommends that the accessibility as well as security of testing and scoring be reviewed as more tests are administered via computer. The Standards also comments on the advent of automatic scoring algorithms. Such algorithms should be empirically and theoretically informed and have a basis for linking scoring rationales to the construct of interest. Lastly, proprietary test makers should be willing to disclose the principals behind their scoring algorithms or have them independently reviewed by experts.
The Standards provide a wealth of information for admission professionals who may consider standardized tests for admission decisions or for school professionals setting up assessment and program reviews with standardized tests. NACAC’s Statement of Principles of Good Practice recommends that members familiarize themselves and their fellow staff with current inter-association standards particularly with respect to educational testing, score use, and score differences between sub-groups. Accordingly, The Standards are an excellent resource for members who are looking for the most current information about the use, interpretation, and design of standardized educational tests.
More information about The Standards and a full copy can be found here.
Which is More Consequential: Field of Study or Institutional Selectivity
The Review of Higher Education
Yingyi Ma and Gokhan Savas
Yingyi Ma and Gokhan Savas have recently published an analysis that demonstrates the intersectional effects of gender, family background, major choice, and institutional selectivity on earnings. In an effort to understand the factors that contribute to the gender pay gap and ways of closing it, their analysis has also shed light on what factors contribute to early career earnings.
Field of study and institutional selectivity are considered to be the most important sources of income variation among college-educated workers. It is known that majors pertaining to science, technology, and business pay more than majors in education, humanities, or the social sciences. Graduation from a highly selective institution has also been associated with above average earnings and that low-income students are underrepresented at these schools. Ma and Savas have sought to understand the extent to which the premium attached to lucrative majors can offset the disadvantages associated with attending unselective colleges, being female, or hailing from a low-income background. Given that the price of tuition does not generally vary by course of study, Ma and Savas suggest that lucrative majors could be a way to ameliorate wage gaps for low-income students.
Ma and Savas found that participation in a field of study varies greatly by gender and class. The most important finding to emerge was that less privileged women benefit more from studying lucrative fields than privileged women who major in non-lucrative subjects but graduate from selective institutions. Other findings were less surprising. Low-income women studying non-lucrative majors at unselective colleges reported the lowest earnings. Men who studied lucrative fields at selective institutions earned the most. It was also found that a substantial proportion of the women who were in lucrative majors hailed from modest backgrounds. Conversely, privileged women were over represented in non-lucrative fields and reaped the lowest return from attending a selective institution.
Ma and Savas feel that the most significant public policy implication from their findings is that educational leaders need to encourage women of all backgrounds to major in gender atypical fields. Majoring in a gender atypical field can offset disadvantages from attending an unselective college or hailing from a modest background. In their words, college majors can serve as a “meritocratic mechanism” which can provide economic mobility for students. The authors note the importance of social connections when obtaining work. While lucrative majors can allow women from less privileged backgrounds to access high paying jobs, Ma and Savas stress that women of all backgrounds need to be given more support with career networking.
The study can be found here.
Revisiting Professional Learning Communities to Increase College Readiness:
The Importance of Pedagogical Content Knowledge.
Jennifer Bausmith and Carol Barry
Student centered “standards movements” such as the No Child Left Behind Act and the Common Core Standards have given way to an interest in teacher centric professional development. “Professional learning communities” (PLCs) represent an opportunity to provide structured professional development that utilizes the lessons and advice from the most senior teachers in the learning community. With the renewed interest in teacher expertise, Jennifer Bausmith and Carol Barry suggest that PLCs are an appropriate means to help teachers improve their understanding of the content they teach and how students learn.
Despite the advantages inherent to the PLCs, Jennifer Bausmith and Carol Barry have argued that PLCs as we currently know them will not be sufficient for helping teachers in light of the Common Core Standards. PLCs are underutilized in their capacity to increase pedagogical content knowledge and student outcomes. The authors note that professional development for teachers usually focuses on active learning, coherence, and collective participation. However, professional development materials regarding subject matter content are rarely found to be included in many PLCs.
For all the promise that PLCs offer, their provincial nature might allow them to spread knowledge that is incomplete or insular. Since it is difficult for schools to recognize that they may be under-resourced, PLCs may allow schools to unintentionally encourage “best practices” that have already been supplanted with more updated methods. Bausmith and Barry cite literature which has found that teachers who received substantial training by university-level educators have increased effectiveness as instructors. However, it would be difficult to increase the scale of this time intensive training to the national level.
Bausmith and Barry position PLCs as a venue through which a "library of lessons" could be widely disseminated among the teaching profession. They recommend the creation of externally developed and standardized professional development literature that is indexed to the Common Core State standards. A library of lessons would allow improvements in pedagogical knowledge to be brought to the national scale while also avoiding the chaotic situation where each district develops its own materials.
Overall, the development of current and research based instructional materials delivered through PLCs could help school systems align themselves with the Common Core Standards while also providing equitable access to quality teaching and college readiness. In an environment with budgetary constraints, directing funds to resources which have proven to be successful is a cost effective way to improve the quality of a school’s faculty.
The study can be found here.
College on Credit: A Multi-Level Analysis of Student Loan Debt.
Review of Higher Education
Nicholas Hillman, of the University of Wisconsin, has recently published a review of literature pertaining to loan default and conducted an analysis of factors that are strongly predictive of loan default. Overall, Hillman has found that “non-traditional” students are the most likely to default. His findings suggest that the federal loan system, despite its successes in promoting college access, may be rewarding existing privilege while also “sanctioning those who come from lower socioeconomic classes.”
The role of loans in facilitating college education has increased over the last thirty years. Subsequent reauthorizations of the Higher Education Act of 1965 have each coincided with a shift away from grant-based aid to loan-based aid. Half of all post-secondary students are now currently borrowing to facilitate their educations and the average college graduate is expected to accumulate around $26,000 in loan debt. By 2011, the amount of outstanding loan debt reached $867 billion, an amount that exceeds nearly all other forms of credit. In the last 10 years, the two-year post-baccalaureate default rate has doubled with 374,940 individuals currently in default.
Hillman’s study analyzed a sample that included 16,700 students obtained from the National Center For Education Statistics. He found that the leading predictor of default was attendance at a proprietary institution. His data showed that slightly over half of all those currently in default attended a for profit two-year college. Even when controlling for student level characteristics, such as socio-economic status and academic profiles, these students had 2-3 times the odds of defaulting compared with students who attended public two-and four-year institutions. Hillman found that students attending community colleges are no more likely than students attending four-year institutions to default after controlling for student characteristics. However, 19 percent of those currently in default had attended a public two-year institution. The relationship between overall debt burden and default was found to be non-linear. Most defaulters have relatively low balances and default rates do not increase as debt levels go up. Student family income was also a strong predictor of student default. Default rates decline steadily as student family income levels go up.
Hillman finds that minoritized[sic.] and non-traditional students are the most likely to default. White and upper income students and students’ without dependent care obligations are the least likely to default. Overall, the role that for-profit colleges play in contributing to student loan debt cannot be attributed to merely preexisting conditions. Hillman suggests that these findings could contribute to the policy development process because our current loan policy overly sanctions individuals who are already underserved by higher education.
The report can be found here.
College Initiatives Redefined
A Responsive Approach to College Counseling & Alumni Support
YES Prep Public Schools has released a report that details the strategies which have contributed to YES Prep’s considerable success in sending low-income and first generation students from the Houston area to college. YES Prep is a network of high performing public charter schools that was founded in 1998. It currently serves 8000 students, 84 percent of which are “economically disadvantaged” and 90 percent are first generation college students. Despite these challenging demographics, 44 percent of YES Prep alumni have earned a college degree. YES Prep alums have a six-year college graduation rate of 41 percent. These statistics are remarkable since the six-year graduation rate among other low income Houston students is only 8 percent. Given its success in enrolling and graduating economically disadvantaged students, YES Prep believes its methods can be successfully employed around the nation.
Through experimentation and research, YES Prep has developed five revised assumptions about guiding students to college. They have found that college access does not nearly equal college success, non-academic skills are an important factor in student success, college partnerships are key, and college affordability is still pivotal. Overall, YES Prep has created a variety of organizational, philosophical, and research based reforms inspired by these findings.
YES Prep currently devotes four percent of each school’s budget to college counseling. This has a resulted in a student-to-counselor ratio of only 40:1, or a 12 fold improvement over the national average of 471:1. In addition to counseling, the YES Prep network has worked to align its curriculum with its college counseling efforts. The College Assessment Portfolio Program Project allows students to measure and document their progress towards college readiness during their four years of high school. The YES Prep curriculum includes significant emphasis on the non-cognitive factors which have been documented to contribute to college and life success. The YES Prep network has also worked to formalize alumni support in order to provide students with an extensive range of mentors. Their success is consistent with the finding that mentored minority students are twice as likely to persist and have higher GPAs than non-mentored students.
The YES Prep network has looked outside its facilities to assist students during the transition from high school to college. Their network has been able to sign on 29 IMPACT partners which agree to meet 100 percent of admitted YES Prep alumni financial need and provide ongoing YES Prep programming. YES Prep has also worked to address college affordability by significantly expanding the amount of scholarship aid available to its students. These renewable awards are specifically designed to meet gaps in financial aid packages.
YES Prep believes that success in expanding access to college for the disadvantaged will come as the result of constant evaluation and experimentation. YES Prep cites data mining, reflection, and the continual adaptation of programs as key to finding real solutions to allowing more students to graduate from college.
text of the report can be found here.
Platters of cheese empanadas and fresh fruit greeted guests as they entered the Rotunda of the Ronald Reagan Building and International Trade Center to attend the University Innovation Alliance’s Announcement and Public Rollout event on Tuesday morning. But if the finger food was a notch above the usual, the content of the event cleared the bar with room to spare. Through well-moderated discussions and lively presentations, leaders from the UIA member institutions and their sponsors laid out plans to realize a vision of partnership that has been taking shape behind the scenes for the past fifteen months.
The University Innovation Alliance (UIA) is a group of eleven public universities committed to increasing the country’s number of graduates through collaboration and sharing of successful methods. The goals are to find scalable models of retaining and graduating students, particularly those who are underserved, underrepresented, and non-traditional. Alliance members are: Arizona State University, Georgia State University, Iowa State University, Michigan State University, Oregon State University, Purdue University, The Ohio State University, University of California at Riverside, University of Central Florida, University of Kansas, and the University of Texas at Austin. The Alliance has approximately $5.7 million in funding from the Ford Foundation, Gates Foundation, Kresge Foundation, Lumina Foundation, Markle Foundation, and USA Funds. The Alliance’s focus on these universities is an acknowledgment that, in order to meet college completion goals set by the Lumina Foundation and by the Obama Administration, public institutions of higher education will need to be the primary drivers of increased completion rates. There is no question that elite liberal arts colleges and Ivy League institutions are producing strong results, said Michael Crow (President, Arizona State University and Chair, University Innovation Alliance), but the problem with those models is that they are not scalable to impact the number of students who need to be reached if the country is to meet completion goals.
“Scale” is one of Mark Becker’s (President, Georgia State University and Vice Chair, University Innovation Alliance) four go-to words to describe UIA: Innovation, collaboration, translation, and scale. UIA member institutions will innovate on their own campuses and collaborate through a mentor/mentee relationship to learn from one another, and the projects they pursue will be able to be translated to the other member schools and scaled to affect large populations of students. Each member institution has a particular method it is using, successfully, to help attain the Alliance’s goals. Institutions that are leaders in one area will serve as mentors to the other institutions, who will work to adopt and adapt the strategies to their campuses. Some examples include:
- Georgia State University – GPS Advising system that uses predictive analytics to predict an individual student’s performance in any given class and any given major
- Oregon State University – E-Campus (Extended Campus) to reach 25-64 year olds with some college and no degree
- Iowa State University – Learning Communities of 25-40 students that bring students together around shared areas of interest (specific academic program, social concern, identity, etc.) – students share classes, residence hall, dining, etc.
- University of Texas at Austin – University Leadership Network financial intervention program that uses predictive analytics to identify the students at the very bottom of expected 4 year graduation rate and then provides them $5,000 scholarships each year, spread out in increments that are awarded as they meet certain academic goals.
- University of Central Florida – Direct Connect program that creates very easy transfer between FL community colleges and UCF
- Michigan State University – Campus neighborhoods created by dividing the 24 residence halls into 5 communities, each of which has access to academic advising, study spaces, a health facility, and financial counseling.
When pressed on whether the Alliance will be open to other institutions to join, Kim Wilcox (Chancellor, University of California at Riverside) demurred, saying, “This isn’t a club you join; it’s a commitment you make.” He elaborated on the UIA member institutions’ determination to make sure they are able to accomplish their goals; before broadening the Alliance, the first step will be to ensure that the collaboration is producing results. Hilary Pennington (Vice President, Education, Creativity and Free Expression, Ford Foundation) expects that even if no other institutions are added to the Alliance, the ideas and methods developed will spread organically throughout higher education as administrators, who work on UIA projects, talk with non-UIA colleagues and move from one institution to another.
University leaders expressed differing views on the role public policy should play in helping prod along college completion rates. Current policies, David Leonhardt (Managing Editor of The Upshot, The New York Times) acknowledged, stress enrollment, not completion--though this is changing as more states turn to performance based funding models. John Hitt (President, University of Central Florida) welcomes his state's performance based model, while Bernadette Gray-Little (Chancellor, University of Kansas) was more hesitant. Performance based funding is a good idea, she said, but has not necessarily been used "in a sensitive way." She argues that metrics must be adjusted to account for the students an institution serves; an admirable graduation rate at University A may be subpar for University B and truly unattainable at University C. Given budget realities, it is unlikely that colleges will lower tuition any time soon, but UIA members hope to help lower the cost of a degree by streamlining the process and accelerating the timeline on which students earn their credentials. Joseph Steinmentz (Executive Vice President and Provost, The Ohio State University) expressed a sentiment that seemed widely shared among UIA representatives when he said he feels "like much of the national commitment to have students go to college regardless of income has been lost." UIA member institutions are dedicated to reclaiming that lost commitment and taking a leading role in producing a high number of new college graduates.
None of the panelists or presenters feigned naiveté; all recognize that the work ahead will be challenging. Wilcox lamented that "the sense that some universities provide access and some provide excellence is a profoundly disappointing notion"--one that UIA hopes to counteract. While the universities have their work cut out for them, as Becker noted in his introductory remarks, "these are not insurmountable problems." Indeed, each of the UIA member institutions is already making progress alone. In the coming years, the universities will work together to continue expanding opportunity. Their first initiative? Predictive analytics, for which Georgia State University, Arizona State University, and the University of Texas at Austin will serve as mentors.
Last Thursday, the Department of Education announced the recipients of its 2014 Elementary and Secondary School Counseling Grants. The grantees represent forty school districts in twenty states, and award amounts range from $180,928 to $398,279. Grantees will use the funds to train and support counseling staff, as well as to hire more school counselors. While most of the projects awarded funding are geared exclusively toward elementary students, some will also target secondary schools. Mental health is a prominent concern in the project abstracts, with grantees hoping the increased counseling manpower and training will allow their schools to more effectively aid students, particularly those coping with external pressures, such as poverty and dysfunctional home lives.
Descriptions of the projects awarded funding are available here. Earlier this summer, Secretary of Education Arne Duncan urged the Chief State School Officers to maximize their use of counseling professionals, and issued a comprehensive list of federal resources and funds available to support school counselors.
NACAC has endorsed and submitted comments on the Senate HELP Committee’s draft of the Higher Education Affordability Act (HEAA), which would update and reauthorize the Higher Education Act. NACAC identified several provisions that may be of particular interest to members. You can read the full summary of these provisions here.
If the current draft of the HEAA were to be enacted:
- Colleges would not be able to use federal education funds for marketing and recruitment.
- Colleges would have to use a standardized financial aid award letter.
- Colleges would have to make certain their Net Price Calculators meet specific standards, and the Department of Education would develop a Universal Net Price Calculator, which would allow a student to generate net price estimates for multiple colleges simultaneously.
- Colleges would not be able to use incentive-based compensation and would have to annually notify employees of the policy.
- Colleges would not be allowed to enter into revenue sharing agreements with third party entities.
- Colleges would not be allowed to include pre-dispute arbitration agreements in enrollment contracts, and all previously-signed pre-dispute arbitration agreements would be voided.
- Colleges that pose a high student loan default risk would have to provide admitted students with a two week waiting period and full financial aid information before requiring them to sign enrollment agreements.
- Colleges would be subject to review by the Department of Education if they meet certain outcome (low graduation rate, high student loan default rate, etc.), financial (fiscal health, percent of revenue spent on marketing/recruitment/executive compensation, etc.), or consumer protection (high complaint rate, investigation or disciplinary action by an accrediting body or government agency, etc.) criteria. Additional criteria (high profit margin, percent of revenue derived from federal education funds, change in ownership, etc.) may trigger reviews of for-profit colleges.
- Colleges under review by the Department of Education (above) would have to post a notice of this review on their websites’ home pages.
- Several federal grant programs would be established, including one to incentivize states to keep tuition costs low and to enroll more low-income students and another to help certain immigrant students attend college.
While the Senate has adopted a comprehensive approach to reauthorizing the HEA, the House is approaching the task in a piecemeal fashion and has passed a few stand-alone bills. It is unlikely that both chambers will come to an accord soon. NACAC will keep membership updated on developments through the Bulletin, Admitted Blog, and monthly Legislative Updates (contact Michael Rose at firstname.lastname@example.org if you would like to receive these).
The Civil Rights Project of UCLA hosted a briefing on September 2, 2014, to introduce seven draft research papers examining the potential effect of the Obama administration's proposed Postsecondary Institutional Ratings System (PIRS) on minority students and minority-serving institutions (MSIs).
The idea behind PIRS is to measure which colleges are doing the best job at affordably and effectively educating their students. Eventually, federal aid may be linked to an institution's performance under the ratings system. PRIS is still in the development stages; the Department of Education has not yet issued specifics of how it intends to rate institutions. Even so, the possibility of a federal rating system tied to funding has drawn criticism from higher education groups that are worried that the system, when developed, will be unable to equitably capture institutional differences (NACAC submitted comments expressing concerns with the proposal in February).
The research discussed at Tuesday's briefing set aside general concerns with rating/ranking systems to instead focus on the practical implications of PIRS, particularly if federal funding is contingent upon institutional performance. Unsurprisingly, minority-serving institutions enroll higher percentages of high-need students, making federal aid all the more vital to their sustainability. At the same time, these institutions tend to not stack up as well against predominantly white institutions in metrics such as graduation rate. MSI advocates point to student demographics and lack of resources as primary causes of the disparity, and they stress that many MSIs outperform expectations when these factors are taken into account. One of the more provocative presentations came from Nicholas Hillman of the University of Wisconsin - Madison. Dr. Hillman's research exposes so-called "education deserts" -- communities in which there are few, if any, public institutions of higher education. These regions align with lower socioeconomic and minority demographics. Hillman concentrated on public institutions because the majority of students enroll in public sector institutions, and they tend to be lower cost to attend than private non-profit or proprietary options. What would happen to college access, Hillman asks, if the few public options available to students in education deserts lose funding under PIRS?
All of the presenters used their research to urge the Department of Education to carefully consider these aspects in developing and implementing PIRS. According to Deputy Undersecretary Jamienne Studley, who attended the briefing, the department is committed to doing just that, and she promised that the department would seek feedback and advice from researchers like those present to help make sure PIRS supports, rather than suffocates, success among minority and underrepresented students.
The papers discussed at the briefing are still in draft form and are undergoing the peer-review process. When finalized, they will be posted on the UCLA Civil Rights Project website.
This was one of the questions a new report from the National Center for Education Statistics sought to address by examining a cohort of first-time beginning undergraduate students who began their postsecondary study in the 2003-04 school year (BPS longitudinal study). The analysis found that about one-third of students in the cohort transferred or coenrolled at least once during a six-year period and of those who did transfer, approximately:
- 32 percent transferred all previously earned credits in the first transfer,
- 28 percent transferred some credits, and
Students who transferred all previously earned credits retained, on average, 24 credits, whereas students who lost credit, lost approximately 13 credits on average following their first transfer.
- 39 percent transferred no credits.
In addition, the report also looked at factors that consistently contributed to the likelihood of credit transfer. The two factors that stood out were:
- Academic performance prior to transfer (as measured by GPA)- “Specifically, higher GPAs were related to lower probabilities that zero credits will transfer and to a higher number of credits accepted.”
- Transfer direction (e.g., vertical, reverse, or horizontal)- “Reverse or horizontal transfer was related to higher probabilities that zero credits will transfer and a lower number of credits accepted at the destination institution… Overall, the findings suggest that when student transfer is aligned with how the higher education system is designed to accommodate credit transfer (e.g., from 2-year institution to 4-year institutions)… credit transfer is more likely to occur.”
Visit NACAC’s Transfer Knowledge Hub to learn more about the latest research related to transfer success. You can also find a wealth of resources such as journal articles, presentations, and other links related to transfer in the Knowledge Center.
Reference: Simone, S.A. (2014). Transferability of Postsecondary Credit Following Student Transfer or Coenrollment (NCES 2014-163). U.S. Department of Education. Washington, DC: National Center for Education Statistics. Retrieved from http://nces.ed.gov/pubsearch.