Home » About MHCC » Research and Planning »  Research Studies and Publications » The Assessment Matrix: Communicating Assessment and Accountability Requirements to the Campus Community

The Assessment Matrix: Communicating Assessment and Accountability Requirements to the Campus Community

Paper prepared for presentation at the 8th Annual Summer Institute on Institutional Effectiveness and Student Success, The Consortium for Community College Development, Charleston, South Carolina, June 23-26, 1996.

Colleges and universities have been increasingly inundated by assessment and accountability requirements from numerous sources with inconsistent specifications. Often overlooked is how to effectively communicate assessment and accountability requirements to the larger campus community. This paper illustrates the effort of one college to address this problem by developing an assessment matrix that specifies major requirements and indicates overlapping and unique areas. Secondly, a calendar was created to help departments understand the sequence needed to complete various requirements in a timely manner. The paper describes each of the assessment requirements included in the matrix and the reactions of various campus constituencies.

Colleges and universities have been increasingly inundated by assessment and accountability requirements from numerous sources with inconsistent specifications. There are federal and state government requirements, regional and specialized accreditation standards, and professional guidelines, in addition to the systems that colleges have in place for internal program review. Especially in the area of assessing student learning gains and outcomes, there is an ever increasing mountain of literature describing research results, how to conduct assessments, and the problems and issues involved in assessment (Pascarella & Terenzini, 1991).

Often overlooked, however, is how to effectively communicate assessment and accountability requirements to the larger campus community. Many administrators and faculty lack an overall understanding of the various requirements and how best to respond. As a result, departments often adopt a piecemeal approach, responding separately to each requirement, thus making more work for themselves. This situation can easily lead to a sense of being overwhelmed, resulting in poor assessment practices or complete failure to conduct assessments. It is suggested that perhaps a role for institutional researchers is to assist colleges in building on previous assessments and incorporating these into subsequent reporting requirements.

The solution at this college was to develop a matrix that specifies the major assessment requirements and highlights areas of overlap and uniqueness. In addition, a calendar was created to help departments understand the sequence needed to complete various requirements in a timely manner. We describe each of the assessment requirements identified in the matrix and the problems associated with developing a coordinated response. The reaction of various campus constituencies is discussed with a tentative appraisal of this particular technique.

Although this study takes place at a community college, the requirements are common enough for any college or university to adapt. For example, almost all states now have performance reporting requirements related to student outcomes. Those using the model presented here would simply substitute the specific requirements from their state. The same reasoning holds true for accreditation; all regional accrediting bodies now include assessment of student learning gains and outcomes in their standards. We have also incorporated leading proposals for standardizing indicators of effectiveness such as being developed by the Joint Commission on Accountability Reporting (JCAR, 1995).

Finally, the paper discusses some of the inconsistencies in definitions and indices related to the assessment of student outcomes. The lack of standardization creates both a problem in communicating requirements to the campus community and creating an additional workload for those who develop information systems to assist in meeting the various reporting requirements. It is especially important that presidents and other lead administrators understand the inconsistencies in reported information in terms of both resource implications for maintaining variable indices and in communicating with external constituencies.

Table 1 presents the assessment matrix. The first column lists the various measures or indicators of effectiveness grouped among headings for student retention/completion/success, special populations/disadvantaged, and so on. The remaining columns describe the assessment purpose based on a specific internal or external accountability source.

Accreditation. Unique among the sources listed in Table 1, accreditation requirements do not mandate or specify measures. Guidelines do discuss areas of assessment such as retention, mid-program assessment, and capstone experience. However, colleges and universities are free to choose and design how those areas will be addressed in terms of measurement. "Standard V" refers to the guidelines prepared by the Northwest Association of Schools and Colleges (1994) dealing with the "educational program and its effectiveness."

Internal Program Review. The indicators listed here are derived from the college's "program/function review" process, which is designed to serve both program improvement efforts and resource allocation decisions (Goal #3 Task Force, 1994). Thus, some of the measures deal with cost

Perkins Quality Assurance. The indicators identified in this column are from the state plan (ODE, 1993) developed in response to requirements of the Carl D. Perkins Vocational and Applied Technology Education Act Amendments of 1990, Public Law 101- 392. The requirements cover all professional technical state approved programs. Programs must be reviewed annually using the indicators listed in Table 1, in addition to undergoing a comprehensive review every five years. In the comprehensive review, college staff examine what are referred to as "process variables" which include quality of instruction, work experience opportunities for students, and other qualitative issues that presumably would affect student outcomes.

State Key Effectiveness Indicators. These indicators were developed to respond to legislative accountability concerns (OCCS, 1992). The colleges and the state office have since designed and are implementing a statewide information system which includes monitoring these indicators.

Core Indicators. The Community College Roundtable (1994) discussed thirteen measures related to the mission of a comprehensive community college. It is important to note that Core Indicators are a first cut, not a final and exhaustive treatment of the subject. The authors anticipate evolution and further refinement based on practice in the field. Nonetheless, this document represents a starting point for the development of a common framework from which community colleges can demonstrate effectiveness.

Many of the indicators have been and continue to be used by colleges, especially in the area of institutional research. For example, placement rate in the workforce is commonly used in the evaluation of professional technical education programs. In this case, the indicator is also tied to federal reporting requirements as discussed with regard to the Perkins Act. Other indicators, such as demonstration of critical literacy and citizenship skills are employed by community colleges less frequently. Where such indicators are used, there may be a lack of documentation and/or evidence of the validity and reliability of the assessment instruments and methods. In general, the assessment of student learning is taking on more importance due to new and revised regional accreditation standards. Because of a relative lack of experience in this area, implementing formal assessment of student learning probably represents the most significant challenge to community colleges attempting to follow Core Indicators.

In addition to a discussion of each indicator within the context of the community college mission, there is a "technical description" section dealing with measurement strategies and data sources for each indicator. Again, this material is simply a starting point. Community college practitioners will need to conduct a systematic review of the indicators in relation to their current level of sophistication in assessment and evaluation, specific state accountability and regional accreditation requirements, and resources.

Joint Commission on Accountability Reporting. The latest entrant into setting standards for assessing institutional effectiveness is the Joint Commission on Accountability Reporting (JCAR, 1995). Involving the American Association of State Colleges and Universities, National Association of State Universities and Land-Grant Colleges, and American Association of Community Colleges, the goal of the commission is not only to identify key indicators but also to define very specific technical definitions so that the resulting information would be comparable across institutions. The commission's work represents an effort by the higher education community to take the lead in defining accountability requirements so as to prevent other groups, such as the federal government or the states, from mandating requirements. The difficulties in operationalizing the Student Right-to-Know Act has been a major influence in galvanizing action from the higher education community. However, it is too early to determine whether the work of the commission will have a long-term impact.

The schedule and calendar for assessment can be found in Figure 1. The schedule was created to help departments manage the timing of the activities needed to meet the assessment requirements. The calendar was also useful in depicting how the various assessment pieces fit together.

The first column in the calendar lists the major assessment pieces that need to be completed over a two year period. The major assessment pieces are further divided into smaller steps. Note that some of the same steps appear under more than one assessment piece. To the right of the first column, a calendar

Figure 2 goes here.
See file 'AIRPROJ.MPP'.

divided by month and year runs across the top of the schedule. Bars depicting the timeline for each assessment activity appear underneath the calendar heading.

The assessment activity that spans the longest time period is accreditation. All other assessment activities within the two year period were integrated to meet the accreditation requirements and timeline. The hope is that as the departments complete the steps of the other assessments, they will be compiling data, program information, and narrative reports that will serve as the major components of the accreditation self-study. With regard to the professional technical programs, the Perkins evaluation greatly overlaps with the internal program review requirements. The phase 1 data are the same as many of the Perkins outcome standards; phase 2 of the internal program review is based primarily on the Perkins process standards. The yearly area assessments that are part of the strategic planning process have also been incorporated into internal program review, and will be incorporated into the accreditation self-study. The ability to establish benchmarks and analyze trends over time is another benefit of planning for ongoing assessment activities.

A significant complication in responding to various accountability requirements is the inconsistencies prescribed or suggested in measurement strategies. This problem is particularly pronounced in the area of student tracking, as the controversy and difficulties surrounding attempts to implement the Student Right-to-Know Act have demonstrated.

Student retention and persistence rates are identified in accreditation guidelines, internal program review, by the Community College Roundtable (1994) and JCAR (1995). As is typical with accreditation guidelines, there are no prescribed procedures for measuring student retention. The other three are only consistent to the extent that student cohorts would include first-time, degree-seeking students. JCAR (1995) includes transfer students as a separate cohort while the other two are silent on transfer students. Of course, many community colleges do not identify transfer students at entry. Previous college experience is only relevant for students on financial aid or students applying for graduation and requesting transfer credits be applied toward their degree.

There is no consistency in the definition of timelines for reporting persistence rates which spills over into the problem of calculating a graduation rate. The Community College Roundtable (1994, p. 17) notes "that community college students take as long as seven or eight years to earn a degree, the proposed indicator has no 'endpoint' time limit." As a result, both the Community College Roundtable and JCAR simply recommend annual reporting of persistence and graduation rates. For Perkins reporting, many community colleges track and follow-up on both graduates and what are termed '"early leavers" or "completers." This would include students who have completed a substantial part of a program without a degree and have not returned to the college for a specified time period.

For accreditation, Perkins and internal program review persistence and graduation rates by program or field of study are required. However, beyond the generally accepted use of CIP (Classification of Instructional Programs) codes for categorizing students, the source material referenced in Table 1 does not address issues surrounding the calculation of persistence and graduation rates by student major. For example, how are students that change major dealt with in the tracking system? Even more fundamental is the problem of identifying a major for a particular student. Many colleges simply use student self-declared major, but such declarations may reflect student intent rather than actual enrollment. Dealing with this in a straightforward manner is complicated by various financial aid and Perkins requirements (e.g., students must be in an approved program).

In summary, it is extremely difficult for even a single institution to design a student tracking system that effectively responds to all the accountability requirements and guidelines. It is no wonder that attempts to develop comparability across institutions, as in the case of Student Right-to-Know, has become such a daunting task.

Prior to the implementation of the assessment matrix, frustration with assessment practices was widespread across campus. Lack of communication between institutional researchers, administrators, and faculty created a number of obstacles to meeting the assessment requirements.

One problem was that campus constituencies were unclear as to how the various assessment pieces fit together. Without a sense of how to integrate the assessment requirements to meet multiple purposes, they were overwhelmed.

Second, the departments did not know what was expected of them in meeting the assessment requirements. The main complaint was that the materials outlining the assessment tasks were too vague and confusing. Often the associate deans aborted their attempts to respond to assessment requirements as they quickly became dismayed by the lack of specific guidelines provided for the task at hand.

Third, the associate deans had little understanding of how to interpret data that were generated in the Office of Research and Planning and distributed for use in assessment activities. There were questions regarding how the various measures and standards were calculated and skepticism about whether the data were accurate. Due to the communication breakdown on both sides, the unfortunate outcome was that the associate deans did not know what to make of the data and the Office of Research and Planning was not made aware of errors and discrepancies in the data that could be corrected. As a consequence of this climate of confusion, college constituents were at best frustrated and uncooperative participants in assessment activities.

A final problem was that the lack of communication between the departments and the Office of Research and Planning led to further communication breakdown between the associate deans and program faculty. The associate deans, being frustrated and confused themselves, had a difficult time explaining and justifying the assessment practices to their faculty.

In response to this situation, the Office of Research and Planning redirected their efforts to focus on coordinating assessment activities so that they fit multiple purposes, and also communicating the integration of assessment requirements to campus constituents. In addition, the institutional researchers worked closely with the associate deans to develop specific step-by-step instructions for how to complete assessment activities, provide concrete examples based on program data, and create an activity calendar setting realistic timelines for completing the activities.

So far, the reactions of the associate deans and faculty have been promising. Although they are still reluctant to begin the assessment activities, at least now they know how to approach the various demands and complete them in a timely manner. Because of their involvement in developing the assessment approach, the associate deans have a much deeper understanding of the assumptions underlying the data and what the figures mean. Another outcome of the new approach is that the associate deans are more likely to work with each other, sharing the strategies and information they have found useful in responding to the assessment requirements. This approach to assessment demands has also fostered better communication between the Office of Research and Planning, the associate deans, and the faculty.

The activity calendar has helped campus constituents in two ways. First, it provides administrators and faculty with a way to pace their efforts and reduce the likelihood of missing windows of opportunity in which they can complete assessment activities. Second, it reiterates that there is overlap and coordination between the assessment components. In other words, it enables constituents to see the "big picture".

Although the new approach to integrating assessment on campus has yielded positive results, there are still further improvements that need to be made. Constituents still experience frustration about finding the time to meet the requirements. Perhaps they do not yet believe that cooperation with the present assessment demands saves time in the long run. In addition, the associate deans and faculty now have a better understanding of the figures, so they are also better able to question the credibility of and point out the inconsistencies in the data. The positive side of this is that they are better able to bring these problems to the attention of the Office of Research and Planning. In turn, the Office of Research and Planning needs to respond promptly to these concerns.

Given an institutional decision to implement any or all of the indicators discussed above, what are the major elements of an action plan? What are the resource requirements? Assuming the decision involves all of the indicators, what is the order of priorities? Can the same system established to address one set of indicators be used to meet the other requirements? These are the key questions college staff must ask when designing the information system to support an institutional effectiveness program.

The organizational structure of Core Indicators provides a logical basis for grouping the indicators. For example, there are three indicators under the "student progress" mission category: student goal attainment, persistence (fall to fall), and degree completion rates. All three of these measures would be derived from a student tracking system. There are numerous resources available to assist colleges in developing a student tracking system (Ewell, 1987; Palmer, 1990). At the same time, colleges must respond to the federal requirements contained in the Student Right-to-Know and Campus Security Act of 1990, the Perkins Act of 1990, and Higher Education Act amendments of 1992, all of which focus on student retention, graduation rates, and outcomes. Since there are no unresolved technical obstacles to implementing student tracking systems, the lack of one at any college today may simply reflect a lack of resources, or more importantly, a failure to give such a system high priority.

The major complication with student tracking systems, especially at community colleges, is the diversity of student populations. Most student tracking systems are premised on continuous attendance and degree-seeking students. However, practitioners know that these assumptions do not apply to the majority of community college students. As a consequence, a considerable body of literature has developed that advocates the incorporation of student intent in the analysis of retention patterns (Walleri, 1990; Walleri, Seybert & Cosgrove, 1992; Palmer, 1993).

Finally, many states have or are in the process of developing shared information systems which facilitate the collection of follow up information on employment and transfer (Walleri, 1990; JCAR, 1995). By matching on student SSN, colleges can exchange information on transfer among themselves as well as with other state agencies that collect information on employment, wages and so on.

Community College Roundtable (1994). Community colleges: Core indicators of effectiveness, AACC Special Reports No. 4. Washington, DC: American Association of Community Colleges.

Goal #3 Task Force (1994). Progress report on college goal #3, position/program review. Gresham, OR: Mt. Hood Community College.

JCAR (1995). Joint Commission on Accountability Reporting. Washington, D.C.: American Association of State Colleges and Universities.

Ewell, P. (1987). Principles of longitudinal enrollment analysis: Conducting retention and student flow studies. In J. Muffo and G. McLaughlin (Eds.), A primer on institutional research (pp. 1-19). Tallahassee, FL: Association for Institutional Research.

Northwest Association of Schools and Colleges (1994). Accreditation handbook. Seattle: Northwest Association of Schools and Colleges.

Office of Community College Services (1992). Commissioner's task force on accountability: Final report. Salem, OR: Office of Community College Services.

Oregon Department of Education (1993). Assuring the quality of professional technical education in Oregon. Salem, OR: Oregon Department of Education.

Palmer, J. (1993). Student outcomes data at the community college. Washington, DC: National Center for Academic Achievement and Transfer, Working Papers 4(1).

Palmer, J. (1990). Accountability through student tracking: A review of the literature. Washington, DC: American Association of Community Colleges.

Pascarella, E., and P. Terenzini (1991). How college affects students. San Francisco: Jossey-Bass.

Walleri, R. (1990). Tracking and follow-up for community college students: Institutional and statewide initiatives. Community/Junior College Quarterly of Research and Practice 14: 21-34.

Walleri, R., J. Seybert and J. Cosgrove (1992). What do students want? How student intentions affect institutional assessment. Community, Technical and Junior College Journal 62: 28-31.