Category 7 – Measuring Effectiveness


7C1 Information Collecting, Storing, and Accessing

ECC’s primary centralized information system is Datatel Colleague®, a modular collegiate
administrative suite. It is used throughout the College for student, financial and human resources
record keeping. Users can interact with the data securely through a desktop Windows program,
and to a limited extent, remotely, via the internet. Access to the data varies according to the
nature of the data; e.g., anybody can view course offerings and schedules on our website, but
very few can view or edit employee salary information.

Tied to Colleague® is the document imaging and management software ImageNow® from
Perceptive Software, which is used by Registration, Financial Aid and the Business Office to
conveniently view digital versions of students’ paper documents.

Decentralized software systems are used by various departments to meet their particular work
needs. Some examples of these are the following:

  • Blackbaud® Fundraising Software, used by the ECC Foundation for accounting, event
    management and fundraising information.
  • TutorTrac™, used by the Learning Center to monitor student use of their facilities.
    AIM (Academic Improvement Management) a home-grown system used by faculty and
    staff as an early alert referral system.
  • Track-It!®, used by the Information Technology (IT) department to manage calls to the
    help desk.

ECC also collects data from surveys (Figure 7.1), both electronic and paper. Feedback from
these surveys is used to improve teaching and learning, course management, student satisfaction,
and helps drive institutional planning.

Figure 7.1

Survey Instrument
Purpose
Graduating Student Questionnaire
Assess the satisfaction of graduating students.
Counseling and Advising Survey
Assess student satisfaction with their academic advisor and counseling services.
Student Evaluation of Faculty
Evaluation given at the conclusion of each course to assess the satisfaction of instructor and course.
Non-Returner Survey
Sent to students who did not re-enroll to assess what factors led to their departure.
Career and Technical Graduate Survey
Administered to recent vocational graduates six months (180 days) following graduation to determine their employment status.
Employee Development Survey
Assess the training needs and professional development needs of faculty and staff.
Technology Survey
Assess the technology needs for faculty and staff, as well as the satisfaction with technical equipment and staff at ECC.
New Employee Survey
Assess the effectiveness of the hiring process and interest in employee training/development, and support services. 
RTEC Survey
Survey required by Missouri Department of Higher Education (MDHE) to provide a comprehensive overview of postsecondary technical education.
Performance Indicators
Survey required by MDHE to assess recent degree recipients performance on nationally normed or locally developed assessments, state licensure or certification exams, and/or state/national level competitions. 
Student Success Survey
Assess the effectiveness and impact of the Student Success Course (now Foundation Seminar).
Community Opinion Survey
Survey conducted by The Warren Poll to assess the districts impression of the College and gauge their support for a proposed bond issue.  

Central to the College’s data management is the department of Institutional Research,
Assessment and Planning (IRAP). This office is responsible for the collection, interpretation and
dissemination of institutional and assessment data. It takes data from the aforementioned sources
to compile reports for various stakeholders, including students, faculty, administration, the state or
the federal government. Statewide information is housed and compiled by MDHE and is shared
to institutions and the public through the Statistical Summary of Missouri Higher Education (www.
dhe.mo.gov/statsum_0708.shtml)
.

Specific individuals or departments store other information. These areas are responsible for
collecting, storing, maintaining and disseminating this information as needed. Often, these
individuals or departments submit data requests to IRAP to enhance their research needs, foster
improvement efforts or fulfill external reporting requirements.

7C2 Key Institutional Measures for Tracking Effectiveness

East Central College uses a variety of methods to track effectiveness. Key internal measures
are set in the ECC Strategic Plan, which is currently being revised. Measures are addressed in
seven strategic areas: Student Support Services, Student Success, Academic Affairs, Community
Leadership and Collaboration, Institutional Environment, Facilities and Grounds, and Technology.
In addition to the Strategic Plan, the Assessment Committee reviews and monitors the use of
assessment outcomes and maintains the Assessment and Effectiveness Plan (details in 1P11).

In specific areas, the Statistical Summary of Missouri Higher Education provides comparative,
longitudinal data on measures of effectiveness. Information includes measures on student
preparation, retention and graduation, enrollment and demographics, student transfer, tuition,
finances, and financial aid. Some Missouri public four-year institutions provide feedback on ECC
transfers at their institution. Information on the success and retention of ECC transfers provides
valuable feedback on the effectiveness of a student’s ECC education. ECC also uses the IPEDS
Peer Analysis System to compare information concerning our students, faculty, staff and finances
as a measure for tracking effectiveness.

For program effectiveness, ECC uses the Perkins Performance Indicators for career and technical
programs and the Program Review Process for all degree and certificate programs as the primary
measures for program effectiveness. Many career and technical programs also use an advisory
board to assist in identifying key measures for tracking the effectiveness of their program.

7P1 Selecting and Using Information and Data

Primarily, information and data are gathered to support institutional objectives (Category
2) and identify improvement efforts (all categories) on an as-needed basis. Federal and state
reporting requirements drive data collection and dissemination. The IRAP office is responsible
for collecting, managing and disseminating institutional data to these reporting agencies. These
requests often come from the President or other upper at the President, the Deans or other staff.
Furthermore, several ad hoc requests to satisfy AQIP Projects, Title III needs and other departmental
or committee issues determine the remainder of data selection and use.

With the Strategic Plan under revision, the College will develop a foundation for selecting and
using information to support annual and long-term College goals and initiatives (Category 8).

Specific information related to student learning (Category 1) is primarily driven from each
academic department’s assessment plan. Each assessment plan addresses departmental
objectives, goals and measurements of student learning using data. Exit assessments also provide
valuable information regarding student learning. An important goal of the College is to allow
students to improve their quality of life by preparing them for immediate work in the community.
Career and technical students are given standardized WorkKeys tests upon graduation to measure
the effectiveness of their program, and industry-based tests to measure what they have learned
in their core discipline. As evidence of continuing success, the College gathers employment
information about its recent career and technical graduates, including numbers employed within
their discipline and average salary by discipline. This information is summarized in the Career and
Technical Graduate Follow up report Report and supplied to all staff involved in the vocational
programs. Students in non-vocational areas (AA degrees) take the standard CAAP test and
enrollment at transfer institutions is indicated in reports via the Missouri Department of Higher
Education. Additionally, the College receives details regarding student success at institutions such
as the University of Missouri – St. Louis and University of Missouri – Columbia.

7P2 Determining Unit Needs

Data and information collected by departments are primarily articulated through the program
review process, assessment plans, or other departmental objectives. Once a department or unit
justifies their need and use of the data, they are expected to use the information to further the
mission and objectives of that department and the institution. As the steward of ECC data and
information, IRAP tries to monitor and manage the usage of the information, striving to provide
reliable, accurate, consistent and timely information to its stakeholders. To address many of these
information needs, the ECC Factbook was created as a comprehensive collection of data related
to ECC. Information includes student demographics, student outcomes, developmental student
retention and success, student programs, course enrollment and other data.

7P3 Selecting Comparative Data

Determination of comparative data is the direct responsibility of the IRAP office. Since external
data and information are mandated by state and federal agencies, information and data for
these constituents are often a top priority. Comparative information made available from these
constituents is then provided to the key institutional decision makers. Institutional projects, often
requested from the President or other upper management staff, receive a higher priority than
other projects. Individuals, departments and other institutional projects that require comparative
information and data are then prioritized according to the need for usable and valuable feedback
for the institution.

ECC Utilizes Comparative Data in Three Distinct Areas:

  1. Comparison to other Missouri community colleges – Comparison information is
    collected by institutions annually through the MDHE and IPEDS. The State of Missouri
    compiles the data and information and the comparative results are presented in the
    Statistical Summary of Missouri Higher Education. Comparative information is presented
    for all community colleges in Missouri. ECC also uses the IPEDS Peer Analysis System
    to generate reports and view descriptive data regarding other community colleges
    in Missouri. Another source for comparative data is through the Missouri Community
    College Assessment and Research Consortium, which meets throughout the year to
    discuss and share information and best practices regarding data collection within
    Missouri community colleges.
  2. Comparison to other national community colleges – ECC recently subscribed to the
    National Community College Benchmark Project (NCCBP), which provides outcome
    and effectiveness benchmarks against a cohort of 178 community colleges throughout
    the nation. Comparative benchmarks include completion rates, retention and success
    rates, cost per credit hour comparisons and a variety of other areas. ECC also utilizes
    the IPEDS Peer Analysis System to compare our performance against a national set of
    institutions.
  3. Longitudinal Data – The ECC Factbook tracks trends in enrollment, FTE, retention,
    graduation and a variety of other key indicators and helps ECC benchmark performance
    against itself by looking at data over time. Longitudinal data also exists for budget and
    finance information. Trends in financial assist the Board, President and the Executive
    Dean of Finance and Administration to make informed decisions regarding the future
    budget of the institution.

7P4 Institutional Data Analysis

In the past, the College has lacked focus on measuring and sharing performance information
college-wide. Information and data were collected, reporting requirements were met and no
further action was taken. ECC recognized the need for a full-time Institutional Research office.
With the addition of this office ECC has placed a high priority on producing and sharing information
with the institution.

Under new leadership, the College is revising the Strategic Plan. The plan that was in place lacked key performance measures. The Director of Institutional Research, in conjunction with
the AQIP liaison, Deans and other campus Directors provided feedback that was used to
identify a series of measurable performance indicators. Although the plan is still under revision, measures have been identified and strategies have been linked to each of the nine AQIP Categories. Once the plan is finalized, progress toward the key performance measures will be shared with the institution annually.

7P5 Unit Data Analysis

Each department has been charged with the task of creating and adhering to its own assessment
plan. Each plan addresses the assessment of student learning (Category 1) and is aligned with the
objectives of the College-wide assessment plan. Periodically, the IRAP office conducts analyses
of success rates in sequential courses to ensure that the goals of each department are being met.
These are shared not only with the respective departments, but also at assessment sessions and
division meetings.

The College has also recently improved its Datatel interface for advising students, allowing staff
and faculty advisors to more efficiently access student information and advise them accordingly.
The AIM early alert system has also been implemented to help track students’ progress and
difficulties and to aid in providing counseling and other services.

Finally, ECC has continued to maintain a low student to faculty ratio to ensure that students
have the opportunity to be an active part of the class and to be successful. (See Figure 7.2)

Figure 7.2

2001
2002
2003
2004
2005
2006
2007
Student to Faculty Ratio
19:1
19:1
18:1
19:1
18:1
18:1
17:1

7P6, 7P7 Information Systems: Ensuring Effectiveness

In order to evaluate the effectiveness of the College’s infrastructure, the Board of Trustees
approved a proposal to conduct an IT Audit of the institution’s computing operations in 2005. During
the audit, several operating weaknesses were observed. To improve service delivery and alleviate
other operational weaknesses, the audit was suggested the College hire a Systems Administrator.
Other items included lack of IT staff training, poor password controls, insufficient hardware and
system security issues. Since the audit, a new Director of IT was hired and improvements have
been made, as detailed below.

The college has a dedicated department to maintain the effectiveness of its IT infrastructure.
This department has benefited from significant investments in hardware, software, staff and training
to help support the mission of ECC.

To keep in line with technology improvements, bug fixes and security patches, upgrades are
necessary. The College’s IT systems updates are driven by two primary agents, the vendor and IT
staff. The vendor drives upgrades to both the hardware and the software of the College’s most
critical database, Datatel Colleague® . The same is true of some of the decentralized systems like
Blackbaud® and TutorTracTM. It is then up to IT staff to implement these updates in a timely fashion.
The IT department has set its own goal of upgrading the servers, which run core services (e.g.
e-mail) on a three-year basis. The IT Systems Administrator ensures system stability and keeps the
software on the servers current.

The integrity and reliability of information is crucial. The loss of digitally stored information would
severely hinder the function of the College. To protect against such, the College has contracted
an off-site third party to store backup tapes of important College data. Also, the servers which run
Colleague® and core services are configured to store data across three or more storage devices
so as to protect against one failing, and they are kept in a locked, air-conditioned room.

To maintain the integrity of Colleague® data, the College has an in-house program that runs
periodically to look for predefined anomalies. Built into the Colleague® desktop interface are
validation codes that prevent certain fields from receiving wrong data; e.g., a student can only
have a gender of male or female. The IRAP office also checks for errors as part of its operations.

Confidentiality and security of information is important to the College. At the staff level, the
College mandates all passwords to Colleague® be changed every 90 days. If user access to student
data is required, new employees are made aware of the College’s policy regarding release of
information. They must complete a form which affirms their acceptance of the policy, and notifies
IT what data categories they need access to. This complies with the “Family Educational Rights
and Privacy Act of 1974” (FERPA). At the network level, the College has a dedicated service
that looks for suspicious network behavior. Also, an IT staff member is part of the Research and
Education Networking Information Sharing and Analysis Center (REN-ISAC), a body dedicated to
improving network security at the nation’s higher education establishments, as encouraged by
the Presidential Decision Directive (Clinton PDD 63: Protecting America’s Critical Infrastructures).

Finally, to protect the effectiveness of our IT infrastructure, the College enforces an Information
Technology Policy which governs faculty, staff and student use of technology resources to
“purposes related to the College’s mission of education and public services within the context
and traditions of academic freedom.”

7R1 Results

In general, the overall confidence, accuracy, reliability and satisfaction with information and
data collection at the institution have improved. This is a result of several things:

  • Creating a full-time institutional research office that uses time and resources to improve
    the accuracy, reliability and transparency of the College’s data and information.
  • Through the College’s in-house program that checks for data abnormalities, the College
    has been able to improve the quality of information produced and presented to
    stakeholders.
  • Increased awareness by staff and faculty of the College’s data and the scope and
    magnitude of its use and capabilities.
  • The College’s commitment to continuous quality improvement has engaged the College
    in conversation on how data and information can be used to further accomplish the
    goals and mission of the institution.
  • Revising the Strategic Plan has placed focus on producing and evaluating a set of key
    performance measures to ensure the College is focusing on its mission and performing at
    standards that are comparable or better than peer institutions.

To increase the scope, usability and accessibility of information presented on the East Central
College website, the College recently engaged in a comprehensive study of its existing website.
The study, conducted by the University of Missouri Information Experience Laboratory (IE Lab), was
intended to redesign the College’s website with more focus on prospective students’ needs.

Focus groups and participant studies were conducted in fall 2007. Participant groups were
divided into traditional students, non-traditional students, employees, parents and high school
students. Findings from the study indicate that all users had:

  • Mixed feelings about the effectiveness and usability of existing website
  • Problems locating contact information
  • Perceived the website as too cluttered
  • Navigation issues

Based on the study the IE Lab has made recommendations to improve the website architecture,
content, aesthetics and navigation. The College has considered these recommendations and
plans to hire an outside firm to implement the suggested changes in 2009.

7R2 How Results Compare

The College has no comparable results regarding evidence that its system for measuring
effectiveness meets the institutions mission and goals.

7I1 Improving Current Processes for Measuring Effectiveness

As the institution’s involvement in AQIP and the creation of the Systems Portfolio grows, it is apparent
that information and data regarding the performance and effectiveness of the institution must be
monitored and shared. Enhancements to the IRAP website have improved the availability and
accessibility of this information to its stakeholders. The College must also assess and compare the results of other higher education institutions. Realization of such needs led to the creation of the
College’s first Peer Comparison Study.

Additionally, the concept of using information and data to support learning, effectiveness and
planning is becoming a part of the institutional culture. To be an effective institution, we must
have an effective plan. The creation of key performance indicators is an integral part of the
revised Strategic Plan.

7I2 How the College Chooses Improvements

The first target for improvement is to finalize the Strategic Plan (See 7P4). ECC must also actively
seek out institutional benchmarks regarding its performance. Participation in the National
Community Benchmark Project (NCCBP) will provide a framework for developing future tactical
and operation goals.

 
   
   
     
East Central College Home | Quality Improvement Portal Home
 
 
©2008 East Central College |1964 Prairie Dell Rd., Union, MO 63084 | Phone: (636) 583-5193