The Problem
The problem in this example is common to many organizations: courses are being administered but there is not a clear process for gathering and analyzing data beyond a learner post-survey. In this case, the courses were part of a professional program at large university. A new curriculum had been introduced and was in the process of being evaluated and refined, but the only instrument being used to gather data was a post-course survey of students. Additionally, the process for assessing courses and making decisions about how to improve them was inefficient.
The new curriculum needed a new assessment process, one that got beyond student evaluations of learner experience and could account for key factors including student achievement, curricular alignment, and pedagogical practice. Equally as important, the process for gathering and analyzing data needed to be efficient and presented in a way that provided clear pathways for course refinement.
The Solution
I developed a comprehensive system for course evaluation that leveraged student evaluations including quantitive measures as well as text analysis, exam data, and a qualitative self-study completed by the course instructors. This portfolio of data was submitted to the curriculum committee for review and discussion after which a summary report was issued to instructors including a single page overview that included action steps for course refinement.
Tools
- Qualtrics – this was our primary tool for gathering survey data
- ExamSoft – all exams were given in exam soft which provided rich data on student performance as well as exam item performance
- SPSS – Was used for wrangling data and generating reports
- Excel Macros – were used for text mining, key words related to curriculum goals, as well as essential course features, qualitative responses were processed to flag and count instances of these terms
- Adobe Captivate – I used Adobe Captivate to create a mini course on how to access and read assessment data, so faculty could more effectively access and analyze assessment data.
Examples
Given the proprietary nature of the content, I can not provide specific examples here.
Lessons
As an organization we gained a lot of insight from developing this process. For starters, we succeeded in completing the entire process of course evaluation with all courses in the program–something which did not even happen with the previous procedure. The text analysis of qualitative response data provided rich data on the true nature of what students found sub-par and also allowed for the tracking of key curricular goals. It also illuminated a disconnect between student expectations and the goals of the new curriculum This disconnect eventually prompted a complete revision of new student orientation content to emphasize the why behind the new curriculum. Exam data analysis led to a larger movement within the college to learn more about exam data and how to craft better exams.
On a personal note, this project was a huge learning experience for me. In addition to refining some of my technical skills, it was very demanding on my soft skills. This project really required frequent communication with the students, faculty, and curriculum committee members. I also benefitted greatly from being able to consult with a psychometrician in an associated college. His ability to identify key data points and keep the most essential things front and center was extremely helpful.