Outcomes Assessment

01 November 2009 | Uncategorized | Frank Vozzo

Sage brought two outside experts on outcomes assessment to the Albany campus on October 30.  An all-day workshop was presented to a group of faculty and administrators, including department chairs and deans.  The main purpose of the workshop was to prepare an action plan to improve student learning outcomes assessment at non-accredited programs at Sage in advance of our Middle States self study and site visit.

The facilitators were:

Dr. Donna Fish, on the faculty at Brown University and former Dean of Liberal Arts at Excelsior College.
Dr. Sean McKitrick, Assistant Provost at the University at Binghamton for Curriculum, Instruction and Assessment.

Workshop attendees included: Frank Vozzo, Kevin Stoner, Daniel Lewicki, Sharon Robinson, Terry Weiner, Michelle Napierski-Prancl, Mary Rea, Bobbi Gabrenya, Tom Sweeney, Dan Robeson, David Salomon, Jean Poppei, Penny Perkins, Stacie Kutz, Carol DiMambro, Jeffrey Soleau, Betty Fryer, Brian Robinson, Tina Mancuso, Leigh Davies, Jayne Boisvert, Deb Lawrence, Michael Musial, John Heimke, Tonya MacArthur, and Lisa Brainard.

The Department Chairs met for a little over one hour at the beginning of the workshop.  The group discussed a number of challenges including how to involve adjuncts in our assessment plans, what to do with transfer students, how to develop valid and reliable assessments, and how best to use technology such as electronic portfolios to assess programs and general education.   

When the larger group convened, the facilitators presented the results of an online survey of the workshop participants.  45% self-identified as novices, 45% as intermediate level, and only 10% as experts in assessment.  Another theme emerged from the survey: participants want to know how assessment can be meaningful to them.  In preparing for the workshop, participants were asked to rate their own perceptions about Sage in four areas: 1) degree to which student learning outcomes are defined and lend themselves to student learning; 2) degree to which assessments address student learning objectives; 3) degree to which faculty meaningfully discuss students’ achievement of learning outcomes and make recommendations to act; 4) degree to which discussed actions are implemented in areas such as instruction, curriculum, student learning (departmental) objectives, etc.

Points made during the ensuing discussion: getting department members together is difficult, getting them to share information about their courses so as to incorporate program outcomes across courses is even harder, and deciding what to do and setting aside the time to do it often prevents us from even beginning.  We need to start small, but start somewhere!

Here are some ideas either suggested to us directly by the facilitators, or provided by the participants and endorsed by the facilitators:

1. Spend two hours per year in department retreats having meaningful discussions about student learning, and make plans for improvement (this is the essence of “closing the loop” on student learning outcomes assessment).  Each faculty member should come prepared with one concrete idea for spurring discussion and making progress on assessment.  Document these discussions, the resulting changes, and the effects of those changes.  Don’t try to change everything at once: start small and keep up efforts.
2. Improve Office of Institutional Research tracking of graduates and their employers.
3. Develop a campus-wide plan for implementation of Mahara, including more training for faculty. Departments should decide what kind of data to collect and for what purposes, including what is needed simultaneously for both program and general education outcomes (“building a funnel”).  Discuss with students what they can accomplish for their own benefit with electronic portfolios.
4. Use capstone courses in the major more purposefully.  Build backward: figure out what students need as preparation– identify formative and summative assessment measures at critical points early in the curriculum; decide what we want students to achieve in the capstone, and what evidence will be used to demonstrate achievement.
5. Check the content of syllabi at the department level on a regular basis.  Get faculty to include the department’s core competencies in their syllabi.
6. Use internship evaluations to assess program and general education outcomes (e.g. communication skills).  Ask field supervisors open ended questions such as “what are the intern’s strengths and weaknesses in the area of…?”  Aggregate the data.
7. Allow transfer students to self-assess their readiness for upper level work in our programs.  Develop an inventory of “competencies” that transfer students are expected to have and the implications for movement through the curriculum towards the degree.
8. Partner with more-advanced departments to develop meaningful assessment surveys.  When surveying large groups, coordinate across units in designing the questions (make it possible to demonstrate value added: track aspects of personal growth from admission, through undergraduate years, to just after graduation, to alumni status five or more years out).

ACTION PLAN

The workshop concluded with the writing of an action plan.  The first item in the plan is for personal reflection on three levels:

• What are three things I can do in the next three months at the course or department level to advance assessment of student learning and preparation for Middle States?
• What are two things I can do in the next three months to partner with others across campus to meet institutional goals that reflect our mission and strategic plan?
• What is one thing I need immediate help with (or resources) to propel myself forward in addressing assessment vulnerabilities; and where can I offer assistance to help others in identifying and addressing their vulnerabilities?

All participants were asked to complete this reflection individually or in groups.  This serves as a personal development plan for the near term.  Next, departments have been charged with holding departmental retreats (see idea #1 above) by the middle of January 2010 to identify assessment strengths and weaknesses and come up with plans for addressing the weaknesses (take minutes and post them electronically).  Finally, the institution has been charged with developing by early next year a list of expectations in assessment that will apply to all faculty, and with making resources available that will allow them to meet the expectations (for example: each faculty member will attend x number of in-house or external workshops or conferences related to assessment over y number of years).

---

I will assist anyone with the reflection process, and with developing and implementing their plans.  I can also provide in-house workshops to individuals and groups in any of the following areas (I’m open to others):

• How to incorporate standardized tests into program-level assessment (ETS Major Field Exams, GRE, etc.).
• Designing valid, reliable, and useful surveys, administering them electronically, documenting and using the results.
• Writing and revising program level mission, goals, and objectives.
• Embedding assessment activities into coursework.
• Writing and using grading rubrics so that students can self-assess.
• Writing and revising an effective course syllabus.
• Making connections between the major and general education.
• How to harvest and use (electronic) SOS returns for course improvement.
• Writing and revising departmental academic plans.
• Documenting the connection between assessment data, assessment results, and curricular revision.
• Surveying assessment expectations for majors that carry accreditation.
• Assessing the effectiveness of instructional technology.
• Department retreats: making and documenting decisions that affect student learning.