News
 
Herrup, MacSwan & Associates
 
About Us

Qualitative Research: Getting a Competitive Advantage by Asking the Customer

Presented by:

Harold Wingood
Associate Provost, Dean of Admissions & Finanical Aid
Clark University
Worceter, MA

Peter Herrup and Jack MacSwan
Herrup, MacSwan & Associates, LLC
Marketing for Higher Education
Concord, MA

Moderator:

Paul Marthers
Dean of Admission
Reed College
Portland, OR

NACAC 60th National Conference
October 1, 2004
Milwaukee, WI

Background

Clark University conducts quantitative research on its undergraduate student body through the Office of Institutional Research. By the time Clark students graduate they have been surveyed at least a half dozen times and, in fact, cannot participate in the graduation ceremony without completing a comprehensive questionnaire distributed in March of their senior year. These students may be among the most carefully scrutinized undergraduates in the United States.

Despite the volume and consistency of undergraduate research conducted at Clark there are several problems with the typical research models.

1. There is an institutional bias in the questions that is difficult to eliminate.
2. The quantitative model that is normally employed generates black or white responses with no shades of gray or nuance and the either-or/multiple choice format often leaves participants without an accurate response option.
3. The questions reflect the interest of the researchers, not the participants and confines response to the aspect of the topic being surveyed instead of allowing participants to steer the topic toward relevancy
4. The research generates institutional self-knowledge, but the outcome tends to lack sufficient gravity to compel change.

The Admissions Department, which has a substantial responsibility for institutional promotion and enrollment, wanted to assess student life and the Clark "product" as a way to enhance the University's marketability. The objective was tounderstand the undergraduate experience from the students' point of view so that University resources could be allocated for greatest impact. Clark intended to improve the "product" by asking the "customer".

The problem:

How to collect unbiased, thorough and actionable information that would be accepted by the administration and faculty?

The solution:

Two critical decisions were made that would enrich the survey outcome and encourage acceptance by all stakeholders:

  • Contract an outside agency for help with research management.
  • Employ a qualitative model to enrich the results.

A third-party expert, it was reasoned, delivered several advantages to the project, which were:

1. Expertise in the development and administration of the research.
2. A lack of preconceived notions about the University and an ability to interpret the results without coloration or political correctness.
3. A likelihood that survey participants would speak more candidly to outsiders than to University personnel
4. The probability that the results as reported by an external agent would validate existing assumptions or prior research, would carry greater impact than in-house research and would be more widely accepted by key faculty and administrators

The Student Life Research Project

Clark commissioned Herrup, MacSwan & Associates, a marketing firm with a specialization in higher education, to develop and manage the Student Life Research Project. Working together, the Dean of Admissions and HMA principals established the following parameters:

  • The information-gathering portion of the project would take place during the month of February to avoid conflict with other research projects scheduled for later in the semester, mid-term exams, spring break and other distractions
  • The final report on the survey would be completed and delivered within the following month.
  • 5% of the undergraduate enrollment was set as a statistically valid sample and the minimum target for participation.

The Dean of Admissions established the survey topics, a veritable kitchen-sink approach to understanding the undergraduate experience. Every aspect of student life was to be included:

  • Academics
  • Faculty & teaching
  • The learning environment
  • Residential life
  • Cultural & social life
  • Recreation
  • The Clark "community"
  • Support systems
  • Community engagement

The first challenge to the researchers was to cover all topics in a period of time that roughly corresponded to the attention span of the typical college student. The second was to get a minimum of 100 Clark students to participate. As an incentive, all participants were given the chance to win free travel vouchers on Southwest Airlines*. The entire undergraduate student body was invited to participate via e-mail from the Dean of Admissions to nullify after-the-fact complaints that non-participants were not given a fair shot at the travel prize. About 150 students volunteered to attend discussion groups and 125 actually participated, 7% of the undergraduate enrollment. Students were instructed to reserve a session by time and day via e-mail. Volunteers were then telephoned the day before their session as a reminder.

(* In HMA's experience, the response to an opportunity to win a high-value prize will exceed the response to a guarantee of a less valuable gift such as a t-shirt, coffee mug or free pizza.)

Methodology

Discussion groups were formed of 4 to 6 students each. Groups with fewer than 4 participants tend to be dominated by one outspoken individual or may be stifled by shyness if the few participants are overly introverted. A group with more than 6 can be difficult to control and may restrict participation from all survey subjects. The meetings were scheduled for 5 weekday mornings and 2 weekday afternoons over a 2-week period, 10 days in total. The objective was to capture participants between morning classes and around the lunch hour.

Participants were informed that there was considerable ground to cover in a short period of time. Therefore, the researchers would be aggressively managing the conversation and the clock. This would help assuage any hurt feelings during the session if someone felt they were cut off abruptly. The discussion groups started with a general question of no importance to the research, specifically, "why did you enroll at Clark?" This question was initiated to loosen the students up, to get them comfortable and talking. We could have asked what they think about cats or existentialism. Any question that could not reasonably be answered "yes" or "no" would get the conversation flowing.

Students would determine the direction of each question and the aspect of each topic that was at the top of their minds. Consequently, each topic was introduced in the broadest possible manner to avoid prejudice. The conversation was stimulated by prompts from the research managers only if the students seemed puzzled by the topic or stumped as to the direction their answers should take.

Example:

The question about faculty and teaching was posed as follows: "What do you think of your professors?" instead of the typical quantitative approach, which might have asked, "Do you like your professors?" ž Yes ž No ž Maybe ž Sometimes

The researchers were looking for a discussion about the quality of teaching, access to faculty outside of class, opportunities to interact with faculty on a social level and the quality of faculty advising, but would only prompt survey participants toward these sub-topics if they were struggling with the question in general or if these aspects of the topic failed to arise at all. All of the broad topics covered in the discussion groups were introduced in the same manner. "What's good and bad about the study-abroad program?" "How do internships work?" "How does the health service department treat you?" If a participant in one of the early sessions made an unusually prescient remarks that demonstrated significant like-mindedness with the others it became a prompt for subsequent discussion groups.

In this survey, it was revealed that Clark students regard the faculty with an appreciation and esteem that borders on reverence. However, a substantial number of survey participants voiced strong dissatisfaction with faculty advising and the methods employed by Clark to match students with advisors, a process of random assignment that seems rewarding only when the student and advisor are involved in the same academic discipline.

Example #2:

There was a general topic about residential life. The sub-topic about residence halls was expected to reveal how students felt about the suitability of their housing. The question was: "What's the housing experience like?"

Most participants glossed over the physical aspect of the residence halls, with which they seemed content, and focused intense criticism on the Residential Life staff, which landed far short of satisfactory on any reasonable customer service scale. The responses were populated with the terms "hostile" "uncooperative" "contentious" "fascist", etc. This pent-up consumer outrage would have remained in the closet if a quantitative model had been used to measure student satisfaction with the dorms.

Outcome

The Student Life Research Project generated a few surprises, but generally confirmed what was already known or suspected. The good news was that Clark students are more than satisfied with the faculty, respect the administration and love the staff, save for the few exceptions previously noted. The improvements needed to satisfy most student objections were generally inexpensive and amounted to fixing some aspects of the University support system that had fallen into disrepair due to inattentive oversight. Through this survey, students asked for:

  • Longer hours of operation for the health service
  • Better trained and more visible security guards
  • More food options in the dining hall, including better selections for kosher dieters and vegetarians
  • A place on campus to get late-night coffee and a snack
  • Reinstatement of the University transit bus with a wider orbit of service

Contrary to some administrative fears toward this project at the outset, there was no clamoring for a new football stadium or a linear accelerator. No one called for the provost's head on a stick. Most of the complaints were easy to cure with a small investment, a change in personnel or enhanced availability of services.

The most substantially needed improvement revealed by the survey was expressed as general dissatisfaction with the library. The collection was deemed adequate, material is appreciated as readily available and the library staff is thought of as exceedingly helpful. However, the physical plant was described as "scary", "uncomfortable", "cold" and participants said some parts of the building are avoided out of fear. As a result, capital improvements are now in the planning stages and a significant budget for library renovation has been approved. Interest in a new stadium, had it been revealed, could be ignored, but a library is a more crucial element in the academic lives of students. This was a matter that compelled immediate action. For the most part, the extension of health service hours, the availability of food or coffee late into the evening, more conscientious security personnel and an attitude adjustment at the Residential Services office were enough to improve student life in a palpable way.

Improving Clark's Competitive Edge

The goal of the Student Life Research project was to gain a marketing advantage over Clark's competition by making what the for-profit sector would call "customer-driven" improvements. The "customers" (Clark undergrads) were asked.

Some of the enhancements will get into the marketplace in the form of lore and perhaps through anecdotal references in published college guidebooks. Students talking to their friends and family about residential life, the food, intra-city transit and career services will also build the University's brand. Some of the changes, such as the library renovation and a better system of faculty advising will take center stage in campus publications and addresses to prospective applicants. Meanwhile, the Clark students who participated in the project have gained a sense of empowerment because they see their requested improvements being implemented. They could be Clark's best ambassadors for future generations of applicants.

Contact Information:

Harold Wingood
Associate Provost and Dean of Admissions & Financial Aid
Clark University
950 Main Street
Worcester, MA
(508)793-7431
e-mail: hwingood@clarku.edu

Paul Marthers
Dean of Admission
Reed College
3203 SE Woodstock Boulevard
Portland OR 97202
(503)777-7511
e-mail: paul.marthers@reed.edu

Peter Herrup
Herrup, MacSwan & Associates
60 Thoreau Street, Suite 287
Concord, MA 01742
(978)461-0500
e-mail: pherrup@aol.com

Herrup, MacSwan & Associates

For an initial discussion about HMA services, please email us your questions or provide us with your name and telephone number, and we will contact you promptly:

Herrup, MacSwan & Associates
60 Thoreau Street
Suite 287
Concord, MA 01742-9116

Tel: 978-461-0500
Fax: 978-287-5901
Email: HMA@hma.cc