Sector-wide measurement of student engagement – validity for online learning

The 2015 Australian national survey of the undergraduate student experience report was released recently. In exploring the Quality Indicators for Learning and Teaching (QILT) website, I came across this explanation of the student engagement indicators that stated that the measure of student engagement was based on the following questions:

“During 2014 – 2015, to what extent have you:

  1. felt prepared for your study?
  2. had a sense of belonging to your institution?
  3. participated in discussions online or face-to-face?
  4. worked with other students as part of your study?
  5. interacted with students outside study requirements?
  6. interacted with students who are very different from you?
  7. been given opportunities to interact with local students (where applicable for international students)?

Note that Learner engagement results do not include survey responses from students studying by distance education, because the current form of questions used in the SES may not adequately capture the Learner engagement experience of these students (my emphasis).”

I was taken aback by the statement above that distance education students had been removed from the results because the questions ‘may not adequately capture the Learner engagement experience of these students.’’ I’ve been learning and teaching online since 1999, and I believe that these questions are valid for distance education. Certainly, old school, Web 1.0 distance learning would not have allowed students to interact and participate in discussions unless they attended on campus intensives, however, given the range of online interactive learning environments and social media that has become ubiquitous in this century, there is no excuse for these questions not being valid.

The survey also included indicators for learning resources (see extract below). Interestingly, there was no corresponding statement  that the responses from distance education students were removed, despite these questions being heavily weighted towards on campus resources and support services (emphasised in bold green). The examples given of teaching spaces direct the student towards physical spaces rather than being inclusive of virtual spaces such as learning management systems. It is also probable that ‘student spaces and common areas’ and ‘laboratory or studio equipment’ would be perceived as physical rather than virtual spaces. In terms of support services, the questions mention careers advisors, academic or learning advisors, counsellors, financial/legal advisors and health services. Very few of these services would be available to distance education students (though at my University, the counselling service will conduct phone sessions if needed).

“Thinking of this year, overall how would you rate the following learning resources provided by your course?

  1. Teaching spaces (e.g. lecture theatres, tutorial rooms, laboratories)
  2. Student spaces and common areas
  3. Online learning materials
  4. Computing/IT resources
  5. Assigned books, notes and resources
  6. Laboratory or studio equipment
  7. Library resources and facilities

To what extent have you:

  • received support from your institution to settle into study?
  • experienced efficient enrolment and admissions processes?
  • felt induction/orientation activities were relevant and helpful?
  • found administrative staff or systems (e.g. online administrative services, frontline staff, enrolment systems) to be:1. available? 2. helpful?
  • found careers advisors to be: 1. available? 2. helpful?
  • found academic or learning advisors to be: 1. available? 2. helpful?
  • found support services such as counsellor, financial/legal advisors and health services to be: 1. available? 2. helpful?
  • 1. been offered support relevant to your circumstances? 2. received appropriate English language skill support?

This lead me to explore the other student experience surveys to examine their validity for distance education students. The now defunct Australasian Survey of Student Engagement (AUSSE) was last administered in 2012. It was based on the US National Survey of Student Engagement (NSSE) which is still being administered annually in the US. The 2016 US survey has a range of questions that are specifically related to the on-campus experience. I have extracted them as follows:

During the current school year have you:

  • Worked with a faculty member on activities other than coursework (committees, student groups, etc)

Which of the following have you done or do you plan to do before you graduate?

  • Hold a formal leadership role in a student organization or group
  • Work with a faculty member on a research project

Indicate the quality of your interactions with the following people at your institution:

  • Academic advisors, student services staff (career services, student activities, housing, etc.)

How much does your institution emphasize the following:

  • Using learning support services (tutoring services, writing centre, etc)
  • Providing opportunities to be involved socially
  • Providing support for your overall well-being (recreation, health care, counseling, etc.)
  • Attending campus activities and events (performing arts, athletic events etc)
  • Attending events that address important social, economic, or political issues

About how many hours do you spend in a typical 7-day week doing the following?

  • Participating in co-curricular activities (organizations, campus publications, student government, fraternity or sorority, intercollegiate or intramural sports, etc)
  • Working for pay on campus
  • Commuting to campus

These questions are indicative of a rich, on-campus experience, something that US universities and colleges are well-known for. In comparison, Australian universities are less likely to have on-campus residences and do not have the strong US tradition of fraternities/sororities, athletic scholarships and intercollegiate sports.

In both the Australian and US surveys, there was no option for distance education to indicate that the question was ‘not applicable’. I couldn’t find any statement on the NSSE website to the effect that distance education students students’ responses had been excluded from any or all of the indicators. All of this has made me wonder about how we should or could measure and describe student engagement in online learning environments. I will explore this idea in future posts.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s