Assessment of the Impact of Ubiquitous Computing
Paper Prepared for Presentation to:
The Association for Institutional Research (AIR)
39th Annual Forum
Ross A. Griffith
Director of Institutional Research and Academic Administration
Associate Director of Institutional Research
David G. Brown
Vice President and Dean
International Center for Computer Enhanced Learning (ICCEL)
Wake Forest University
Wake Forest University is in the final stages of implementing an innovative strategic plan ("Plan for the Class of 2000," 1995) that has received attention from many sources and was featured in The Chronicle of Higher Education (DeLoughry, 1995). Major elements of the plan include:
with the fall semester 1996
An evaluation committee consisting of faculty and administrators is conducting the assessment of the strategic plan. To encourage use of assessment data, the evaluation committee members serve as liaisons with different committees and administrative offices throughout the University. They provide the groups results of surveys and other data with analyses as appropriate (Griffith, 1999).
The fundamental goals of the plan being measured by the evaluation committee are listed as follows:
The evaluation committee also monitors twenty-seven key measures in the following areas: quality at the entrance door, quality of activities while students attend Wake Forest, quality at the exit door and opinions concerning quality (Griffith, 1999). "Examples of key measures being monitored are freshman retention rates, student/faculty ratio, percent of graduates receiving degree credits abroad, graduation rates, alumni giving rates and academic reputation as measured in U. S. News & World Report" (Griffith, 1999).
The primary instruments being used to measure the effectiveness of the entire strategic plan are as follows:
College Student Experiences Questionnaire (CSEQ) - The survey was administered to freshmen, sophomores and juniors 1996 - 1998. It is designed to indicate how students use their time. Comparative data with other schools are available while Wake Forest's CSEQ results were highlighted in Change (Banta and Kuh, 1998).
Cooperative Institutional Research Program (CIRP) Freshman Survey - The survey has been administered to the freshmen in different years since 1980. Trend analysis has enabled Wake Forest to learn about demographic changes of the entering freshmen. Comparative data with other schools are available.
Freshman Essay - The freshman classes of 1996, 1997 and 1998 were asked to respond to questions about their expectations for college. A follow-up essay questionnaire will be administered to the respondents upon their graduation to enable comparisons with original responses.
Higher Education Data Sharing (HEDS) Consortium Alumni/ae Survey - The survey was administered to the classes of 1989 and 1993. Results enable Wake Forest to assess the effects of the undergraduate education on graduates' lives five years later. Comparative data with peer institutions are provided by HEDS.
HEDS Senior Survey - The survey was administered to the graduating classes of 1993-1998. Results of the survey enable Wake Forest to assess graduating seniors’ overall experiences. Comparative data with peer institutions are also provided by HEDS.
Higher Education Research Institute (HERI) Faculty Survey - The survey was administered to the undergraduate and MBA faculty in 1998. Comparative data of other institutions will be provided by HERI.
In-house Faculty Computer Survey - The survey was developed by faculty in the department of communication to assess faculty use of computers. It has been administered to undergraduate faculty in 1996 - 1999.
In-house Faculty Survey (1998) - The survey was developed by the evaluation committee to assess the effectiveness of all portions of the strategic plan and was administered to the undergraduate faculty in 1995 and 1998.
In-house Student Computer Survey - Faculty in the department of communication developed the survey to assess students’ use of computers and how computers aid them educationally. It was administered to the students in 1996 - 1999.
Wake Forest Fact Book (1999) - This publication, produced annually since 1992, contains approximately 60 statistical studies about students, faculty and staff, facilities and finances.
The Computing Initiative
"The most distinctive outcome of the strategic plan is that all faculty and entering freshmen received IBM laptop computers effective fall 1996. They are given an upgrade every two years, and seniors keep the computers upon graduation" (Griffith, 1999). The faculty approved a standing committee on information technology, and this committee recommends, monitors and evaluates policies for computing in behalf of the faculty. The entire campus has been completely wired for computing, and seventeen professionals with the title of Academic Computing Specialist (ACS) have been employed to provide faculty support. A totally revamped Information Systems Support Center (ISSC) is open until midnight and provides immediate technical assistance to students, faculty and staff. The library staff provides a formal training program for both students and faculty in the use of computing.
Faculty members have responded to these changes by establishing the "Wake Forest Computer-Enhanced Learning Initiative" (1998), known as CELI. It is "faculty-based and has as its mission the development of effective uses of computers in instruction. Directed by a faculty member given a reduced teaching load, CELI sponsors campus seminars on using technology in instruction and provides a number of other services as well" (Griffith, 1999). Also, Student Technology AdvisoRS (STARS) were created enabling students proficient in the use of computing to work in a collaborative role, one-on-one with faculty members.
Changes by Students and Faculty
The implementation of ubiquitous computing of this magnitude is pioneering; fewer than twenty liberal arts colleges in the United States have such a program. Thus, many efforts have been made to assess its impact (Griffith, 1999). The CSEQ, In-house Faculty Survey, In-house Student Computer Survey and In-house Faculty Computer Survey were administered to students and faculty before and after the strategic plan was implemented.
Utilizing .05 as the level of significance, Wake Forest University has been able to demonstrate changes of computer usage (McCoy, Griffith and Gu, 1999). CSEQ results show that significantly higher scores from students under the strategic plan took place in the following items: use of computers, courses using computers, use of computer in doing research and discussion of computers. See Figure 1.
Figure 1. College Students Experiences Questionnaire (CSEQ)
Faculty Survey results indicated increases for faculty under the plan in the following computer use items: teaching effectiveness, individual instruction, communication, presentation, modeling/simulation and information gathering. See Figure 2.
Figure 2. Faculty Survey Results
Student computer skill has significantly increased, as reported by both students and faculty. Faculty perception of their own computer skill is significantly greater in 1998 than it was in 1995. According to the CSEQ, student means in 1997 were significantly higher than that of 1996 in computer skill and computer training. See Figure 1. On the Faculty Survey, mean scores for computer skill and computer support and training are significantly higher in 1998 than in 1995. Faculty reported that a significantly higher number of their students were proficient in use of computers in 1998 than in 1995. See Figure 2.
Results from the In-house Student Computer Survey and In-house Faculty Computer Survey have revealed different types of computer behaviors. For example, students under the computing initiative use e-mail to communicate in general but prefer to communicate with faculty and other students face-to-face; faculty attitudes toward computing overall have become more positive under the computing initiative and the highly skilled faculty tend to use the computer for specialized purposes. (Mitra and Hazen, 1999)
In non-computer areas, the CSEQ results (1997 vs. 1996) indicate that students under the computing initiative participated more in class discussions and had more positive relationships with faculty and administration. However, these students had significantly lower scores in items regarding discussion of music, economy and personal situations as well as lower participation in volunteer activities, attending performing arts and other related campus events. These results suggest, at least initially, that the initiative might have impacted students so that they may be less rounded than previous students.
In non-computer areas for faculty, significant differences from the Faculty Survey indicated higher scores for intellectual climate among students as well as religious development of students and students being prepared for graduate/advanced education and employment after college. However, the faculty had significantly lower scores in number of papers or compositions published as well as number of professional meetings attended. These results suggest that computer training might have occupied time for faculty they otherwise would use in research and professional pursuits.
There is indication that the computing initiative has had a positive impact on academic achievement. The retention rate of first-year students who entered in 1996 (the first year of the strategic plan) was higher than that of those entering in the previous three years. Also, the 94.3% retention rate of the 1997 freshman class was even higher than the 1996 freshman class retention rate of 93.4%. See Figure 3.
Figure 3. Freshman Retention Rate
The mean grade point average of first-year students at the end of the first year was higher in 1996 (2.81) and 1997 (2.83) than in the immediate years prior to the implementation of the strategic plan. See Figure 4.
Figure 4. Mean Cumulative Grade Point Average
End of Freshman Year
With the improvements indicated, the academic quality of the entering freshman class has been essentially unchanged as far as students' mean SAT scores and academic rankings in high school graduating classes (McCoy et al., 1999).
The data suggests that the computing initiative of the Wake Forest strategic plan has been effective to this date based upon the faculty and student survey results as well as retention and average GPA of freshmen who enrolled under the initiative (McCoy et al., 1999). The computing initiative received such a vast amount of interest from other institutions that the International Center for Computer Enhanced Learning (ICCEL) at Wake Forest was formed (1997), and Wake Forest faculty and administrators have been satisfied with the state of the strategic plan (Cox, 1998).
However, recurring and thorough assessment is necessary since the first class to enroll under the computing initiative will not graduate until 2000. Senior and faculty survey results as well as measures including graduation rates and other outcomes can then be compared to previous classes and faculties and comparable institutions to evaluate the complete effectiveness of the computing initiative.
Banta, T. and Kuh, G., (March/April 1998), "A Missing Link in Assessment," Change, pp. 40-46.
Cox, K. (1998). Fall semester marks new stage in undergraduate plan. News Service,
Wake Forest University. Retrieved March 24, 1999 from the World Wide Web: http://www.wfu.edu/wfunews/1998/081498p.htm.
DeLoughry, T. (May 5, 1995). "Mandatory Computers," The Chronicle of Higher Education (pp. A37, A39).
Fact Book (March 1999). Office of Institutional Research, Wake Forest University. Available HTTP: http://www.wfu.edu/ir/factbook.html
"Faculty Survey" (1998). Office of Institutional Research, Wake Forest University. Available HTTP: http://www.wfu.edu/ir/docs/1998Results.PDF
Griffith, R. (April 1999). "Connecting Students and Faculty through Technology, Collaboration, and Globalization at Wake Forest University," The Technology Source, Available HTTP: http://horizon.unc.edu/TS/cases/1999-04.asp
"International Center for Computer Enhanced Learning." Retrieved May 17, 1999 from the World Wide Web: http://iccel.wfu.edu/
McCoy, L., Griffith, R., and Gu, Y. (1999). Assessment from Multipurpose Surveys and Other Sources. In D. Brown (Ed.), Electronically Enhanced Education. Winston-Salem: Wake Forest University Press, pp. 95-100.
Mitra, A. and Hazen, M. (1999). Longitudinal Assessment of Computer Enrichment at Wake Forest University. In D. Brown (Ed.), Electronically Enhanced Education. Winston-Salem: Wake Forest University Press, pp. 101-106.
"Plan for the Class of 2000" (January 1995). Office of the Provost, Wake Forest University. Available HTTP: http://www.wfu.edu/Academic-departments/Program-Planning-Committee/Plan-for-the-Class-of-2000.txt.
"Wake Forest Computer-Enhanced Learning Initiative." Retrieved March 24, 1999 from the World Wide Web: http://www.wfu.edu/CELI/