Fulton (2002) predicted that by 2006, a majority of American high school students would have taken atleast one online course prior to graduation. Though this prediction at that time seemed implausible in reality, subsequent developments in K-12 Education made this prediction realistic. In the Michigan Merit Curriculum Guidelines: Online Experience, the State of Michigan outlined their decision to become the first state in the United States to require that, all students take at least one course online prior to high school graduation ( Department of Education, 2006 ). This state mandate for online learning provided the basis for a dramatic increase in the number of students enrolled in virtual school courses in Michigan only to be followed by the other states across the nation ( Barbour & Reeves, 2009).

Cavanaugh, Barbour and Clark (2009) in their review of literature indicated that most of the research works published in relevance to K-12 online education focused on the experiences of the virtual school teacher or administrator and the majority of their reviewed literature described the experiences of these individuals. Another study by Barbour and Reeves (2009) suggested that, the body of published literature on K-12 online learning could be classified as falling under two categories: (i) potential benefits of K-12 online learning and (ii) the challenges facing K-12 online learning. Barbour and Reeves also mentioned that, there was a “deficit of rigorous reviews of the literature related to virtual schools” (p. 402). Barbour (2010) discussed how much amount of published research on K-12 online education is limited and also some of the published researches conducted on K-12 online education suffer from methodological flaws or have attempted to investigate beyond the scope of the researcher's inquiry.

In the following sections of this research article, the author discuss in details the Online Learning Requirement in Michigan and how the Michigan Virtual School helped the schools in Michigan to fulfill this requirement. They also discussed Career Forward as an online learning program, how it was offered, how it was accessed by students, the modules of the program and the five step learning cycle in Career Forward modules. Following this, the methodology of research for this evaluation study, were proposed and they analyze the data from the study to discuss findings and draw conclusions about the effectiveness of Career Forward as a career preparatory course. The authors end this article with the discussion of the limitations of this study and its implications.

The Michigan Virtual School (MVS) is a state sponsored virtual school designed to provide online learning opportunities to students and teachers in the state of Michigan ( Clark, 2001). In April 2006, the Michigan State Board of Education and Michigan Legislatures adopted a rigorous package of high school graduation requirements, one of which made Michigan the first state that incorporated an online learning graduation requirement in the K-12 curriculum. All Michigan's students entering high school during 2008-2009 school year were required to complete online learning during their course of high school studies in order to graduate. This requirement could be met by completing a completely online course or by blended instruction format with 20 hours of online instruction within an in-class course. To help Michigan's schools meet this requirement, the Michigan Virtual School developed a 20- hour online learning course called CareerForward. In December 2008, the Michigan Virtual University (MVU) provided the National Repository of Online Courses (NROC) access to the CareerForward course content, allowing students from anywhere in the United States the ability to freely access CareerForward.

CareerForward (from http://nroc.careerforward.org/) is an online learning program created to assist middle and high school students with planning future career paths, and developing an understanding of what it takes to achieve that desired career. Developed through a partnership between Michigan Department of Education and MVU, and with funding provided by the Microsoft Corporation, this free online learning experience was designed to be a self-contained motivational tool that allowed students to st explore 21 century career possibilities, and at the same time students would also meet the mandatory online learning graduation requirements of the state. This 4-6 week long course address questions like:

• “What am I going to do with my life?”
• “What is the working world like?” and
• “How do I match my interests with work?”

The course also uses a variety of multimedia and online resources to address the content, allowing Michigan students to meet the new online learning requirement for graduation.

CareerForward can be accessed or downloaded in three different learning environments:

(1) MVU-hosted web-based version;

(2) MVU-hosted Blackboard CMS (Course Management System) version, and

(3) School-hosted Blackboard or Moodle versions (Figure 1).

Figure 1. The Modes of Delivery of CareerForward

Schools are required to register their students and teachers to the CareerForward program. This registration process allows Microsoft to keep tract of the users enrolled in the CareerForward program even though, it is offered free to all users. Schools then decide the mode of delivery of the course to their students. Some schools chose to integrate it into Business and Computer courses, while others implement it as a stand-alone course. CareerForward can be also be companioned with a variety of online career planning tools (e.g. Career Cruising and Career Explorer), which allow students the opportunity to assess their career interests, explore career options, and create an Educational development plan ( Barbour, 2009).

CareerForward is divided into four modules that students complete, each of which guides them through a five step learning cycle (Figure 2); beginning with a scenario or challenge. The student is given framing questions for the module, and they must provide their initial thoughts. Next, the student reviews resources, that come in the form of online videos and reading material. After viewing the resources, the student is asked to complete the same framing questions to see if their initial thoughts have changed. The final step of the cycle is completing a packet of activities related to the module's content.

Figure 2. The Five Step Learning Cycle in each CareerForward Module

This evaluation study was conducted to provide Michigan Virtual School with reliable data and information required to improve the design and delivery of the CareerForward course, in order to improve the learning experiences of the future students and to improve the overall efficiency of the course. This led to the following research questions:

1. What impact does taking the CareerForward course have on student attitudes towards career planning?

2. What are the student experiences of the CareerForward course?

3. How would students improve the CareerForward course?

Two surveys, designed to be taken as a pre-course survey and a post-course survey were used by the MVS to gather data from students who participated.

The pre-course survey was used to gather information on demographic data related to gender, grade level, why the students were taking CareerFor ward, how the CareerForward course was structured (whether as a standalone course or as a part of another course) and the medium of delivery. The pre-course survey was also used to collect student ratings of 14 statements related to career planning, based on Likert scale. The post-course survey contained all of the items from the pre-course survey and some additional items. Students were asked where they completed most of their CareerForward course; also included where survey items that collected students' impressions of the CareerForward course; and finally in open-ended questions, students were also asked to comment on the changes they would suggest for the CareerForward course as the best and the most difficult things about the course. Both instruments were designed to the MVS prior to the evaluation request made of the researchers.

A total of 3899 students participated in the pre-course survey and 382 students completed the post-course survey. The MVS provided the researcher with de-identified data. As the researchers were unable to link the pre-course and post-course data, the quantitative data were compared based on the overall means on each item. The open-ended qualitative data was analyzed using a method that utilized MS Word ( Ruona, 2005).

A total of 49.4% of the participants who completed the precourse survey were female, while 55.2% of the participants who completed the post-course survey were also female.

Grade nine students made up almost half of the precourse survey sample (46.6%), while it reduced to less than a third of the post-course survey (29.1%). Grade eight students more than doubled from 13.4 to 30.9% from precourse to post-course survey. Proportion of grade ten students also increased (i.e., 18.3% to 25.9%), while grade seven and eleven remained fairly consistent (i.e., 2.1% to 0.3% and 10.8% to 9.4% respectively). Grade twelve student participants reduced to half as much from precourse to post-course survey (8.7% to 4.5% in post-course).

Table 1 Indicates the student feedback on why students took CareerForward(as indicated in the pre-course and the post- course survey data).

Table 1. Why Students Took Career Forward?

In the following sections, the author discusses the findings from the data analysis based on the pre-course and postcourse surveys. The findings are organized in the order of the research questions that guided this inquiry.

1. What Impact does taking Careerforward Course have on Student Attitudes towards Career Planning?

The impact of CareerForward on student attitudes towards career planning can be found by studying Table 2.

Table 2. Effects of CareerForward on Student Attitudes

The difference in students' average attitudes over the fourteen items as listed in Table 2 as a result of the CareerForward course is 0.046 which indicates that the course content had little positive significance on impacting students' attitude towards career planning. Table 2 data actually indicate five areas (the red cells) where student responses showed a decline as an impact from pre-course to post-course response. It is infer red from this trend that, the CareerForward course was in general ineffective in impacting student attitude towards their future career planning in a positive way.

Table 2 indicate the largest positive difference between the pre-course and post-course surveys was for that statement, “My career will be affected by the global economy.” After removing the participants who chose not to disclose their gender, it was observed that CareerForward was twice as effective in impacting attitudes of male students as female students (Table 3).

Table 3. My Career will be Affected by the Global Economy by Gender

The original difference in the pre and post course rating was 0.20 for the category “My career will be affected by the global economy”. Data was analyzed by gender after removing the participants who chose not to disclose gender. Data from Table 3 indicates that, CareerForward was twice as effective in impacting the attitudes of male students as females.

Interesting observations were noted about the impact of CareerForward on student attitudes based upon their grade level. Grade seven had only one respondent for post-course survey, so that, the data was excluded. Table 4 shows an average difference in the student attitude by grade.

Table 4. Average difference in Student Attitude (by Grade)

Table 5 indicates that, students who had taken CareerForward as a “part of a business course” had benefited the least from the course, in fact there was a negative impact of CareerForward towards student attitude. The most significant positive impact of change in attitude towards career planning due to the course was observed for students who had indicated they had taken CareerForward as a “part of another course”. The survey was not designed in a way to record what other courses students were enrolled in with CareerForward. Positive gains were also recorded for students who had taken CareerForward as a part of career planning course. Students who had taken CareerForward as a standalone course also did not benefit much from the course.

Table 5. Average difference in Student Attitude by how CareerForward was offered

2. What are the Student Experiences of the CareerForward Course?

Student experiences and satisfaction with CareerForward were collected using Likert scale from the data gathered by post-course survey items:

• I enjoyed CareerForward.
• I feel CareerForward was helpful.
• I think this course was a powerful way to gain new information.

Data analysis from these items in the post-course survey revealed that, the students did not enjoy the CareerForward course. 35.6% students who completed post-course survey strongly disagreed with the statement “I enjoy CareerForward”. Also, almost 60% of the students or 6 out of every 10 students either strongly disagreed or disagreed that, they enjoyed CareerForward. Almost half of the participants though agreed or strongly agreed that CareerFor ward was helpful and thought that CareerForward was a powerful way to acquire new information. Hence, it can be concluded that, though the students did not enjoy the CareerForward course, they found the information helpful and that the course was a powerful way to gain information.

While there were no significant differences in student satisfaction level in using CareerForward based on gender, there were some differences observed based on grade levels of the population. As discussed earlier, while majority of the students did not enjoy the course, students in Grade eleven were the only group who rated CareerForward favorably on satisfaction with the course. Grade level analysis of data also indicated that, while most students found CareerForward helpful, it was most helpful to students in grades eight and eleven. Grade eleven students followed by grade eight students found the course a powerful source of information. The course was not helpful to students in grade twelve and students in grade ten and twelve did not find the course a powerful source to acquire new information. Hence, grades eight and eleven were the main beneficiaries of the course, while the course was least effective for students in grade ten and twelve.

Finally, students who took CareerForward as a part of career planning course were more satisfied with the course than other students. Attitude data also indicated that, students who took CareerForward as a “part of another course” gained higher average gains in attitude due to the course. Similar results were observed with course satisfaction and students who had taken the course as “a part of another course” reported positive experience with the course. Students who had taken CareerForward as an independent and standalone course reported most negatively on course satisfaction items.

Three open-ended questions were included in the postcourse survey to gather student perspectives on how learning experiences with CareerForward could be improved. The prompts used were:

• If I could make one change in CareerForward it would be…
• The best thing about CareerForward was…
• The most difficult thing about CareerForward was…

While there was no specific demographic trends observed in the open ended data, the authors present below some selected student perspectives that emerged as a result of the data analysis on the questions.

Best Thing about CareerForward

Three themes emerged from analysis of data in response to the open ended question on the Best thing about CareerForward. Students mentioned how the content of the course forced them to think about future career and life after graduation. For some high school students, this might have been the first time when they thought about future careers in a systematic way. The second theme was about how the use of videos in the course content helped the students gain information. The students provided feedback that the videos were helpful in conveying a lot of information in a compact way and it helped them receive useful information from actual individuals discussing their career choices. The third theme that emerged from the students' responses again related to the course content and how it helped the students relate careers to a specific skill set, and in a way to think and to identify careers that were suitable to their own skill set. There was a subgroup of students, approximately 10% of the participants, who responded that nothing was best about CareerForward.

Most Difficult Thing About CareerForward

Four themes emerged from the responses of the participating students for this question. The first thing that the students reported as most difficult was the volume of written work that CareerForward course included. Students also found some questions too vague, sometimes the instructor imposed minimum sentence or work requirements which made it more difficult for the students, as sometimes they were repeating the same or similar answer but related to different videos. The second issue that emerged focused on the relationship of the videos to the written work. The students felt that, the written work was not always aligned with the videos and in an effort to find information for the responses from the videos they had to watch those videos repeatedly. The third theme that emerged was the nature of the videos. As discussed in the previous section, while many students found the video content helpful and informative, students also complained that, the videos were lengthy, repetitive and boring. Another theme in this section that for some students was the material and course content for CareerForward was difficult to understand. It is possible that the students found the course difficult due to the large volume of written work or not always being able to find the necessary information from the videos right away; but the student responses were general and hence this connection cannot be drawn as an inference.

One Change in CareerForward

Three main themes emerged in this category from the open-ended responses of the students. The first theme related to the videos and the students suggested that, there should be a decrease in the number of videos that the students had to watch. The second theme also related to the videos used as instructional materials. Students are suggested to the use of shorter videos and also segmenting longer videos into shorter videos. Students are also suggested using varied methods like games and other interactive activities for instruction rather than using just videos as the main instructional materials. The third suggestion from the students related to the nature of the written work. Like the content of the videos students found the written work repetitive and boring. Students suggested varying the nature of activities, specifically incorporating more computer based activities to submit as required coursework.

This study had a number of limitations that could potentially affect the generalizability of the results. First, since all of the identifiable information had been removed from the data by the MVS, the researcher was unable to conduct any statistical analysis (e.g. t-tests, ANOVA, ANCOVA, regression etc.), which would have helped to determine the impact of CareerForward on changes in attitude by comparing a single student's pre-course survey scores with post-course survey scores. Hence comparison of means was based on finding differences that would represent between 5% and 10% of the mean score.

Another major limitation was, the sample for the postcourse survey was approximately 10% of the pre-course survey sample. This disproportional difference in sample sizes would have limited the ability to draw conclusions even if the data had been identified and the evaluator could have conducted a more sophisticated statistical analysis. The smaller sample size in case of the post-course survey was problematic while analyzing the data based upon demographics. For example, there was a single seventh grade student who completed the post-course survey, also a single student who took the course as a part of a World Studies course. There were only four students who found the course on their own. Finally, there were no students who took the course using a CD-ROM and only six students who took the course using something other than the Blackboard, Moodle, or a website. These low response rates for these characteristics, made it impossible for the researcher to include these variables as a part of the demographic analysis.

Finally, approximately half of the students who indicated that, they took CareerForward using “a website” also indicated that, they took CareerForward using either “Blackboard” or “Moodle”. This level of student misunderstanding between these of variables led the researcher to exclude “a website” in the analysis of the demographic data.

Overall, there was a little impact of CareerForward on student attitude towards career planning. Students did not enjoy the course though they found the course content helpful and CareerForward also enabled access to new and valuable information. Students also found videos an effective component in the course delivery and in fact indicated videos as the best part in the course. At the same time, students also reported that, the use of lengthy videos and written work throughout the course rendered the course, repetitive and boring. They suggested shorter videos and segmenting of longer videos into smaller segments. Students also wanted CareerForward to incorporate games and other computer based activities as additional ways to deliver content and student assignments.

In terms of implications for practice, according to the data, the online career preparatory course content should be revised, so that the content contains fewer videos. Also, the length of videos should be shortened, with longer videos segmented into parts that can be independently viewed. Further, students suggested that, the inclusion of gaming and other interactive activities within the course content. Designers of such courses should carefully consider multiple pedagogically sound methods to deliver information and explore and incorporate varied formats of presenting information that appeals to the learners' preferences and styles while making the content instructionally sound and effective. Finally, as CareerForward was found to have a little effect for grade twelve students and more effect for grades eight and eleven, administrators should consider whether career preparatory courses may be more effective for students who still have some time for exploring alternatives and decision making in career choices.

There are three main areas for future research. First, more research is required to investigate the option “as a part of another course.” Students who selected this option had higher than average scores in terms of effect of CareerForward on their attitudes towards career planning and their overall enjoyment with the course. Research is needed to determine which courses were considered under this category that complemented CareerForward better than other specific courses. Second, survey items should be designed more specific. Future investigations should provide clear distinctions between the options “a There are three main areas for future research. First, more research is required to investigate the option “as a part of another course.” Students who selected this option had higher than average scores in terms of effect of CareerForward on their attitudes towards career planning and their overall enjoyment with the course. Research is needed to determine which courses were considered under this category that complemented CareerForward better than other specific courses. Second, survey items should be designed more specific. Future investigations should provide clear distinctions between the options “a RESEARCH PAPERS i-manager’s Journal on School Educational Technology, Vol. 11 l No. 3 l December 2015 - February 2016 7 website”, “Blackboard” and “Moodle” which were confusing to the respondents for this study. This is an important implication since analysis of learning environment and platform might have yielded valuable information on the course delivery. Third, further exploration of usability and pilot testing of newer methods of content delivery and pedagogically sound activities are required to improve the course effectiveness.