Pradeep on Evaluation -return if not happy

Summary of results from evaluation survey

2009-06-25T04:31:00.000-07:00

On second thoughts it is probably best to view this document on google docs as I have lost most of the formatting of the tables below.

Summary of all results
The learning styles survey showed up a preference by students for visual learning as opposed to verbal. Other learning styles were evenly spread. A visual learning preference may be considered at odds with the traditional teaching style being predominantly verbal (Fielder, index of learning styles, 1988). The proposed new resource does lend itself towards a more visual presentation.
Results from the technology usage survey indicates a high level of computer/web competencies by students. They all have access to computers and use them almost daily. Student felt confident with using email, searching the web, word processing and using spreadsheets. A little less confidence in the use of library resources, on line chat or discussion boards. Students had little or no experience with web pages or blogs. With respect to introducing the interactive resource it appears that student can be expected to have the skills necessary to do so, but some caution and possible extra training or help will be required if discussion boards are to used
The survey on Trial interactive learning resource showed up that students were able to use the resource, but that not all felt that it was a good way to learn. This finding is probably reflective of the range of learning style in a classroom. With approx 50 to 75% of survey students either strongly liking or non committal to the resource there is still a case for the introduction of such a learning resource as part o a blended learning delivery.
Three interviews were carried out, one with Diane Tamiti the elearning specialist in the IT department, a second with Patsy Deveral as someone how has done some work with e- portfolios and thirdly with Adel Awaan
Fielder, R. m. (n.d.).
Fielder, R. m. (1988). index of learning styles. Retrieved May 10, 2009, from http://www.engr.ncsu.edu/learningstyles/ilsweb.html
an engineering tutor as someone who maybe using the resource for teaching. (Due to scheduling and time constraints, the interview with an industry person, a future employer of engineering graduates did not take place)
Diane Tamiti supported the use of the resource, commenting that it was of good quality and graphics. She did warn that students would be wary of the resource until they got familiar with it. It will help in creating interest and enthusiasm. She also pointed out that this type of resource could be put together using power point, and hence features can be tailored in house by tutors. As tutors get used to using Power point more and more can be achieved. She suggested other types of elearning resources that can be used and emphasized that today’s students are familiar with audio and video by virtue of their mobile phones.
Patsy Deveral showed how a e portfolio could be used as part of teaching – suggesting that it may be appropriate in engineering as in other disciplines to help students to keep a record of the work they do and the progress they make throughout their time on the program. She said it would change the way we looked at assessing.
Adel Awaan was supportive of the use of the resource as part of a blended delivery, but reservations about the level and type of support that student or tutors may get. He was also concerned that students attendance may fall as students think they can get the information in other forms.
Results as they assist towards answering the questions proposed
TD3 How does the e-learning encourage a realistic progression towards self direction and recognises varied starting points in levels of confidence and motivation?
Sub Questions
· Will students be suitably skilled to use the technology?
From the students surveys it appears that students are computer/web literate. There may be some unfamiliarity with discussion boards but there overall computer skills indication was high.
· How do students feel about using web based interactive learning resources.
Students were mixed in their feeling towards using the resource, Since the resource was never intended to be the only teaching resource, the fact that some students like it and that maybe those that did not may still get used to; still support the use of the resource.
· Will there be sufficient technology support
The peer teacher expressed that there is some question as to the adequacy of the support available, in both the setting up the hardware, and in helping students.

· Will tutors embrace the new technology and the use of e-learning for teaching.
The tutors interviewed were all supportive of the use of the elearning interactive resource, but did foresee some difficulties including time, upskilling and support services. In addition a number of other methods of using elearning were suggested

TD3 How does the e-learning encourage a realistic progression towards self direction and recognises varied starting points in levels of confidence and motivation?
It appears that all students consider themselves skilled in the use of emailing, web surfing, word processing and excel spread sheets, along with this students have a good knowledge of web applications. Students entry skills are high. Dianne suggested that as students become more aware of the use of the electronic resource, they would become more independent. Given the high skill level of the students it is envisaged that students technology skill progression should not impose an undue usage of their time. The resource needs to be made to accommodate increasing skill level, but not to progress so much as to add extra learning burden onto students

Note
Unfortunately this following second Question (Do students gain knowledge relevant to employment and/or current thinking in their field?) did not get answered as the main contribution, the interview with the industrial expert, did not occur.
Appendix 1 to 4 are the summary of the results of the surveys and interviews

Appendix 1 Learning styles survey
The learning style questionnaire used is on line at http://www.engr.ncsu.edu/learningstyles/ilsweb.html . It is a 44 item questionnaire that can be submitted and automatically scored on the web. It was design by Richard M Felder (Fielder) (Fielder, index of learning styles, 1988). The paper arose out of trying to explain learning styles in Engineering education, where he grouped 4 categories of skills. The following are brief descriptions of each for further descriptions and how each affect learning refer to Fielders web site. http://www4.ncsu.edu/unity/lockers/users/f/felder/public/ILSdir/styles.htm .
Active and reflective learners Active learners tend to retain and understand information best by doing something active with it--discussing or applying it or explaining it to others. Reflective learners prefer to think about it quietly first
Sensing and intuitive learners Sensing learners tend to like learning facts, intuitive learners often prefer discovering possibilities and relationships.
Visual and verbal learners Visual learners remember best what they see--pictures, diagrams, flow charts, time lines, films, and demonstrations. Verbal learners get more out of words--written and spoken explanations. Everyone learns more when information is presented both visually and verbally.
sequential and Global learners Sequential learners tend to gain understanding in linear steps, with each step following logically from the previous one. Global learners tend to learn in large jumps, absorbing material almost randomly without seeing connections, and then suddenly "getting it."
The web questionnaire gives students a result on a scale of 0 to 11 towards one type of learner or the other. To fully quantify results, one learner preference has been awarded -11 while the other is awarded 11, hence a score of -6 in the visual/verbal category indicates a moderate preference towards Visual.
A score of
· 1-3 indicates fairly well balanced,
· 5-7 a preference for one dimension
· 9-11 a strong preference for one dimension
Two classes of students were surveyed – one of prospective engineering students (10 in number) and the second class was of prospective Information technology students( 9 in number).
Results
1. There does not appear to be a large difference between the groups of students
2. Looking at the average for the total number of students, the only outstanding trend was for visual learners as opposed to verbal learners. The other learning styles showed up as fairly well balanced. The visual preference is very significant given that our traditional face to face learning has a large component of verbal instruction. Visual learners remember best what they see – pictures, diagrams, flow charts, films and demonstrations

Appendix 2 Results from student survey Technology usage and comment on trial resource
The students surveyed were from a class of prospective engineering students, made up of 4 males all of 18 to 91 years old, 4 electrical engineers and 4 mechanical engineers.
The results showed that students have access to computers, and use them for approx 28 hours per week. The all use email almost every day, synchronous and asynchronous communication was used by about half of students while only a few made use of web pages or blogs. Overall students usage of computers and the internet was high.

Summary of results as follows.
Q1: Approximately, for how many years have you been using a computer for: Answers ranged from 3 to 12 years with an average of 7.63. So all students have had computer use.
Q5 Approximately, how many hours a week do you spend at home or somewhere else on the internet. The answers ranged from 14 to 40 hours per week, with an average of 28.8 hours; there is a high usage of the net. This corresponds with the frequency per week question where 75% of students said they used the computer daily
Q3: I have access to a networked computer All students had access to the net from home and 5 considered they had access also at the polytechnic
Q7: I use social networks and web 2.0 tools . 50% of the students did
Q9: I use messaging and discussion tools All students use the for messaging
Q10: I play online games or use virtual worlds and talk to other players (E.g. World of War Craft, Battlefront 2, Sims Online, Second Life) Approx 38% played online games, with a wide variety of games being played
Q11: I have an online personal space other than a social network (E.g. Web pages, blogs No students had made their own blogs or web pages
Q12: I use other social networking and communication tools online (E.g. Online dating, Friends Reunited No students involved here – may reflect their age and culture
Q2: I normally use a computer: 75% use it daily
Q4: I normally access email and/or the Internet: . 75% use it daily
Q8: How often do you use synchronous (At the same time) text chat or voice over tools? 50% of students use synchronous communication tools , but 38% did not use it .
Q9: How often do you use asynchronous messaging and discussion tools? All students were at least familiar with this tool
Q10: How often do you play online games or use virtual worlds the weighted average corresponded to occasion use only
Q11: How often do you use your online personal space? All student answered never
Q12How often do you use your other social and communication tools online close to never

Appendix 3 Technology Skill Level
The results showed that 100% felt they were skilled and very skilled with email, web surfing and word processing, and 92% felt confident with Excel spread sheets. The confidence decreased as for presentation software, video and audio, searching for online library resources and course management tools. The interesting result was that there were still some that felt they had high degree of skill in each of these tools. Figure 1, shows a definite cluster towards the very skilled and skilled end.
Figure 1

Appendix 4 Section B Survey on Trial interactive learning resource
Results showed that only 1 student had difficulty with the instructions (with a comment to match), and no student had any problems with navigation around the resource. When it came to learning from the resource, 38% of students strongly disagreed and comments about that were expressed – “very technical” and “prefer to be taught by someone not something”. There was defiantly a negative reaction towards the resource by some students, but on the other side more than half of students were either neutral or for it - Figure 2 shows this result from the survey graphically.
Figure 2
As the resource is not intended to replace traditional face to face teaching, only to supplement it, those students that do like it can still benefit from it. A point to note is that this type of teaching resource does not suit everybody.

Appendix 5 Raw Data

Learning styles survey
Results
Engineering students
IT students
Active and reflective learners
Sensing and intuitive
visual and verbal
Sequential and global
Active and reflective learners
Sensing and intuitive
visual and verbal
Sequential and global
1
-3
-7
1
-5
-11
3
-11
3
2
3
1
-3
1
-7
3
-11
7
3
-5
-3
-9
1
-5
-3
-9
-1
4
-3
-1
1
-5
5
-3
3
-7
5
7
-9
-3
1
-5
-3
-9
-5
6
-5
7
-11
9
-5
-3
-7
1
7
-5
9
-11
1
-3
-1
-9
7
8
7
-3
-9
-1
-5
-3
-5
-3
9
-5
7
-11
9
3
-3
-5
3
10
3
1
-7
3
Average
-0.6
0.2
-6.2
1.4
-3.7
-1.4
-7.0
0.6
Total for both engineering and It students
-2.1
-1.6
-6.6
1.0

Results from student survey
Technology usage and comment on trial resource
Q1: Approximately, for how many years have you been using a computer for?
Average
=
7.67
Q5 Approximately, how many hours a week do you spend at homw or somehwere else on the internet
28.8 hours
At home/ student residence
At work
At University/ College/ Learning Centre
Q3: I have access to a networked computer (please tick all that apply)
8
0
5
Q7: I use social networks and web 2.0 tools (E.g. MySpace, Flickr, Facebook)
yes
4
No
4
Q9: I use messaging and discussion tools (E.g. Email, forums, phone texting
yes
8
No
0
Q10: I play online games or use virtual worlds and talk to other players (E.g. World of War Craft, Battlefront 2, Sims Online, Second Life)
yes
3
No
5
Which?
9dragons, lords of evil, call of duty4, GTA4, counter strike, book work and Star grab
Q11: I have an online personal space other than a social network(E.g. Web pages, blogs
yes
0
No
8
Q12: I use other social networking and communication tools online (E.g. Online dating, Friends Reunited)
yes
0
No
7

4
3
2
1
Every day
A few times a week
Occasionally
Rarely/never
Weighted average
Every day
A few times a week
Occasionally
Rarely/never
Q2: I normally use a computer (please tick one)
6
1
1
0
3.6
75%
13%
13%
0%
Q4: I normally access email and/or the Internet (please tick one)
6
2
0
0
3.8
75%
25%
0%
0%
Q8: How often do you use synchronous (At the same time) text chat or voice over tools?
4
0
0
3
2.7
50%
0%
0%
38%
Q9: How often do you use asynchronous messaging and discussion tools?
2
5
1
0
3.1
25%
63%
13%
0%
Q10: How often do you play online games or use virtual worlds
1
2
1
4
2.0
13%
25%
13%
50%
Q11: How often do you use your online personal space? (please tick one)
0
0
0
7
1.0
0%
0%
0%
88%
Q12How often do you use your other social and communication tools online?
0
0
1
6
1.1
0%
0%
13%
75%
Skill levels Fill out the following table, place a tick in the skill level you feel you have for each of the computer programs listed

Very skilled
Skilled
Unskilled
Very unskilled
Do not use
Weighted average
Very skilled
Skilled
Unskilled
Very unskilled
Do not use

5
4
3
2
1
5
4
3
2
1
Actual student number
% of students
Email
2
6
0
0
0
4.3
25.0%
75.0%
0.0%
0.0%
0.0%
Web surfing
4
4
0
0
0
4.5
50.0%
50.0%
0.0%
0.0%
0.0%
Word processing
2
6
0
0
0
4.3
25.0%
75.0%
0.0%
0.0%
0.0%
Excel spreadsheets
1
6
0
1
0
3.9
12.5%
75.0%
0.0%
12.5%
0.0%
Presentation software (Power point)
1
4
3
0
0
3.8
12.5%
50.0%
37.5%
0.0%
0.0%
Creating and editing video/audio
2
1
3
0
2
3.1
25.0%
12.5%
37.5%
0.0%
25.0%
On line library resources
1
2
3
1
2
2.9
12.5%
25.0%
37.5%
12.5%
25.0%
Course management systems (moodle, black board, web-CT)
1
3
0
1
3
2.8
12.5%
37.5%
0.0%
12.5%
37.5%

Section B Survey on Trial interactive learning resource
Strongly agree
Average
Strongly disagree
Weighted average
Strongly agree
Average
Strongly disagree
Number label
3
2
1
3
2
1
(Actual student numbers)
% of students
instructions for using the resource were easy to follow
3
4
1
2.3
37.5%
50.0%
12.5%
Was it easy to navigate around the site
4
4
0
2.5
50.0%
50.0%
0.0%
Was the resource a helpful way to learn.
1
4
3
1.8
12.5%
50.0%
37.5%
Did you feel that you succeeded in learning what was intended
0
5
3
1.6
0.0%
62.5%
37.5%
How would you feel to use this type of learning resource as part of your course
2
2
4
1.8
25.0%
25.0%
50.0%
Do you think this type of interactive resource could be fun to use
2
4
2
2.0
25.0%
50.0%
25.0%
Comments
instructions for using the resource were easy to follow
Instructions not clear
Was it easy to navigate around the site
Was the resource a helpful way to learn.
It was difficult and very technical
I prefer being taught by someone not something
I have no background in Mechanics of Materials so things were hard
Did you feel that you succeeded in learning what was intended
I need to learn basic things and terminology
I learned some of it with difficulty
How would you feel to use this type of learning resource as part of your course
if I was advanced
I do that
Not affective
Do you think this type of interactive resource could be fun to use
it could be fun if refined and made in a easier more user friendly way
Yeah why not
not fun
no interaction with humans
Section C: Personal Details

Age
Average
18.5

Gender:
Male
8
Female
0

Which program do you intend to study
Engineering Electrical
4
Engineering mechanical
4

Interview with Diane Tamiti, the elearning specialist in the IT department.
summary
The following is a summary of an interview I held with Diane Tamiti[1]. The trial resource was shown and explained to her briefly.
Do you think this is an appropriate resource for students for a blended delivery course
This is great, its of good quality, nice graphics, and interactive, good navigation too. There may be some resistance to using the resource in the early stages and require extra time form the tutor as you get students used to the idea of using it. After that initial period you could use it to do section or topics of teaching by just getting them into a computer lab. And telling them which web page to go to.
Definitely good to be used in a mixed/blended learning situation. Students need have face to face, elearning is good as variety, and you may need some form of a ‘hook’ to get them started to use it. Maybe assessment marks or if using moodle make it the way you present your notes or answer to exercises to them.
Advantages in using elearning -create interest, enthusiasm, get collaboration going if you use some form of discussion board, and variety
Can this resource be prepared by us at Bahrain polytechnic.
You can use power point to duplicate it, you can add action buttons to link up pages, add sound, graphics etc. Make it available on moodle (I don’t like moodle but that’s what we have.) The power point part of it would be relatively quickly done and we have the expertise here at the polytechnic, the difficult part is preparing what you actually want to put into the resource, that will take the majority of the time.
Any other type of elearning resource that could be used
Try audio and video – you could get students to make videos, they will know how to do it, they will all have cell phones, they could even upload them up to moodle. Students like to look at themselves. You could explain some topic to them and then get them to make a video of it, present it to the class.
Have a FAQ section on moodle
Allow emailing of tutors.
What about the idea of video recordings of lectures?
Keep them short. Good for revision. Make a video of FAQ.
What about discussion boards –how can they be used for engineering – a primarily maths based /analytical course
I don’t know what is in your course, but I’m sure that there must be some discussion questions that must be asked. Or give them a situation and then ask them to comment is it true/ why might it not be true?
You would have to make a certain time when all were available on Moodle and get them all to sign in during that time. You will need some form of hook to catch their interest, but these students as I have seen them are very keen to use discussion boards

Expert Interview with engineering tutor.

This is a summary of an interview I had with Adel Awaan[2], a mechanical engineering tutor.
What is your opinion on the introduction of elearning into engineering programs
It is a good thing but it needs to be done in line with tradition class room teaching, as an extra resource. Students seem to like using the computer, so this may appeal to them, but they do not like to read too much, so it needs to be interactive. They certainly are good at using the computer.
Do you feel you have the skills, time to use e-learning software
The skills will improve as we use it more, but I can foresee that it will take a lot of time to set up, and there will be hiccups with the technology. That can take up a lot of time, be nice if we could have it all set up before classes start
How much e-learning should there be %?
I would guess about 25%
Do you feel that there is sufficient technology support to be able to deliver e-elearning in a blended course?
Support can be a problem here, the technicians are under pressure from the start up of new projects and they are not always up to date with versions of software. Added to that there is the problem that we do not have an IT helpdesk which student can access.
What drawback or difficulties do you foresee
The support side of things and secondly to get student to come to class, if they find they can get what they need via the internet or computer then that may give them another reason for not coming to class.
Bibliography
Fielder, R. m. (n.d.).
Fielder, R. m. (1988). index of learning styles. Retrieved May 10, 2009, from http://www.engr.ncsu.edu/learningstyles/ilsweb.html

[1] Interview held 7-6-2009 at 2:30 Bahrain Polytechnic.
[2] Interview held at Bahrain Polytechnic on 18 June 2209, at 1.30 pm

Final version -Proposed evaluation

2009-06-13T08:28:00.001-07:00

Title: Evaluation of the Introduction of e-learning into engineering mechanics

Introduction: - great introduction

This a front end or needs assessment evaluation for the introduction of e-learning into the mechanical engineering course at the Bahrain Polytechnic. We Staff in the ....School/Department aim to trial some interactive web based resources into the Engineering Mechanics paper, which is a level 5 paper, carrying 15 credits and will run in the second semester of the new academic year. The actual e-learning resource has not yet been developed but the Merlot website has an interactive resource along the lines that is intended to be introduced into the engineering mechanics paper.

In the words of Allison Rossett (Rossett, 1995) a needs assessment seeks to

Gather information to assist professionals in making data driven and responsive recommendations about how to solve the problem or introduce the new technology.

This needs evaluation is to provide information to help make decisions about the introduction of new e-learning resources.

This evaluation will include two questionnaires to students, one to ascertain their learning styles, and the second to find out how they feel about merlot resource. We are using the merlot website to mimic our resources, from which information about student’s opinions and feelings can be gained with some relevance. The evaluation will also include interviews with teaching staff and prospective employers to determine that the resource is in line with teaching requirements and relevant to industry.

a sentence about what else is in the plan would be good at the end of this paragraph. -Bronwyn 6/3/09 3:24 PM

Background: - excellent description of the reasons for your evaluation.

What is to be evaluated?

It is intended that web based e-learning be introduced into engineering learning resources. A pilot resource is to be introduced into Engineering mechanics in the form of a interactive resource along the lines of a resources that has been developed and available in the public domain from the Merlot website namely “MecMovies” on this address: http://web.mst.edu/~mecmovie/ . For the sake of this evaluation we will use the MecMovies resources to get some idea of how students react to the resource. The actual resource will need to be developed to meet the requirements of our Mechanical engineering course.

Why

In the Mechanical engineering course that are currently being developed at Bahrain polytechnic, e-learning take the form of electronic notes and assignments made available to students on a web based filing system. A more dynamic use of interactive e-learning in the form of quizzes, video recordings of lessons and possible group discussions is proposed.

It will take some time to introduce e-learning into the engineering course, the intention is not to replace face to face learning but supplement it with additional resources i.e. to develop resources towards an appropriate blended delivery. The introduction of e-learning is envisaged to increase as students and tutors become more aware and familiar with types and capabilities of tools available.

Bahrain Polytechnic is committed to providing a modern facility that is internationally recognized with student focused teaching methods. E-learning offers another form of teaching which may appeal to students who require a more time flexible platform and those students who are at different skill levels. With this in mind e-learning is to be introduced as an additional teaching resource. David ~~Haffaker~~ (Huffaker (2003) concluded

“Elearning application can be personalized to the learners needs and still provide communication that foster collaborative, pro-social work. Since e-learning offers an anytime-anywhere transfer of information, it cultivates learning both inside and outside the classroom.”

Which is supportive of the benefits of a flexible platform, While Mason and Rennie (Mason & Rennie, 2006) pointed out that

“..blended approaches can encourage participants to make better use of face to face contact in the knowledge that preparations and follow up can be conducted online”. .

Which supports the use of blended learning.

This needs assessment is being conducted to ~~is to~~ try and clarify issues that may arise for both students and teachers and to ensure that the resources used are appropriate for the learner in terms of current usage and for their future prospect employment.

Purposes: good

To investigate appropriate methods to introduce online interactive e-learning into a engineering mechanics course if it was offered by blended delivery?

Audiences:

The stake holders are

Students present and future

Engineering tutors at the polytechnic

Future employers of graduates

Degree Accreditations boards

Decisions: good

Estimation of how much time, effort and money should to be spent into establishing e-learning resources in the engineering course based on recommendations from the needs assessment.

Questions: great guidelines and questions

There are two guidelines taken from Massey university elg web site (Massey University, 2006) “Guidelines for the support of e-learning in New Zealand tertiary institutions” as follows with sub questions for this evaluation.

TD3 How does the e-learning encourage a realistic progression towards self direction and recognises varied starting points in levels of confidence and motivation?

This guideline looks at student’s familiarity with technology and whether they are capable of using the e-learning without the need to up skill at the same time as they are trying to learn the course material. This is relevant to the introduction of e-learning as if the new resource is not pitched at the correct level, or with the support required the e-learning can be contrary to the learning objectives.

Sub Questions

Will students be suitably skilled to use the technology?

How do students feel about using web based interactive learning resources?

Will there be sufficient technology support?

Will tutors embrace the new technology and the use of e-learning for teaching?

SD3 Do students gain knowledge relevant to employment and/or current thinking in their field?

One of the mandates for the existence of Bahrain polytechnic is that their graduates have relevant skills for their chosen vocation. There has been a common complaint by employers that graduating students are not ‘work ready’. That may well be an unreasonable request by employers but Bahrain Polytechnic’s aim is that courses will give students skills that are aligned with industries requirements .

Sub Questions

How important is online learning for preparing graduates for e-engineering in their careers?

Is investment into e-learning resources of this type justified in term of relevance to the working environment?

Methods: this is really clearly explained.

How the evaluation will work.

This evaluation will use a 'multiple methods' model to collect data and information. The paradigm used is the Eclectic-Mixed Methods-Pragmatic Paradigm. This methodology allows for a number of different ways to look at the same question. As Professor Tom Reeves (~~Reeves~~) (2003) suggested ~~that~~ this paradigm is most useful “Because it is the one approach most capable of handling the complexity that is the hall mark of contemporary society and technology”. Instructional design is complex and does not lend itself to large volumes of data collection or being able to isolate any one variable. This evaluation uses both surveys and interviews, with some quantitative data collection for students current learning styles and technology usage, and some open ended questions aimed to get students, tutors and industrial experts opinions and comments.

Types of data collection devices that will be used and the sample

Student learning styles survey –To find student demographics and learner types. Prospective students currently enrolled in the foundation engineering course are to be given a ‘learner style’ questionnaire – this will help establish how many students will benefit from the use of e-learning resources. The Learning styles questionnaire in on line http://www.engr.ncsu.edu/learningstyles/ilsweb.htm it is a 44-item questionnaire that can be submitted and automatically scored on the Web.

Student survey on a trial resource. A number of students, volunteers from the group of currently enrolled engineering students, approximately 6-8, will trial the “mecMovies resource and fill out a survey as to their opinion and feelings about the use of the on line interactive learning resource. This survey would have open ended questions encouraging students to give opinion and possible recommendations.

Student interviews on following a videoed trial ~~after being videoed.~~ Video record students while they do use the trial software, and then interview them using the recording to prompt them. This tool will not be used in this evaluation due to time and equipment constraints.

Expert review – Conduct an Interview with engineering tutor/show many? -Bronwyn 6/3/09 4:04 PM . as to their view on e-learning resources. In respect of difficulties and advantages they see in the implementation of e-learning.

Conduct an interview of one relevant industry person (preferably a prospective employer) to see if the resource will compliment skills that are required in the work force, i.e. that the technology skills student acquire while using on line learning resources are relevant and of future use to students when they gain employment.

A matrix of the instrumentation used to gather data to answer the evaluation questions is shown in Table 1.

Table 1: insert caption -Bronwyn 6/3/09 4:07 PM

Matrix Evaluation tools Vs Questions	Student learning style survey	Student technology skill survey	Students trial resource commentsno need to include this if it is part of the survey -Bronwyn 6/3/09 4:05 PM	Peer expert interview	Industry expert interview	[1]Student interviews on the trial after being videoed
Will students be suitably skilled to use the technology?	X	X		X		X
How do students feel about using web based interactive learning resources.	X	X	X			X
Will there be sufficient technology support				X
Will tutors embrace the new technology and the use of e-learning for teaching				X
How important is online learning for preparing graduates for e-engineering in their careers?				X	X
Is investment into e-learning resources of this type justified in term of relevance to the working environment?				X	X

Instrumentation:

The trial software to be used as part of the needs assessment is that on Merlot “MecMovies”url address http://web.mst.edu/~mecmovie/

Learner style questionnaire “Index of Learning Styles Questionnaire” developed by Barbara A. Soloman and Richard M. Felder of North Carolina State University

Raleigh, North Carolina url address http://www.engr.ncsu.edu/learningstyles/ilsweb.htm

Student technology usage and triail software survey Attached appendix 1

A questionnaire for interviews of teachers and prospective employers Appendix 2

Limitations:

The small numbers of staff and students involved gives a limited view for the results.

The actual e-learning resources which we will end up use ing in the course may vary from the one we trail.spelling -Bronwyn 6/3/09 4:11 PM

Logistics:

Learner style survey, trial software “MecMovies” and questionnaire to be conducted outside of class time.

Organize students trial – in their own time outside of class

Conduct a student review on the trail software (Questionnaire as per appendix 1)

Prepare an informal interview with peer tutor and prospective employer

Time Line: looks achievable all going to plan. :)

Preparation of evaluation questionnaires and trail assessments – Complete by 21 may

Carry out the trail with students in week 24 to 28 May

Interview with tutor and industry expert also in week 4 to 28 May

Collation of results 31may to 5June

Compilation of report 26 June

Budget: good budget

Staff time 48 hours

Preparation of proposed evaluation plan

Preparation of surveys and interview

Completion of surveys and interview

Collation of results

Compilation of report

Cost Staff time 48 hours at $65/hr cost $ 3120

Incidentals $ 350

Total Cost $ 3470

Pradeeps EVALUATION PLAN 17 | Page May 09

Appendix 1 - very comprehensive questions. :)

E-learning Participants Questionnaire

Thank you for taking the time to fill in this questionnaire. This questionnaire is being used to investigate the online technology you use and your experiences.

Responses to this questionnaire will be absolutely confidential and no other participants will be able to see your data. The answers to this questionnaire will go only to the project researcher (Pradeep Nathoo) and any information which could potentially identify participants will not be disclosed.

Please tick the relevant boxes and be as full and comprehensive as possible with your other answers.

Section A: Information about your technology use

Q1: Approximately, for how many years have you been using a computer for?

(Please enter a numerical value only)

Q2: I normally use a computer (please tick one)

Every day

A few times a week

Occasionally

Rarely/never

Q3: I have access to broadband (please tick all that apply)

At home/student residence

At work

At University/College/Learning Centre

Other Location (please state)

Q4: I normally access the Internet to look for information(please tick one)

Every day

A few times a week

Occasionally

Rarely/never

Q5: Approximately, how many hours a week do you spend at home or somewhere else on the Internet including using the internet(for work, recreational and educational purpose)? (Please enter a numerical value only)

yes no does not seem to fit this question and some others further on -Bronwyn 6/3/09 4:14 PM

Yes

Q6: When I use a computer, I customise it to suit my personal preferences e.g. background colours, icon sizes, mouse buttons, menu items, size of print on screen:

If yes, please briefly list the changes you most often make:

Yes

Use of online tools

Q7: I use social networks (E.g. MySpace, Flickr, Facebook)

If yes:which do you use?

Q8 How often do you use synchronous (At the same time) chat tools? (please tick one)

Every day

A few times a week

Occasionally

Rarely/never

Yes

Q9: I use messaging and discussion tools (E.g. Email, forums, phone texting)

If yes which do you use?

How often do you use messaging and discussion tools? (please tick one)

Every day

A few times a week

Occasionally

Rarely/never

Yes

Q10: I play online games or use virtual worlds and talk to other players (E.g. World of War Craft, Battlefront 2, Sims Online, Second Life)

b) Which do you use?

How often do you play online games or use virtual worlds (please tick one)

Every day

A few times a week

Occasionally

Rarely/never

Yes

Q11: Do you have a personal website/blog?

If yes: which do you use?

How often do you use your online personal space? (please tick one)

Every day

A few times a week

Occasionally

Rarely/never

Yes

Q12: I use other social networking and communication tools online (E.g. Online dating, Friends Reunited, ebay)

If yes:which do you use?

c) How often do you use your other social and communication tools online? (please tick one)

Every day

A few times a week

Occasionally

Rarely/never

Q13. Overall, how confident are you in the use of computer-based and Internet-based communication and information methods?.

Extremely unconfident

unconfident

confident

extremely confident

Section B Survey on Trial interactive learning resource

With regards to the MecMovies interactive learning resource that you trialled in class can you answer the following questions. Please be honest and as critical as you wish, The results are fully confidential. Your comments are most valuable

1 The instructions for using the resource were easy to follow

Strongly agree

Neutral

Strongly disagree

Comment

2 To navigate around the resource was easy

Strongly agree

Neutral

Strongly disagree

Comment

3 the resource was a helpful way to learn.

Strongly agree

Neutral

Strongly disagree

Comment

4 I feel that I learnt something from the resource

Strongly agree

Neutral

Strongly disagree

Comment

5 I would like to use more of this type of learning resource as part of my course

Strongly agree

Neutral

Strongly disagree

Comment

6. what sort of support do you believe you will need if you are asked to use interactive learning resources (such as the one you trialed) regularly in your course?

Comment

7. what type of skills do you believe you need for eLearning (online- Internet, multimedia, computers)

Comment

Section C: Personal Details

Name:

Age: (Please enter a numerical value only)

Gender: Male Female

Which program do you intend to study

Engineering Electrical Engineering mechanical

Employment/education Status:

In full time education In part time education Not in education

Appendix 2 Interview of tutors and industry expert

Interview questions

This interview is for the purpose of investigating the best method to introduce elearning into a mechanical engineering course that is to have a blended delivery. You opinion is confidential and valued

For the industry expert

Skill levels For the following table, place a tick in the skill level you feel that an engineer must have in today’s modern engineering concern. for each of the items listed

		Very skilled	Skilled	Unskilled	Very unskilled	Do not use
Software applications	Word processing
	Excel spreadsheet
	Presentation software (power Point
	Creating and editing video/audio
	CAD CAM
ICT	Email
	Web Forums
	Professional social networking
	Wikis blogs
Digital information	Information retrieval -= searching on the internet
Digital information	Ability to use on ine library resources

Do you feel it is necessary for engineers to be able to be able to program in a software language?add ? to all these questions. -Bronwyn 6/3/09 4:16 PM

Do graduate mechanical engineers use web based software.

What forms of software are used in industry. Ie CAD CAM Mat lab etc

Can these skills be developed while working – is this preferable?

How much web based / internet usage would an engineer expect?

Questions for tutors

Do you feel you have the skills, time to use e-learning software

What is your opinion on the introduction of elearning into engineering programs

How much e-learning should there be %?

What type?

How should it be used

Do you feel that students will enjoy using elearning

Will it require students to up skill?

Will it require teacher up skilling

What drawback or difficulties do you foresee

What advantages are there

Are you or other staff members able to develop and modify electronic interactive learning software for appropriate use and level in the classroom

Do you feel that there is sufficient technology support to be able to deliver e-elearning in a blended course?

Bibliography - good list and using APA referencing - best to put this before the Appendix

Huffaker, D. (2003). reconnecting theclassroom:Elearning pedagogy in US public high schools. Australian Journal of education Technology , 19 (3), 356-370.

Mason, R., & Rennie, F. (2006). Elearning the key concepts.

Massey University. (2006~~, june~~). Guidelines for the support of e-learning in New Zealand tertiary institutions. Retrieved may 7, 2008, from elg: http://elg.massy.ac.nz

Rossett, A. (1995). Needs Assessment. In G. Anglin (Ed), Instructional technology: Past present and future (2nd Ed.) (pp 183-196) Englewood CO: Libraries unlimited.. Englewood Co libraries unlimited.

~~Tom,~~ Reeves, T. (n.d.). Retrieved feb 5, 2006, from http:/www.educationau.edu.au/archives/Cp/refs/reeves_paradigms.htm

[1] This tool is not to be used in this evaluation

Pradeeps evaluation Plan

2009-05-10T13:24:00.000-07:00

Title: Evaluation of the Introduction of e-learning into engineering mechanics

I have posted my final proposed evaluation on to google docs Please comment and give suggestions.

Summary of EASEIT-ENG project

2009-04-17T11:11:00.000-07:00

Summary of
Development of a practical Evaluation Methodology for the production of Engineering learning Technology Case Studies by the EASEIT-ENG project.

I found this paper with its reference to engineering and evaluation. EASEIT-ENG stands for Evaluative and Advisory Support to Encourage Innovative Teaching – Engineering. A group established in the UK whose aim is to assist engineering academics in selecting learning technology material, for use in their teaching, that was right for both them and their students. They have done this by getting a number of tertiary education providers to do evaluations of engineering computer based learning and teaching materials they are using or trailing. These evaluations are then put in a data bank and made available to academic tutors. The document summarizes each aspect of the project and provides references.

The project puts together a manual form which evaluations of can be carried out. The manual also includes
1. Client contact details
2. Fact sheets
3. Pre-evaluation questionnaire
4. Student question
5. Observation schedule
6. Guidance on focus groups
7. Tutor interview structures
8. Evaluation report template and feedback questionnaire
I.e. a lot of the same type of data collection that I (we) intend to do. The actual questionnaires are available online and will be a valuable resource as a template for my evaluation.

As a list of evaluated software it was only a little useful but the evaluation process that was described gave an idea of the detail required in doing an evaluation and that was very useful. It gave a summary of what the project revealed about engineering leaning technology materials which was very interesting and varied. One point that interested me was ‘that academics seemed convinced of the pedagogic benefits of using learning technology but were less convinced that the right material was available for them or that there would be any time saving associated with its use.”

Finally the reference section gave great reference to elearning materials for engineering.

Reference
EASEIT-ENG Project Phil Baker, Fiona Lamb, Steve Rothberg. (October 2005) Development of a practical Evaluation Methodology for the production of Engineering learning Technology Case Studies by the EASEIT-ENG project. The Higher Education Academy – Engineering Subject Centre.
URL http://www.easeit-eng.ac.uk/

Outline of Evaluation Project

2009-04-16T03:58:00.000-07:00

My evaluation project will be a needs analysis, with a mixed mode method for data collection. I will be evaluating introducing elearning tasks into upcoming engineering courses.

Needs Assessment -·
Who users are,
type of users·
cost-effective·
where the gaps are.

Reason for a 'needs analysis'
I believe that a review of current practice can reveal that little elearning is encompassed within the teaching of engineering. At the best there are electronic notes and assignments made available to students on an electronic filing cabinet. I would like to see more dynamic use of elearning in the form of quizzes and possible group discussions.
Students that study engineering tend to be kinesthetic learners and already use computers in some engineering courses like computer aided design, so they have a familiarity with computers. I suspect that the main reason elearning instruction is not more widely used is due to those that teach engineering not having the skill or knowledge to do so.
The strategic plan of this polytechnic is to provide a state of the art facility and learning environment. That must include the use of modern techniques (such as elearning) where appropriate. And to be student focused. So there is a directive from above that encourages elearning to be developed into our courses. Since we have the job of running the courses for the first time, we have the opportunity to do this.
There is a ‘need for analysis’ to find out if what I believe could be true and if elearning would be aligned with the strategic plan.

What is to be evaluated.
It is intended that elearning be introduced slowy, as instructors get up to speed with the technology. So a pilot is proposed, one which includes the use of interactive learning material with follow up quizzes, and a site which will allow instruction form f2f for review.
I would also like to check the viability of using a discussion board, I not sure how this could be used inside of the engineering curriculum but there may well be instances where it would be appropriate.

How the evaluation will work
It will use a 'multiple methods' model to collect data and information. Methods I aim use are:
Student survey –To find student demographics and learner types.
Focus groups – Get students to trial a quiz and give their opinion. I will endeavor to test if learning outcomes can be met. I will give them a small assessment before the trial and another assessment after the trail, to try and measure effective change to learning outcomes. Both test with a range of outcome levels and should give some quantifiable results. I am hoping to have 10 to 15 students in my focus group.
A separate survey after they have completed the trial to see what their feelings were on using elearning as part of their learning. This survey would have open ended questions and the results would have to be considered with consultation of other instructors.
Expert review - Peers to trial a quiz and be interviewed on their opinion.

Summary - Needs assessment by Rossett

2009-04-16T03:21:00.000-07:00

I picked this article on the basis that I intend to do a needs evaluation. I found Allison Rossett easy to read and the scenarios she used, I could relate to.

Needs Assessment By Allison Rossett - Summary

Rossett give a general overview of needs assessment with the emphasis on occupational instructional training. She give a number of example that are corporate based – one that runs through the whole discussion is of soft and limp French fries in a fast food joint.
She defines it as “needs assessments gather information to assist professionals in making data-driven and responsive recommendations about how to solve the problem or introduce the new technology. While needs assessment may lead to the development of instruction, it does not always do so. The important role that needs assessment plays is to give us information, at the beginning of the effort, about what is needed to improve performance. She defines five purposes for doing needs assessments

To define Optimal performance
To find out what is happening with actual performance
To know how learners feel about
To find the cause of performance problems
a. Lack the skill or knowledge
b. The environment is in the way
c. There are no or improper incentives
d. Lack of motivation
To find solutions

Rossett then talk about how to conduct a needs assessment giving 5 steps

Determine purpose based on Initiators
a. Performance problems
b. New stuff/technologies
c. Mandates from above
Identify sources
Select tools
a. Interviewing
b. Observing performance
c. Examine records and outcomes
d. Facilitating groups
e. Surveying through questionnaires
Conduct the needs assessment in stages
Use findings for decision making.

In her conclusion she remarks the
“Deciding to observe interview or survey isn't the hard part. The challenge is to determine sources, plan stages of assessment , and frame questions that will enable a professional to move crisply form a general problem with limp French fries to a series of actions and recommendations that will guarantee yummy ones. “

summary of "evaluation and elearniging" by Debra Peak

2009-04-13T22:39:00.000-07:00

Week 5 & 6
Review
Turkish Online Journal of Distance Education-TOJDE January 2006 ISSN 1302-6488 Volume: 7 Number: 1 Article: 11 by Debra PEAK & Zane L. BERGE

This article by peak considers levels of evaluation as modeled by Kirkpatric in 1994. Which I had to look up as follows

Kirkpatrick's Four Levels of Evaluation
In Kirkpatrick's four-level model, each successive evaluation level is built on information provided by the lower level.
ASSESSING TRAINING EFFECTIVENESS often entails using the four-level model developed by Donald Kirkpatrick (1994). According to this model, evaluation should always begin with level one, and then, as time and budget allows, should move sequentially through levels two, three, and four. Information from each prior level serves as a base for the next level's evaluation. Thus, each successive level represents a more precise measure of the effectiveness of the training program, but at the same time requires a more rigorous and time-consuming analysis.

This paper compares evaluating elearning and traditional classroom instruction evaluation. It consideres why level 4 evaluations are so difficult to do and why elearning evaluation has evolved to include return-on-investment claculations.

Peak suggest that the main difference in the two delivery methods centers on the data collection methods. Both level 1 and level 2 evaluations(student evaluations and student results) are easier to build into elearning courseware - due to the use of computer technology.

Level 4 evaluation - results - measures the success of the program (Elaine C Winfrey). Peak ask why are these difficult to measure and suggest two reasons

Organisations think they that the results must be quantifiable and that a negitive result implies that training has failed.

"Trainers belive that they must be able to show proof beyond a reasonable doubt that business improvements were a direct result of training." (peak)

There is now a push for level 5 evaluations. ROI evaluation are liked by managers and senior executives. Peak points out that there is a difficulty in converting educational outcomes into monetary values and many organisations do not even attempt too. Peak then gives steps towards finding a rate of return.

Peak concludes with this statement in support of ROI evaluations. 'Regardless of what is measured or how, the consensus seems to be that what is important is that business values are finally being attached to the corporate learning experience. Holly Burkett, an ROI evaluator at Apple Computer, stated “For me,
it’s more empowering to know that our department’s work has a direct impact on performance, productivity, or sales than it is to know that people enjoy the training program” (Purcell, 2000).'

Ref

Turkish Online Journal of Distance Education-TOJDE January 2006 ISSN 1302-6488 Volume: 7 Number: 1 Article: 11, 'Evaluation and eLearning' by Debra PEAK & Zane L. BERGE
UMBC, 1000 Hilltop Circle, Baltimore MD 21250

Elaine C. WinfreyGraduate StudentSDSU Educational Technology
Winfrey, E.C. (1999). Kirkpatrick's Four Levels of Evaluation. In B. Hoffman (Ed.), Encyclopedia of Educational Technology. Retrieved
April 14, 2009, from http://coe.sdsu.edu/eet/Articles/k4levels/start.htm

Week 4 Education Evaluation paradigms

2009-03-20T21:59:00.000-07:00

When I read the article on paradigms by Professor Tom Reeves. I thought that some one was having a joke, they had put together a lot a big words, mostly jargon, into paragraphs to try and baffle me, which they did successfully and then made conclusions about a group of people, as if they existed, about their view on humankind. After I read the other article Compare two different models put together by Bronwyn - I couldn't help thinking why Professor tom Reeves could not achieve the same degree of clarity.

What I understand is that teaching and learning has a vast number of variables, many of which can not be kept constant while studying a particular one. So quantitative methods alone may give misleading solutions. The use of qualitative and objective methods alongside of quantitative methods may help to alleviate that concern. So the use of 'multiple methods evaluation model' as developed by Mark and Shortland in1987 seems to me to make the most sense. Why would you limit yourself to only one form of evaluation. As an engineer we learn that even in engineering things are not absolute, as much as we like to quantify things we also understand that in the end not 'all the variables' can be accounted for and the solution is just a 'best guess'. So many time you end up looking at a problem from different angles and seeing what the common threads are, so as to give you a picture of best fit. I can see evaluation being like this also. So look at a teaching resource with as many evaluation tools as possible, to get as best a picture as possible makes sense to me.

Week 3 Guide lines

2009-03-17T22:27:00.000-07:00

Firstly this week has helped me define my project a bit further. I will be doing a needs analysis on a elearning resource intended for use in upcoming engineering course. The intention is to add a resource into one particular topic area and at this stage continue to maintain f2f learning for the rest of the course, ie a slow introduction of elearnign into the course. So once this was decided coming up with two guidelines was easy.

The two guidelines that I selected from the elearning guidelines for NZ that interested me are

TD1 Is the use of e-learning the best way to achieve the intended learning outcomes?

What I have noticed in the teaching of Engineering is that currently there is not very much elearning being practised, my research is not extensive so I may be proven wrong. It certainly does not exist where I have been teaching. The reasons may be due to lack of knowledge and skills on the part of tutors about elearning and a fear of increased workloads due to the initial setting up of the resource. We do teach high level software applications in the form of Computer aided Design programmes, so students are and need to be familiar with computers. So you would think that the next step would be to use elearning as part of the teaching process

Link to guideline TD1

The second Guideline is
SD3
Do students gain knowledge relevant to employment and/or current thinking in their field?

link to the guidelines click here http://elg.massey.ac.nz/index.php?title=SD3

At Bahrain polytechnic one of our primary objectives is to produce work ready students. So the elarning resource should confirm with that.

Evaluation

2009-03-11T06:30:00.000-07:00

Evaluation

Week 2
I have read a few of the responses already made and to avoid repeating them have taken a slightly different approach

Why is evaluation important to you and how do you define it?
Why is quality important in eLearning?

For me evaluation is about checking that something is fit for purpose. In the case of a educational course, that the objectives and outcomes are met.

Traditional evaluation in f2f teaching generally occurs at the end of a course as course reviews or programme reviews. This is probably suited for traditional course because our expectation of results from those course are seasoned by what has happened in past courses.

With regards to an elearning resource, course or programme the evaluation is there to check that learning is occurring as we suspected it would. We have expectation of new technology to deliver better results than our traditional teaching, otherwise why would we be using it. This expectation can lead to disappointment as actual results do not exceed what we have come to expect from traditional teaching. Unforeseen problems happen and our deliverables do not meet our expectations. So it is very important to ensure that the quality of the course is of a high standard and that quality is being maintained throughout the course. High quality standard also help to identify areas that require attention.

Quality and Evaluation are very intertwined, to maintain quality there needs to be regular evaluation - evaluation is the tool to checking quality. Unfortunately evaluation has been percieved as a tool for catching lame tutors and in fact at MIT student evaluations were linked to salary increases.

What sort of evaluations mentioned on the presentation are familiar to you already and why?

I have not experienced much formal evaluations. Of course I have encountered the summative evaluation that students give of both the teaching and the course at the end of the course. We used this at MIT primarily to check on student satisfaction, coupled with promotion up the career path.

We also had set up moderation process for assessments, which is a type of evaluation, in that it does check that the curriculum has been covered in sufficient depth. This is t check that our qualification standards confirm with other polytechnics around the country.

There was informal evaluation taking place in the form of one’s own work, or like a reflective appraisal of teaching strategies, to see what worked and if any improvement could be made.