Employee Training and Resource Module

Report III

Printable Version (PDF)

Section 1: Results of Evaluation

Part 1: One-to-One Formative Evaluation

1)  The one-to-one formative evaluation was conducted at the GOGAS training facility after the employees finished participating in a formal meeting.  Six GOGAS employees were chosen for the one-to-one evaluation because two of them were each considered to have a low, moderate and high knowledge base for computer hardware and software.  The materials were presented to them in a black and white printed format with several color copies of the materials for them to look at.  The materials had to be presented in this way because it is very expensive to print color copies, so to minimize the expense black and white copies were given to each learner and color copies were provided for them to see the how the final manual will look.  The learners were all female station managers ages 45-75 years old with varying computer abilities and knowledge.   The learners were all literate and able to read and comprehend the materials.  The evaluation was conducted after lunch was served following the conclusion of their meeting.  The learners were instructed to read the materials and follow the directions.  Each learner sat at a table with several others so that they could all see the color copies.  The learners were provided with answer sheets to complete the pre and post tests as well as a feedback questionnaire.  It took the learners approximately 1 hour to complete the module.  The process was monitored by the SME and the instructional designer (me).  We tried not to give any instruction other than reiterating that they must read the instructions and follow them.  Several of the learners looked through the module as they were filling out the pretest so that they could put correct answers on the answer sheet.  We informed them not to do that because we wanted to assess how much they had learned, but several of them did it anyway. When everyone had finished the module and the questionnaire, there was a group discussion about the learners’ likes, dislikes and recommendations pertaining to the layout and content of the module.   It was not possible to talk to each employee individually because of time constraints, but the information that was obtained in the group discussion was very valuable and informative.

2)  The results of the one-to-one evaluations are a compilation of verbal responses to questions and suggestions.  Overall the learners were very pleased with the instruction although they said that the black and white copies were hard to read, which we expected since some of the text was color-coded to match the captions of the illustrations.  Many of the learners did know what individual components were used for since they use these in their stations, but they did not know the correct terminology.  This is useful information because it is just as important for them to know how the components are correctly labeled, as well as how it is used.  Knowing the correct terminology will enable them to communicate with other individuals more effectively because everyone will know what is being talked about.  Many of the employees had different names for different things which left a lot of information open to interpretations especially when the tech support person is trying to troubleshoot over the phone.  If that person can’t see what the cashiers or manager is talking about then they are left to try and decipher what that employee is trying to say.  If the correct terminology is used then communicating the solution to a problem is much easier.   The results of the pre and post tests are not valid because the learners looked through the module for the answers, so they cannot be used.  The embedded test items were not consistently answered and without the pre and post test results it is impossible to assess how well they learned the material from each section.  The results of the questionnaire indicate that in general they learned something from the module, consider it to be useful, easy to follow, something they can use in their job, they were motivated to complete the module, they would use it in their stations if they had an issue, but they may still call someone for help and that they like this form of training.  The revisions that will be made as a result of these evaluations are taking the pre and post tests out of the module and putting them in a separate folder so that the learners are less likely to look through the manual when they are filling out the tests.  The materials also needed to be spaced out more so that they was not so much information on one page.

Part 2: Small Group Evaluation

1)  The small group evaluation was conducted in the same manner as the one-to-one evaluations.  It was not possible to conduct separate evaluations due to time constraints and the convenience of having all employees together at one time in the training facility.  The alternative was to travel to each station and conduct several very small group evaluations over several days or weeks, which would have taken much more time and money (for gas).   The 6 learners were all station managers ages 45-75 years old with varying degrees of computer knowledge.  The difference in the group evaluation and the one-to-one was that the group left after completing the module and questionnaire.  Two of the group learners did not complete the questionnaire at all. 

2)  As stated above, the materials were black and white individual copies for each learner with color copies that they shared.   All learners were given the printed manual with pages stapled together in order and a packet that included two answer sheets (one for the pretest and one for the posttest) and the feedback questionnaire.  They were provided with pencils or pens to record their answers.      

3)  The evaluation took place after a formal meeting at the training facility located at the GOGAS headquarters.  All of the learners were assembled in the training room and told to sit somewhere that allowed them access to a color copy.  Each learner took a sit and began looking through the manual.  They were instructed to start with the first page, carefully read through the instructions and then take the pretest.  The learners began filling out the test and looking through the module for the answers.  We told them not to do this and explained why, but most of them insisted on doing that anyway since they had already answered several of the questions.  Needless to say, the tests are invalid.  When the learner’s finished they either left or hung around to wait for others.  It took most of them around 1 hour to complete the module. Six of the participants stayed to answer questions, which served as the one-to-one evaluation.  When everyone was finished they were thanked for their participation and they were on their way.  The answer sheets were turned in when the learner had finished.

(To view the feedback questionnaire click here)

Table 1: The average answer given for each question in addition to the most frequent response.

(Click here to view the frequency tables for each question)

4)  This answers to questions 1-7 ranged from 1 – “Strongly Disagree” to 5 – “Strongly Agree”.  The last questions had values that ranged from “Not at all useful” to “Extremely useful”.  The data shows that on average, the learners agreed that the materials were easy to follow and read, which a good thing to know since the readability and comprehension of the materials by the learners was of some concern.  The learners, on average, felt that they could use the information in their job; they were motivated to complete the module; they did not dislike this training and they did learn something from the module.  The highest mean score was from the question regarding whether they would use the manual in their station.  All of the learners answered the question and most of them answered “Strongly agree.”  The question that received the most neutral responses was that they would still call someone even if they had the manual.  This answer does not come as a surprise considering the limited scope of the manual compared to the wide range of computer problems that could arise in their stations.  It does appear that they would use the manual if an issue arose that was covered in the material.  Overall, the data clearly shows that the manual will be utilized as a resource within each station and the learners did learn something from the material.  Unfortunately the data is incomplete because of the invalid pre and post test results.  The embedded assessment items were not recorded consistently by any of the participants, so there is no data to report.

5)  There will be several revisions made to the module including the separation of the pre/post tests from the module.  Because of the lack of information that was gathered in the small group evaluation, the module will be revised based on the one-to-one group recommendations and then given to another learner group in the future. The next group session will be more structured and the information will be gathered in a more systematic way instead of letting the learners just turn in the materials after they were finished.  That way each assessment can be gathered individually after the lessons are completed to ensure that the instruments were, in fact, completed before the learner moves on to the next section.  If there were time enough in the semester this re-evaluation would definitely take place in both a group and individual one-to-one format.  The fact that this module will be used in each station for both existing and new employees, it is crucial that the materials be effective.  The data is not available to confirm that the instruction is effective yet.  Based on the preliminary results that are presented here, the attitudes of the employees and the usability of the manual are positive, which indicates that this is a worthwhile project that needs further work. 

6)  There will be many revisions made to this module because it will actually need to be used in each of the 19 GOGAS stations.  Due to the lack of data, it is hard to say exactly what revisions are left to be made.  There will be more information added, which will help the employees troubleshoot computer hardware and software issues, but overall the manual is on the right track according to what information we do have. 

7)  Again, it is hard to speculate about what revisions will need to be made based on the data, but after the current revisions are made another learner group will be assembled.  The results of that evaluation will be more comprehensive and directive in terms of what final revisions need to be made before the final production of the manual.

8)     The instructional procedures as far as the group and one-to-one evaluations are concerned will be revised.  The next learner groups will be more structured in terms of gathering the assessment data from each lesson, as well as the pre and post tests.  I am also hoping to be able to offer the learners color copies instead of black and white since it does take away from the effectiveness of the instruction based on what the learners said in the one-to-one discussion.

Section 2: Planning and Time Log

Part 1: Symbols

This is the only symbol that I used in the manual to indicate to the learner that there was a question that needed to be answered.

List of Terms:  This list is also provided in the glossary of the module

1. CRT: Acronym for cathode-ray tube. A display device used for computers and televisions. 
2. Desktop: Contains the visible elements of Windows and defines the limits of the graphic environment.
3. Control Box:  Used to change the state of the application. It can be used to maximize, minimize, restore, and close the application.
4. Close Button: Used to easily end a program or close a window.
5. Minimize Button: Used to minimize a program or window to an icon on the taskbar.
6. Maximize Button: Used to make a window or program take up all available space on the desktop.
7. Restore Button: Used to restore a window to its previous state. Such as restoring it from the minimized state on the taskbar to its original state.
8. LCD: Acronym for liquid crystal display. It is a thin, flat screen used in laptops, home computers, and flat panel TV’s.
9. Floppy Drive:  A device used to read and write to floppy disks. Commonly called the A drive.
10. Zip Drive:  A device used to read and write to zip disks. Zip disks are similar to floppies, but hold more information.
11. CD ROM Drive:  Acronym for compact disk read-only memory. Optical storage device that uses a compact disk to read and write information.
12. Dot Matrix Printer: An impact printer that uses columns of small pins and an inked ribbon to create the tiny pattern of dots that form the characters on the paper.
13. Laser Printer: A printer that uses high voltage, a laser, and a black toner cartridge to form an image on a page.
14. Ink-Jet Printer: A printer that uses an electric signal to energize a heating element causing the ink to vaporize and get pushed out of a tiny hole onto the paper.
15. Mouse:  A small input device with one or more buttons used for pointing or drawing. As you move the mouse, a on-screen pointer follows the mouse movement. When you click a button, a program specific action occurs.
16. Keyboard:  A device that translates keystrokes into letters or numbers.
17. Modem: A device that allows a computer to transmit information over a telephone line or cable connection.
18. Taskbar:  The area of a Windows desktop which includes the Start button and the system tray, as well as icons for any open programs.

Part 2: Time Log

Activity	Comments	Time
Needs Assessment	This part was easy because the company had already conducted an informal needs assessment.  I did however discuss the performance gap, as well as needs of the client and constraints of the project.	Approximately 4 hours.
Instructional Analysis	Deciding how to actually format the manual and how the information should be presented was a challenge but an interesting learning experience.	Approximately 15 hours.
Learner Analysis	I enjoyed conducting the learner analysis because I got to actually visit the sites and talk with the employees. Creating surveys and analyzing data is also an interest so this was a portion of the project I really enjoyed.	Approximately 18 hours.
Context/Environmental Analysis	All of the contextual information was obtained from the observations that I made in the stations and the home office and talking to the employees, which I found very interesting and fun.	Approximately 10 hours.
Constructing Task Analysis	This was a difficult task because I was learning how to break apart each task I wanted to teach our learners and figure out exactly what concepts I wanted them to learn.	Approximately 15 hours.
Write Performance Objectives	This also was a difficult task because correctly phrasing the objectives and having all the components that they need is difficult to master.	Approximately 20 hours.
Developing Assessments Instruments	Deciding on the format to use for the assessments and actually creating the assessments was a timely process.	Approximately 12 hours.
Developing Instructional Strategy	Determining how the learners would best benefit from the instruction took a lot of time and effort.	Approximately 20 hours.
Developing Instructional Materials	Many revisions have been made to the instructional materials and I see how this is an ongoing process and includes aspects that I never foresaw.  This was probably the hardest part of the entire project because of having to work with the SME for extended periods of time, since there were only 2 of us working on this project.	Approximately 40 hours.
Designing Formative Evaluation	Finding a time where our learners could all meet and actually go through the module was a challenge and also happened later in the semester which gave me little time to analyze data and make the needed revisions.	Approximately 15 hours.
Conducting Formative Evaluation	This process gave me a lot of insight as to what our learners were expecting from this module as well as the information that needed to be covered in the module to best help them in their jobs.	Approximately 12 hours.