According to the definition, as provided in the Merriam Webster dictionary (2006), evaluation is" to determine the significance, worth, or condition of usually by careful appraisal and study". In Instructional Technology the purpose of evaluation is also to determine the worth of something through systematic analysis and study. However, as with all areas of Instructional Technology, the evaluation process is more prescribed, systematic and specifically focused on data in order to make sound decisions .
More specifically, in the field of Instructional Technology, evaluation is defined as "the process of determining the adequacy of instruction and learning" (Seels & Richey, 1994, p. 54).
There are four specific sub-domains of evaluation in Instructional Technology. They are described below.
Problem Analysis:
Best practices would dictate that instruction should only be designed, delivered and implemented if there is some sort of performance problem to be solved, regardless of degree or severity. In order to solve a performance problem the performance gap must be specifically identified. The performance gap is the difference between what is currently happening and what should be happening in any given situation. In order to accurately and effectively identify the performance gap a thorough problem analysis needs to be conducted at the beginning of the instructional design processes. Problem analysis includes: needs assessment, performance analysis, and a contextual analysis (Rossett, 1987). The needs assessment procedures are employed to specifically determine the gaps between the current performance and the ideal performance. Needs assessment procedures include a great deal of information gathering from a variety of sources pertaining to the performance problem. The performance analysis is employed to determine if the problem is an instructional problem that can be solved through instruction. This process may also be called Training Needs Assessment or (TNA) (Rossett, 1987). According to Allison Rossett, TNA (1987) is "the systematic study of a problem or innovation, incorporating data and opinions from varied sources, in order to make effective decisions or recommendations about what should happen next" (p.3).
According to Rossett (1987), in order to conduct a problem analysis a designer should complete each of the following steps:
Common analysis techniques as described by Rossett (1987) include:
Common analysis tools as described by Rossett (1987) include:
Assessment of Learning Outcomes/Criterion Referenced Measurement:
Criterion referenced measurement are assessment items that are created specifically to measure if the learner has mastered a predetermined criteria as stated in a learning objective (Dick, Cary & Carey, 2001). Criterion referenced measurement will provide information on the degree to which the learner has mastered the content or objective of the instruction. (Seels & Richey, 1994). A predetermined mastery level or score is established to determine if the learner has mastered the content. Unlike norm-referenced measurement where learners are scored relative to the performance of other learner, criterion referenced measurements determines each learner's mastery against predetermined criteria (Wikipedia, 2006). Criterion-referenced measurements will allow the learner to know how well they perform relative to a standard as opposed to comparing learner to learner.
Formative Evaluation:
"Formative evaluation involves gathering information on adequacy and using this information as a basis for further development" (Seels & Richey p. 57). Formative evaluation is utilized in the early stages of product development. Formative evaluation may be considered the "while in process" or during "formation" evaluation process. There are several purposes of formative evaluation, which include: (Dick & Carey, 2001)
During formative evaluation a designer will utilize the instructional materials to conduct a one to one evaluation then a small group evaluation and finally a field test will follow (Dick & Carey, 2001).
During these evaluation stages, the designer will observe the learners moving through the instructional content and record any changes that need to be made to instruction. Edits and improvements will be completed after each evaluation therefore instructional materials are revised several times, as needed. During a formative evaluation process both qualitative and quantitative data may be collected. The formative evaluation information is collected, analyzed and used specifically to make revisions and improvements. According to Dick and Carey, the formative evaluation information is not utilized solely to make revisions to the instruction. It is also used to "reexamine the validity of the instructional analysis and the assumptions about the entry behaviors and characteristics of learners" (Dick & Carey, 2001, p,8). During this critical stage in the instructional design process the designer will also reexamine the performance objectives, assessment items, and the instructional strategies.
Summative Evaluation:
"Summative evaluation is a method of judging the worth of a program at the end of the program activities. The focus is on the outcome" (Bhola,1990).
A summative evaluation is conducted after implementation of an instructional product, project, or program. As in a "summary" of the cumulative parts of the instructional design implementation. Summative evaluation entails gathering information on adequacy of the instructional product or program and using this information to make decisions about the effectiveness of the instruction. During summative evaluation processes the designer will ask and answer the question-did the instruction solve the problem (Dick & Carey, 2001).
The purposes of a summative evaluation include:
During the summative evaluation processes both qualitative and quantitative data gathered. The results of a summative evaluation may yield improvements in the instruction or program.
Donald Kirkpatrick's (1979) Four-Level Summative Evaluation Model may be utilized by a designer for summative evaluation. The model includes the stages of: reaction, learning, retention and results as seen in Figure 12.
Kirkpatrick's Evaluation Model
Figure 12: Kirkpatrick's Evaluation Model
Image adopted from:
http://coe.sdsu.edu/eet/Articles/k4levels/index.htm
1. Reactions-did the learners "like: the instruction? This may informally be referred to as the "smiley test".
2. Learning-did learning occur as intended? Did the learners master the performance objectives?
3. Transfer-did the learning transfer in the performance environment/context? Did the instruction yield a change in the learner's behavior?
4. Results-did the instructional program do what it was designed to do? What are the long term impacts and outcomes of the instruction?
Another model that is utilized by evaluators is the Context, Input, Process, Product or CIPP Model of evaluation created by Daniel L. Stufflebeam (1983) (see Figure 13).
CIPP Model
Figure 13: CIPP Model
Image Adopted from:
http://wwledgebank.irri.org/cglrc/icraf/toolkit/The_CIPP_evaluation_model.htmw.know
1. Context-describes the organizational environment in which the innovation has been implemented
2. Input-describes the contributions into the system
3. Process-describes how the innovation is being implemented
4. Product-describes the output or outcome of the innovation
Introduction About Me History & Definition Domains Competencies Artifacts Resume References Works Cited Glossary