The Domain of Evaluation is the process for determining the adequacy of instruction and learning according to Seels & Richey (1994, p.54). The role of the instructional designer during the evaluation process is to first classify the object being evaluated. Some of the most common classifications are programs, projects and materials. An example of a program evaluation in the educational environment is the reading program adopted by a school system. The instructional designer evaluates the educational activities and services provided by the reading program. Next, an example of a project evaluation is a two-day training program on instructional strategies for newly employed teachers. The instructional designer evaluates the activities during and after training. Lastly, in material evaluation, the Instructional Designer evaluates the value or significance of textbooks, curriculum software programs, and other tangible instructional products. The evaluation phase is driven by a search for the answers to the following questions: Have we solved the problem? What is the impact? And what needs to change?

The practitioners working in the evaluation domain base their decision making on instructional models that are based on research and theories in the discipline of Instructional Technology. Instructional designers working in the domain of evaluation have numerous models to utilize in their decision-making process. The Dick and Carey is one model frequently used by designers in this domain. The model focuses on formative and summative evaluation. Formative evaluation, which was proposed by Cronbach and Scriven, is the process of collecting information during the development of instruction to be used to improve the effectiveness of the instruction. The purpose of formative evaluation is to identify problems in the instruction and gather information from learners in order to revise the materials before the instruction is complete. The Dick and Carey (2001) model states that there are three means of formative evaluation: one-to-one, small-group, and field evaluation. The initial evaluation is called a one-to-one evaluation. At this stage the designer works individually with three or more learners in order to obtain information and identify errors in the instruction. The learner's initial reactions to instruction are documented and used to help in the revision process. The next type of formative evaluation is the small group. The designer works with a group of eight to twenty learners to help determine the effectiveness of changes made following the one-to-one evaluation and to identify any remaining learning problems that learners may have . Another purpose of using a small group is to determine whether learners can use the instruction without interacting with the instructor. The third type of evaluation is the field test, which is used to judge the effectiveness of the revisions made in the small-group trial. Generally, the field test is conducted with approximately thirty learners who are representative of the target population (Dick & Carey, 2001). The field test is much like the final dress rehearsal since the instruction is revised and delivered in a manner that is close to the final format. According to the Dick and Carey model, summative evaluation is the process of collecting data and information in order to make decisions about the adoption or continued use of instruction with target learners. The main purpose is to make go-no-go decisions about maintaining currently used instructional materials. In addition, there are two other evaluation models that are widely respected in the field, James D. Russell's evaluation model and Arthur Anderson's CBT formative evaluation model.

Kirkpatrick’s 4-level evaluation model, developed in 1976, is another model used by Instructional Designers in the domain of evaluation. The four levels of evaluation are: reactions, learning, transfer and results. According to this model, evaluation should always begin with reactions (level 1), and move sequentially through learning (level 2), transfer (level 3), and results (level 4). Information gathered from each prior level serves as a base for the next level’s evaluation. Evaluation at level 1 – reactions, measures how participants react to training. It attempts to answer questions regarding the perceptions of the participants. Did they like it? Was the material relevant? Overall evaluation attempts to assess the learner satisfaction. In level 2 evaluation – learning, evaluation assesses the amount of learning that has occurred in a training program. Tests and pretests are used at this level in the evaluation process. Level 3 evaluation - transfer, measures the transfer that has occurred in learner’s behavior due to training. A question a designer may ask at this level is: are the newly acquired skills, knowledge, or attitude being used in the learners everyday environment? If yes, transfer has occurred. Level 4 evaluation – results, measures the success of the training from a managerial level in terms of increased production, improved quality, decreased costs, and return on investment.

Figure 1: The visual was adopted from coe.sdsu.edu/eet/Articles/ k4levels/start.htm

Problem Analysis

Problem analysis involves determining the nature and parameters of the problem by using information-gathering and decision-making strategies (Seels & Richey, 1994, p. 56). Instructional designers analyze, and interpret data to modify and improve instruction, projects or materials. Several terms are used for the process of analyzing a problem: front-end analysis, problem analysis, training needs assessment, or needs assessment (Rossett, 1987). Rossett states that the purposes of the analysis is to seek information about optimal versus actual performance or knowledge, determine the feelings of those involved. She states that it is important to get many perspectives on the apparent causes of the problem and possible solutions to the problem. The purpose of a needs assessment is to use a systematic way of looking at “what is” and how “things” “should be” in a system. These “things” are usually associated with organizational and/or individual performance. The “gap” between “what is” and “what should be” are identified as the problem and used to determine if training is necessary and if so, select the appropriate training needs. In summary, the importance of a needs assessment in the domain of evaluation are:

* To perform a gap analysis
* To identify and prioritize a list of needs for training and development
* To identify the cause of the performance problem
* To identify possible solutions to the problem

Criterion-Referenced Measurement

According to Seels and Richey, criterion-referenced measurement involves techniques for determining learner mastery of pre-specified content (1994). This type of evaluation lets the learner see how they performed based on a set standard. The criterion for determining adequacy is the extent to which the learner has met the objective instead of comparing learners to one another.

Formative Evaluation

Formative evaluation involves gathering information on adequacy of instruction and using this information as a basis for further development. It is done with a small group of people to "test run" various aspects of instructional materials. It is like having someone look over your shoulder during the development phase to help you catch things that you miss. At times, you might need to have this help from a target audience. For example, if you're designing learning materials for an eighth grade language arts class, you should have an eighth grader as part of your formative evaluation.

Summative Evaluation

Summative evaluation is a method of judging the worth of a program at the end of the program activities. For example, did the learners learn what they were supposed to learn after instruction? In a sense, it lets the learner know "how they did," but more importantly, by looking at how the learner's did, it helps you know whether the product teaches what it is supposed to teach.

back to top