IDEAS Grants logo

Abstracts

STScI Education

Evaluation


NASA Logo

   

IDEAS * IDEAS 1997

ELEMENTS OF A PROGRAM EVALUATION

What is Program Evaluation?
Evaluation Types
Choosing a Methodology
Choosing a Data Collection Method(s)
Analyzing Data
Reporting Evaluation Findings
Developing an Evaluation Plan
Evaluation Guidelines
Bibliography

WHAT IS PROGRAM EVALUATION?

By definition, program evaluation is "the systematic application of social research procedures for assessing the conceptualization, design, implementation, and utility of social intervention programs." In practice, program evaluation plays a significant role in program development and assessment. From concept to planning, application to results, the systematic evaluation of each step of a program serves as a mechanism to developing a realistic program that is clear, comprehensive and measurable. A solid program evaluation will also expedite the dissemination and publication process. Some key areas in which program evaluation methods could be beneficial include:
  • new program designs and development
  • program management and tracking
  • efficiency of program implementation
  • accountability
  • dissemination
  • program effectiveness
TOP

EVALUATION TYPES

There are three types of program evaluation: process, outcome and impact. Each type provides different information. In order to choose what type of evaluation would be most appropriate, it is necessary to determine what questions must be answered by the program. Below are explanations of each type of evaluation as well as the question(s) that each should answer.
  • Process (also known as accountability or monitoring) addresses the way(s) in which a program is implemented as well as the conditions under which the program is taking place. A process evaluation can also assess the materials and activities that are being developed for content and implementation appropriateness. Three questions that process evaluation asks are:
    1. Is the program reaching its target population?
    2. Is the program being administered consistently and in accordance with the program's specifications?
    3. What resources are needed to implement the program?
  • Outcome is the most common type of evaluation performed. It examines and measures the immediate effects of a program on the target audience and determines whether objectives were met. An outcome evaluation can be conducted only if program objectives have been clearly defined and stated in measurable terms. An outcome evaluation seeks to answer two questions:
    1. Were there changes in the target audience based on the program's goals and objectives?
    2. Can these changes be attributed to the program?
  • Impact is the most difficult type of evaluation to perform due to the amount of time and resources necessary to adequately assess the impact of a program. An impact evaluation determines whether a program reached its long-term goals. It does not determine whether the program achieved preconceived milestones enroute to those goals. Typically, an impact evaluation answers only one question:
    1. Did the program achieve its long-term goals?
It is common to use a combination of evaluation methods for a program. Process and outcome evaluations are frequently used to measure the effects of a program, however, only a few programs include impact evaluations. If you are uncertain about what type of evaluation you should use, ask:
  • What are the available resources for the project? Limited resources may mean that only a process evaluation can be done.
  • What information should be obtained? To find out whether a program is effective, an outcome or impact evaluation is needed.

TOP

CHOOSING A METHODOLOGY

Once goals and objectives are established and the right evaluation questions are formulated, a method of collecting data must be chosen that best answers the questions. There are numerous ways to collect data for evaluation research, and, depending on the program, different methodologies may be necessary for the evaluation questions. The key to choosing a method of data collection is whether it will yield the precise information needed. A good methodology reveals the impact of a program on the target audience. The three basic methodology designs are:
  • Case study, which examines changes in a single group.
  • Time series, which investigates changes to a group over time. In this type of evaluation, data must be collected at intervals over a period of time.
  • Comparison Group which compares a program's participants with a similar group (control group) that does not participate in the program (either by receiving no services or participating in another program).

The ideal method to use is a comparison group where data is collected from two identical groups at the start of a program as well as several times throughout the implementation of the program. This is also the most difficult method to use and requires extensive resources.

Questions that should be asked when determining what methodology design to use are:

  • Do I want data for only one point in time or data that compares participants both before and after participation in the program?
  • Do I want to compare the program participants with a group that does not participate in the program?
  • Do I have access to a comparison group that will not participate in the program?
  • What expertise do I have or have access to for analyzing the data collected?
  • How much time and money are available for data collection?
A general rule of thumb is to use the most rigorous method that available resources allow.
TOP

CHOOSING A DATA COLLECTION METHOD(S)

Choosing the appropriate data collection tool(s) is crucial to determining the effectiveness of the program. Data collection tools must be designed to obtain the information required by the evaluation questions. Before deciding which method of data collection to use, the following questions must be answered:
  • What type of evaluation is being used (process, outcome, impact or a combination of these)?
  • How much and what type of information is needed to answer the evaluation questions?
  • Will there be data for all program components or just one or two?
  • Do instruments exist to collect the data or do they need to be developed?
  • How much time is there for data collection?

Data Collection Tools

  • Questionnaires measure participants' traits (i.e., knowledge, attitudes).
  • Interviews yield testimonials regarding how much participants like the program or how they have changed while participating in the program.
  • Records/Files yield demographic information as well as other personal data.
  • Observation provides information through direct observation of behaviors.
  • Existing Data/Statistics provide statistical information. If existing questionnaires are used, look for ones that demonstrate a high level of reliability (the ability of the instrument to yield the same results on separate occasions in the absence of changed behaviors, knowledge or attitudes) and validity (the ability of the tool to measure what it is supposed to measure).Use existing standardized instruments when possible, because the reliability and validity have been determined through pilot testing. Before using any custom-designed instrument(s), conduct pilot tests to ensure reliability and validity — a time-consuming task. Results are most conclusive when outcomes are measured using more than one data collection method.
TOP

ANALYZING DATA

Generally, data analysis techniques are determined by the evaluation questions and the methodology used to collect data. Data analysis can be as simple as calculating percentages or as complex as performing a time series or regression analysis. However, be aware that more complex analysis usually requires assistance from someone who has expertise in data analysis. Because specific aspects of analysis are extremely complex, only three basic types of data analysis are represented below.
  • Descriptive Statistics simply describe the people who participated in the program. It is important not to include any statements regarding changes in participants, just the facts. Example questions that can be answered by descriptive statistics include:
    1. How many people participated in the program?
    2. What age group(s) were present in the program?
    3. What percentage of the participants were male or female?
    4. What percentage of the participants were from underserved/underrepresented groups?
  • Correlational Statistics relate one variable to another variable - they do not make a determination of cause and effect. Examples of questions that can be answered by correlational statistics include:
    1. Was there a relationship between teacher knowledge and workshop participation?
    2. Was there a relationship between number of years teaching and the workshop choice?
  • Tests of Statistical Significance determine whether changes actually occurred and if the changes were caused by the program. Example questions that can be answered by tests of statistical significance include:
    1. Was there a change in knowledge after participation in the program?
    2. Was the change due to participation in the program?
The general rule is to begin with the least complex analyses and work toward the most complex analyses possible with the available expertise.
TOP

REPORTING EVALUATION FINDINGS

Once data are analyzed and interpreted, prepare a report to disseminate and publish program results. Please remember that finding no change among program participants is as important as finding changes, and both should be reported.
The standard framework for writing a good evaluation report is:
  • An executive summary, which provides a brief overview of the evaluation
  • An introduction, which describes the program and its components, the target population, and the goals and objectives of the program
  • A methodology, which describes how the program was actually implemented as well as how the data were collected, what instruments were used to collect the data, and how the data were analyzed
  • The results section of the data analysis (it is important to note here that this section should contain concrete data, not interpretations)
  • A discussion section, which explains how the data were interpreted — provides answers to evaluation questions, discloses any problems encountered in the evaluation, and suggests what could be done in the future to improve other similar evaluations
  • A recommendations section, in which recommendations are made based on findings (this section is not always necessary).
Evaluation reports should be written so that they are clear both to lay people and professionals and formatted in a logical, attractive manner; otherwise, no one will read them. Use the following guidelines when writing the program's final report:
  • Do not overgeneralize your findings — if the program was effective with pre-service science teachers, do not claim that the program will be effective with all teachers.
  • Do not call modest changes or differences a success.
  • Report total outcomes as well as partial outcomes.
  • Be candid but do not be tactless.
In the case of the IDEAS program, the above components will need to be formatted according to the IDEAS Final Report Form Template.

TOP

DEVELOPING AN EVALUATION PLAN

Program development is labor intensive and requires plotting a strategy to streamline the process. Below is a checklist of questions to ask when formulating a model of the program and establishing an evaluation plan.


Clarify Goals and Objectives

  • Are goals and objectives defined in measurable terms?
  • Does each goal and objective contain the four required elements (i.e. who, what, when, how much)?
  • Are they directly linked to the intervention?
Create a Model of Your Program
  • Does the model contain the following categories: intervention(s), target population, objectives and goals?
  • Is every element in the model directly linked to another element in the model?
Formulate Evaluation Questions
  • Do your questions reflect the program components?
  • Can you gather the data needed to answer these questions?
  • Will the answers to the questions help those who will use the results of your evaluation?
Determine What Type of Evaluation You Want to Conduct
  • Did you choose the type of evaluation (process, outcome or impact) that will provide the information needed?
  • Do you have the resources (time, money, etc.) to conduct the type of evaluation chosen?
Choose Methodology
  • Will the methodology you chose provide the answers to the evaluation questions?
  • Do you have the resources to utilize the methodology chosen?
  • Can you possibly use a more rigorous method?
Choose Data Collection Method(s)
  • Will the data collection method you chose provide the data needed?
  • Are your methods appropriate to the type of evaluation being conducted?
  • Are your data collection tools reliable and valid?
Analyze Data
  • Is your data analysis appropriate for the type of data collected?
  • Will your analysis answer the evaluation questions?
Report Your Findings
  • Does your report fulfill the requirements given?
  • Is your report understandable to the audience?
  • Did you report your findings clearly?
UP

GUIDELINES OF EVALUATION

There are only a few hard-and-fast rules when it comes to program evaluation.
  1. Know who will use the results of your evaluation.
  2. Always be aware of the resources available to you.
  3. Make sure you ask the right questions.
  4. If at all possible, use a control group.
  5. Report what you find and don't report what you don't find.
  6. Keep your evaluation focused.
  7. Get help from experts when you need it.
  8. Be flexible.
  9. Document everything you do.
  10. A well-done process evaluation is better than a poorly done outcome/impact assessment.
UP

BIBLIOGRAPHY

PROGRAM EVALUATION/RESEARCH
  • Babbie, E.R. Survey research methods. Belmont, CA: Wadsworth, 1973.
  • Bogdan, R.C., and Biklen, S.K. Qualitative research in education. 2nd ed. Newbury Park, CA: Sage, 1992.
  • Borg, W.R., and Gall, M.D. Exploring relationships between variables: The causal-comparative method. In Educational research: An introduction. 5th ed. New York: Longman, 1989.
  • Campbell, D.T., and Stanley, J.C. Experimental and quasiexperimental designs for research. Chicago: Rand McNally, 1963.
  • Campbell, J.P., Daft, R.L., and Hulin, C.L. What to study: Generating and developing research questions. Newbury Park, CA: Sage, 1982.
  • Carey, L. Measuring and evaluating school learning. Boston: Allyn & Bacon, 1988.
  • Cook, T.D., and Campbell, D.T. Quasiexperimentation: Designs and analysis issues for field settings. Chicago: Rand McNally, 1979.
  • Fink, A., and Kosecoff, J. How to conduct surveys: A step-by-step guide. Newbury Park, CA: Sage, 1985.
  • Fraenkel, J.R., and Wallen, N.E. How to Design and Evaluate Research in Education. 3rd ed. McGraw-Hill, Inc., 1996. [This book has an excellent topical bibliography.]
  • Hatry, H.P., Newcomer, K.E., and Wholey, J.S. Handbook of Practical Program Evaluation. San Francisco: Jossey-Bass, 1994.
  • Hopkins, D. A teacher's guide to classroom research. Philadelphia, PA: Open University Press, 1985.
  • Kenny, D.A. Correlation and causality. New York: Wiley, 1979.
  • Krippendorff, K. Content analysis: An introduction to its methodology. Newbury Park, CA: Sage, 1980.
  • Lancy, D.F. Qualitative research in education: An introduction to the major traditions. New York: Longman, 1993.
  • Leibetrau, A.M. Measures of Association. Newbury Park, CA: Sage, 1979.
  • Phillips, D.C. Towards an evaluation of the experiment in educational contexts. Educational Researcher, 10(6):13-20.
  • Shulman, L.S. Disciplines of inquiry in education: An overview. In R.M. Jaeger (Ed.) Complementary methods for research in education. Washington, D.C.: American Educational Research Association, 1988.
  • Tuckman, B.W. Identifying and labeling variables. In Conducting educational research. 4th ed. San Diego, Calif.: Harcourt, Brace, Jovanovich, 1994.
  • Weber, R.P. Basic content analysis. 2nd ed. Newbury Park, Calif.: Sage, 1990.
  • Wholey, J.S., Hatry, H.P., and Newcomer, K.E. Handbook of Practical Program Evaluation. San Francisco, Calif.: Jossey-Bass, Inc., 1994.
  • Wilson, S. The use of ethnographic techniques in educational research. Review of Educational Research, 47:245-65.
SAMPLING
  • Cochran, W.G. Sampling techniques. 3rd ed. New York: Wiley, 1977.
  • Kish, L. Survey Sampling New York: Wiley, 1965.
QUESTIONNAIRE/SURVEY DEVELOPMENT
  • Converse, J.M., and Presser, S. Survey Questions: Handicrafting the Standardized Questionnaire. Newbury Park, CA: Sage, 1976.
  • Fink, A., and Kosecoff, J. How to Conduct Surveys: A Step by Step Guide. Newbury Park, CA: Sage, 1985.
  • Sudman, S., and Bradburn, N.M. Asking Questions: A Practical Guide to Questionnaire Design. San Francisco: Jossey-Bass, 1982.
VALIDITY AND RELIABILITY
  • Brinberg, S., and McGrath, J.E. Validity and the research process. Newbury Park, CA: Sage, 1985.
  • Carmines, E.G., and Zeller, R.A. Reliability and validity assessment. Newbury Park, CA: Sage, 1979.
STATISTICS
  • Bruning, J.L., and Kintz, B.L. Computational handbook of statistics. 3rd ed. Glenview, IL: Scott, Foresman, 1987.
  • Conover, W.J. Practical nonparametric statistics. New York: Wiley, 1971.
  • Schutte, J.G. Everything you always wanted to know about elementary statistics (but were afraid to ask). Englewood Cliffs, NJ: Prentice-Hall, 1977.
TOP