1. INTRODUCTION TO THE TOOLKIT

   • About the Toolkit
   • Toolkit Contents
   • Using the Toolkit
   • Case Study Background

2. INTRODUCTION TO EVALUATION

   • Defining Evaluation
   • Purpose of Evaluation
   • Types of Evaluation
   • Challenges to Conducting an Evaluation

   • Who Should Conduct the Evaluation

     • Assessing Readiness
     • Using a Team Approach
     • Using External Evaluators

   • Considerations for Planning

     • Human Subjects Protections
     • Cultural Competency
     • Budgeting for Evaluation
     • Working with Funders

   • Framework for Evaluation

     • Framework Components
     • Evaluation Standards

3. PLAN YOUR EVALUATION

   • Step 1: Engage Stakeholders

     • Engage Stakeholders
     • Identifying Key Stakeholders
     • Engaging Stakeholders
     • Tools and Resources

   • Step 2: Develop a Logic Model

     • Develop a Logic Model
     • What is a Program Description?
     • Introducing Logic Models
     • Creating a Logic Model for Evaluation
     • Tools and Resources

   • Step 3: Focus the Evaluation

     • Focus the Evaluation
     • Decide What to Evaluate
     • Develop Evaluation Questions
     • What Are Indicators?
     • Start the Evaluation Plan
     • Tools and Resources

   • Step 4a: Plan Data Collection

     • Plan Data Collection
     • Identify Data Sources
     • Select Data Collection Methods
     • Tools and Resources

   • Finalize Your Evaluation Plan

     • Identify the Person Responsible and a Timeline
     • Review Your Evaluation Plan
     • Tools and Resources

4. IMPLEMENT YOUR EVALUATION

   • Introduction
   • Review Your Evaluation Plan

   • Step 4b: Collect Data

     • Design Data Collection Instruments
     • Additional Considerations in Data Collection
     • Tools and Resources

   • Step 5: Analyze and Interpret Data

     • Analyze and Interpret Data
     • Organize the Data
     • Analyze the Data and Synthesize Results
     • Interpret the Results
     • Identify Limitations
     • Tools and Resources

   • Step 6: Share Results

     • Share Results
     • Develop Recommendations
     • Communicate the Findings
     • Follow-up
     • Tools and Resources

TOOLS:Worksheets|Glossary

Reliability and Validity

Most data collection methods require some type of data collection instrument. There are two important concepts to be aware of in instrument design: reliability and validity.

Reliability is the degree to which your assessment produces consistent results over many uses. For instance, say a person is asked the same questions on two consecutive days (without an intervention taking place in between); if the instrument is reliable, their responses should be the same both times. Or, say two different observers review a videotaped counseling session to document specific behaviors; if the observation tool they are using is reliable, they should score the session similarly.

Validity is the degree which your assessment actually measures what you intend it to measure. A simple example is that a ruler is a valid measuring device for length, but would not be valid for measuring weight. In the context of evaluation, using mixed methods can help you be more confident that you are measuring what you intend to measure when collecting data. If a survey question and another method, such as observation, produce similar results, the survey question is probably valid.

The best assessments will be both reliable and valid—they produce stable results that reflect your intended measures. A test that has a high level of reliability but low validity may give you consistent results, but they will be inaccurate. Similarly, a test with low reliability cannot give you valid results.

You may not have to start from scratch when you design your instrument. There are many questions and instruments that have been developed and tested for reliability and validity by other organizations that may be suitable to adapt to your evaluation. If you are designing your own instrument, make sure it is measuring what you intend it to measure and that it is doing so consistently.

Northwest Center for Public Health Practice (NWCPHP) promotes excellence in public health practice and provides training, research and evaluation for state, local, and tribal public health. Content and resource materials from the Data Collection in Program Evaluation module developed by NWCPHP, have been adapted for use in this toolkit for educational purposes with permission from the Northwest Center for Public Health Practice, University of Washington.