PASSIA
Seminars
|
 |
|
Home > Seminars > Civil Society Empowerment Through Training & Skills Development |
|
Chapter 6
6.1 Analyzing
Data
1. Data Management
Organizing evaluation data is
an important step for ensuring effective analysis and reporting. If the amount
of quantitative data is very small and you are not familiar with computer
software and data entry, you might opt to manually organize and analyze data.
However, if the amount of data is huge or you need to carry out sophisticated
analysis, you should enter the data into a computer program. There are a number
of software packages available to manage the evaluation data, including SPSS,
Access, or Excel. Each requires a different level of technical expertise. For a
relatively small project, Excel is the simplest of the three programs and
should work well as a database software. In any event, the assistance of statisticians and computer experts can be
engaged at different stages of the evaluation.
2. Analysis of
Quantitative Data
Analyzing the gathered
quantitative and qualitative data is a major step in project evaluation.
Developing a data analysis plan is important to carry out a successful analysis
and interpretation of information gathered by the evaluation. Following are
some tips to make sense of the quantitative data:
a. Start with the
evaluation goals and objectives:
Before analyzing your data,
review your evaluation goals. This will help you organize your data and focus
your analysis. For example, if you wanted to improve your program by
identifying its strengths and weaknesses, you can organize data into program
strengths, weaknesses and suggestions to improve the program. If you are
conducting an outcomes-based evaluation, you could categorize data according to
the indicators for each outcome. In general, data analysis is facilitated if
the project has clear and measurable goals and objectives.
b. Basic analysis of
quantitative information
Data analysis often involves
the disaggregation of data into categories to provide evidence about project
achievements and to identify areas in which a program is succeeding and/or
needs improvement. Data can be broken down by gender, social and economic
situation, education, area of residence (urban or rural), marital status, age,
etc. Decide what type of disaggregation is relevant to your evaluation and
project objectives and indicators. One of the main advantages of statistical
analysis is that it can be used to summarize the findings of an evaluation in a
clear, precise and reliable way. However, not all information can be analyzed
quantitatively. The most commonly used statistics include the following:
Frequency Count. A
frequency count provides an enumeration of activities, things, or people that
have certain pre-specified characteristics. Frequency counts can often be
categorized (e.g., 0, 1-5, 6-10, more than 10) in data analysis.
Percentage. A
percentage tells us the proportion of activities, things, or people that have
certain characteristics within the total population of the study or sample.
Percentage is probably the most commonly used statistic to show the current
status as well as growth over time.
Mean. The
mean is the most commonly used statistic to represent the average in research
and evaluation studies. It is derived by dividing the sum by the total number
of units included in the summation. The mean has mathematical properties that
make it appropriate to use with many statistical procedures.
The level of sophistication of analysis is a matter
of concern in evaluation. Tables, percentages and averages often give a clear
picture of the sample data particularly for non-specialists, and many users
will only be interested in this level of analysis. In addition, measures of
spread, including percentiles and standard deviations, may add valuable information
on how a variable is distributed throughout a sample population. There is a
wealth of more sophisticated research methods that can be applied. However,
much of the evaluation work can be done using very basic methods.
3. Analysis of Quantitative Information
The use of both quantitative
and qualitative analysis in evaluation has become the preferred model for many
evaluators. Most evaluators and researchers agree that they should be employed
simultaneously. The analysis of qualitative data helps broaden the view of the
phenomena of interest in an evaluation, but can also increase depth and detail,
where needed.
Qualitative
data includes detailed descriptions, direct quotations in response to
open-ended questions, analysis of case studies, the transcript of opinion of
groups, and observations of different types. Qualitative analysis is best done
in conjunction with the statistical analysis of related (quantitative or
qualitative) data. The evaluation should be designed so that the two sorts of
analysis, using different but related data, will be mutually reinforcing.
Analysis of qualitative methods
may produce descriptions (patterns, themes,
tendencies, trends, etc.), and interpretations and explanations of
these patterns. The data analysis should include efforts to assess the
reliability and validity of findings. Following is a list of some useful tips
to improve your analysis of qualitative data:
·
Carefully review all the data.
·
Organize comments into similar
categories, e.g., concerns, suggestions, strengths, weaknesses, similar
experiences, program inputs, recommendations, outputs, outcome indicators, etc.
·
Try to identify patterns, or
associations and causal relationships in the themes, e.g., all people who
attended programs in the evening had similar
concerns, most people came from the same geographic area, most people
were in the same salary range, processes or events respondents experience
during the program, etc.
·
Try to combine the results of
the quantitative and qualitative data.
It is important to keep all documents for several
years after completion in case they are needed for future reference.
6.2 Development
of an Evaluation Report
There is
no common format for reporting. Following is a list of tips that might help in
improving your evaluation reports:
a. Start
the preparation of the evaluation report at an early stage.
It is
useful to start the preparation of the report before data collection. There are
a number of sections that can be prepared by using the material of the
evaluation plan or proposal (background section, information about the project
and some aspects of the methodology, evaluation questions, etc.). Those will
remain the same throughout the evaluation. The evaluation findings,
conclusions, and recommendations generally need to wait for the end of the
evaluation.
Evaluations
generate huge amount of information. Therefore, it is useful to organize
evaluation data and field notes as soon as they are collected and to document
fieldwork experiences and observations as soon as possible. Finally, preparing
sections of the findings chapter during the data collection phase allows
researchers to generate preliminary conclusions or identify potential trends
that need to be assessed by additional data collection activities.
b. Make
the report short and concise
One of
the most challenging tasks that evaluators face is how to organize the huge
amount of data gathered into a useful, concise and interesting report and what
data to include and not to include. It is useful to remember that only a small
and concise amount of tabulations prepared during the analysis phase should be
reported. A report outline will help in classifying information. Always abide
by your key evaluation questions, the indicators you are assessing and the type
of information that your audience needs.
Make your recommendations clear, concise and direct.
Examples include:
1.
Ways for improving management of the program (planning, decision making,
policy development, etc.) and where capacity building/technical assistance and
training are needed.
2.
Actions needed to increase effects of the project.
3.
Actions needed to improve monitoring and evaluation processes and
methods.
4.
Topics for further research.
c. Make
the presentation interesting
Remember
that the level and content of evaluation reports depend on for whom the report
is intended, e.g., donors, staff, beneficiaries, the general public, etc. Presentation
must be clear and adjusted to the target group. The presentation must be made
in simple language that can be understood by non-professionals. Following is a
list of suggestions that might help in
making your report more interesting and easier to read:
1.
The first sentence of paragraphs should be used to make the main point,
and the remainder to supplement, substantiate and discuss the main point.
2.
As much as possible, use a short text. This will
ensure that a large number of people will read it.
3.
The structure of the report
should be simple. The text should be broken down in relatively small thematic
or sequential parts, with simple and clear subtitles precisely identifying the
topics discussed.
4.
Make the
report interesting to read. Display your data in graphs, diagrams,
illustrations and tables that summarize numbers. This should reduce the amount of
text needed to describe the results. Furthermore,
they are more effective than written text. Do not explain the graphs or
illustrations in written form. Focus only on the important points that relate to the problem under discussion. Use of
qualitative information effectively makes the report more interesting.
In addition, direct quotes, short examples and comments heard during fieldwork
personalize the findings, and photographs help in familiarizing readers with
the conditions of the project beneficiaries.
5.
Use simple language that the
readers will understand. Avoid the use of long and complicated sentences,
unclear jargon and/or difficult words. Important technical terms should be
defined in the text or in the glossary at the end of the report.
6.
Different main ideas should be
presented in separate sentences.
7.
The meaning of abbreviations
and colloquial words should be explained.
8.
Simple link words should be
used to split sentences and indicate the direction in which the argument is
moving. Link words should be simple, such as “also,” “even so,” “on the other
hand,” and “in the same way.” Avoid long words like “moreover,” “nevertheless,”
and “notwithstanding.”
9.
Only data tables or diagrams should contain detailed numbers. The
written text should highlight the most important numbers and say what they
mean. Percentages should in most cases be rounded up to the nearest whole
number. It should be possible for the reader to get the main message from a
table without consulting the text. Every table must have a title, table number,
reference to the source of information, sample size, and full description of
what each figure refers to.
10. Use space around the text. Ease of reading and understanding is more
important than reducing the volume of pages.
Consider the following
format for your report:
|
Suggested Contents of Evaluation Report 1.
Title page 2.
Table of Contents 3.
Acknowledgments (optional) ·
Identify those who
contributed to the evaluation. 4.
Executive Summary ·
Summarize the program/project
evaluated, the purpose of the evaluation and the methods used, the major
findings, and the recommendations in priority order. ·
Two to three pages
(usually) that could be read independently without reference to the rest of
the report. 5.
Introduction ·
Identify program/project
description/background. ·
Describe the program/project
being evaluated (the setting and problem addressed, objectives and
strategies, funding). ·
Summarize the evaluation
context (purposes, sponsors, composition of the team, duration). 6.
Evaluation Objectives and Methodology ·
List the evaluation
objectives (the questions the evaluation was designed to answer). ·
Describe fully the evaluation
methods and instruments (e.g., what data were collected, specific methods
used to gather and analyze them, rationale for visiting selected sites). ·
Limitations of the
evaluation. 7.
Findings and Conclusions ·
State findings clearly with
data presented graphically in tables and figures. Include effects of the
findings on achievement of program/project goals. ·
Explain the comparisons made
to judge whether adequate progress was made. ·
Identify reasons for
accomplishments and failures, especially continuing constraints. 8.
Recommendations · List the recommendations for different kinds of
users in priority order. Include costs of implementing them, when possible.
·
Link
recommendations explicitly with the findings, discussing their implication
for decision-makers. · Include a proposed timetable for
implementing/reviewing recommendations. 9.
Lessons Learned (optional) ·
Identify lessons learned from
this evaluation for those planning, implementing or evaluating similar
activities. 10. Appendices
·
Terms of Reference. ·
Instruments used to collect
data/information (copies of questionnaires, surveys, etc.). ·
List of persons interviewed
and sites visited. ·
Data collection
instruments. ·
Case studies. ·
Abbreviations. ·
Any related literature. ·
Other data/ tables not
included in the findings chapter. |
Source: UNICEF, A UNICEF Guide
for Monitoring and Evaluation: Making a Difference?,
New York
PASSIA
The Palestinian Academic Society for the Study of International Affairs, Jerusalem
Tel: +972-2-6264426 / 6286566 • Fax: +972-2-6282819
P.O. Box 19545, Jerusalem
Email:
passia@palnet.com
Copyright
©
PASSIA