PASSIA
Seminars
|
 |
|
Home > Seminars > Civil Society Empowerment Through Training & Skills Development |
Chapter 4
Monitoring & Evaluation
Methods and Tools
4.1 Review of
Main Methods and Tools
Evaluations
often produce controversial results. Therefore, they might be criticized,
especially in terms of whether data collection methods, analysis and results lead
to reliable information and conclusions that reflect the situation.
Methods
of data collection have strengths and drawbacks. Formal methods (surveys, participatory observations, direct measurement,
etc.) used in academic research would lead to qualitative and quantitative data
that have a high degree of reliability and validity. The problem is that they
are expensive. Less formal methods (field visits, unstructured interviews,
etc.) might generate rich information but less precise conclusions, especially
because some of those methods depend on subjective views and intuitions.
Qualitative
methods, especially participatory methods of data collection, can bring rich
and in-depth analysis of the situation of the beneficiaries of projects and new
insights into peoples' needs for project planning and implementation. However,
they demand more skills than most quantitative methods. In addition, they
require time and substantial talent in communication and negotiation between
planners and participants.
The quality of
information, especially in terms of validity and reliability, should be a main
concern for the evaluator. The evaluator may simultaneously employ a number of
methods and sources of information in order to cross-validate data
(triangulation). Triangulation is a term used to describe the simultaneous use
of multiple evaluation methods and information sources to study the same topic.
It provides the means to generate rich and contextual information. As well, it
provides the means to verify information and explain conflicting evidence.
The following table provides an overview of some of the quantitative
and qualitative data collection methods commonly
used during evaluations.
|
Method |
Description/
Purpose |
Advantages |
Disadvantages/
Challenges |
|
Literature search |
Gather
background information on methods and results of evaluation methods used by
others. |
Economic
and efficient way of obtaining information. |
Difficult
to assess validity and reliability of secondary data. |
|
Questionnaires / surveys |
Oral interviews or written
questionnaires of a representative sample of respondents. Most
appropriate when need to quickly and/or easily get lots of information from
people in a non-threatening way. |
* Produce reliable
information. * Can be completed
anonymously. * Easy to compare and
analyze. * Can be administered easily to a large number of people. * Collect a lot of data in
an organized manner. * Many sample questionnaires
already exist. |
* Demanding and could be
costly. *
Might not get careful feedback. *
Wording can bias client's responses. *
Data is analyzed for groups and are impersonal. *
Surveys may need sampling expert. *
Provide numbers but do not get the full story. * Open-ended
data may be difficult to analyze. |
|
Interviews |
To
fully understand someone's impressions or experiences, or learn more about
their answers to questionnaires. Individual
or group interviews could be organized to assess perceptions, views and
satisfaction of beneficiaries. |
* Give full range and depth
of information and yield rich data, details and new insights. * Can be flexible with the
client. * Permit face-to-face
contact with respondents and provide opportunity to explore topics in depth. *
Allow interviewer to probe, explain or help clarify questions, increasing the
likelihood of useful responses. *
Allow interviewer to be flexible in administering interview to particular
individuals or circumstances. |
* Can be hard to analyze and
compare. * Interviewer can bias
responses. * Can be expensive and
time-consuming. * Need well-qualified and
highly trained interviewers. * Interviewee may distort
information through recall errors, selective perceptions and desire to please
interviewer. * Flexibility can result in
inconsistencies across interviews. *
Volume of information too large and may be difficult to reduce data. |
|
Documentation review |
Impression of how program operates without interrupting
the program by review of applications, finances, memos, minutes, etc. |
* Give comprehensive and historical information * Doesn't interrupt program or client's routine in
program * Information already exists. * Few biases about information. |
* Often takes a lot of time * Information may be incomplete. Quality of documentation
might be poor. * Need to be clear about purpose. * Not a flexible means to get data. Data restricted to
what already exists. |
|
Observation |
Involves
inspection, field visits and observation to understand processes, infrastructure/services
and their utilization. Gathers accurate information about how a program
actually operates, particularly about processes. |
*
Well-suited for understanding processes, views, operations of a program while
they are actually occurring. * Can adapt to events as
they occur and exist in natural, unstructured and flexible setting. *
Provides direct information about behavior of individuals and groups. *
Permits evaluator to enter into and understand situation/ context. *
Provides good opportunities for identifying unanticipated outcomes. |
*
Dependent on observer’s understanding and interpretation. * Has limited potential for
generalization. *
Can be difficult to interpret exhibited behaviors. *
Can be complex to categorize observations. *
Can influence behavior of program participants. *
Can be expensive and time-consuming. *
Needs well-qualified, highly trained observers and/or content experts. *
Investigator has little control over situation. |
|
Focus groups |
A
focus group brings together a representative group of 8 to 10 people, who are
asked a series of questions related to the task at hand. Used for analysis of specific, complex problems, in
order to identify attitudes and priorities in sample groups. Explore a topic in depth through group discussion,
e.g., about reactions to an experience or suggestion, understanding common
complaints, etc. |
*
Efficient and reasonable in terms of cost. *
Stimulate the generation of new ideas. *
Quickly and reliably gets common impressions *
Can be an efficient way to get a wide range and depth of information in a
short time. *
Can convey key information about programs. *
Useful in project design and in assessing the impact of a project on a given
set of stakeholders. |
* Can be hard to analyze responses.* Need good facilitators. * Difficult to schedule 8-10 people together. |
|
Case studies |
In-depth
review of one or a small number of selected cases. To fully understand or depict beneficiaries’
experiences in a program, and conduct comprehensive examination through cross
comparison of cases. |
*
Well-suited for understanding processes and for formulating hypotheses to be
tested later. *
Fully depicts client's experience in program input, process and results. *
Powerful means to portray program to outsiders. |
* Usually time consuming to collect, organize and describe.
* Represents depth of information, rather than breadth. |
|
Key informant interviews |
Interviews with persons who are knowledgeable about the
community targeted by the project. A key informant is a person (or group) who has unique
skills or professional background related to the issue/intervention being
evaluated, is knowledgeable about the project participants and/or has access
to other information of interest to the evaluator. |
*
Flexible, in-depth approach. *
Easy to implement. *
Provides information concerning causes, reasons and/or best approaches from
an "insider" point of view. *
Advice/feedback increases credibility of study. * May have side benefit to solidify relationships
between evaluators, beneficiaries and other stakeholders. |
*
Risk of biased presentation/ interpretation
from informants/interviewer. *
Time required to select and get commitment may be substantial. * Relationship between evaluator and informants may
influence type of data obtained. * Informants may interject own biases and impressions. |
|
Direct measurement |
Registration of quantifiable or classifiable data by
means of an analytical instrument. |
* Precise. * Reliable and often requiring few resources. |
Registers only facts, not explanations. |
Source:
Information on common qualitative methods is provided in the earlier User-Friendly
Handbook for Project Evaluation (NSF 93-152).
4.2 Selecting
Monitoring and Evaluation Methods
Monitoring is an ongoing
function and can be incorporated into daily management operations. It can
involve a wide range of methods such as interviews with project beneficiaries,
field visits, regular reports, observations, interviews with key informants,
etc.
Evaluation can involve a number
of methods. No recipe or formula is best for every situation. Some methods are
better suited for the collection of certain types of data. Each has advantages
and disadvantages in terms of costs and other practical and technical
considerations (such as ease of use, accuracy, reliability, and validity). For
example, there is no best way to conduct interviews. Your approach will depend
on the practical considerations of getting the work done during the specified
time period. Using a focus group - which is essentially a group interview - is
more efficient than one-on-one interviews, if done well. However, people often
give different answers in groups than they do individually. They may feel freer
to express personal views in a private interview. At the same time, group
conversations can draw out deeper insights as participants listen to what
others are saying. Both approaches have value.
Project staff and evaluators
must weigh pros and cons against program goals. In selecting evaluation
methods, evaluators consider the use of methods that could generate the most
useful and reliable information, be the most cost-effective and is the easiest
to implement in a short period of time.
Following is a list of
questions that might help in selecting appropriate evaluation methods:
1.
What information is needed?
2.
Of this information, how much
can be collected and analyzed in a low-cost and practical manner, e.g., using
questionnaires, surveys and checklists?
3.
How accurate will the
information be?
4.
Will the methods get all of the
needed information?
5.
What additional methods should
and could be used if additional information is needed?
6.
Will the information appear as
credible to decision makers, e.g., to donors or top management?
7.
Are the methods appropriate for
the target group? If group members are illiterate, the use of questionnaires
might not be appropriate unless completed by the evaluators themselves.
8.
Who can administer the methods?
Is training required?
9.
How can the information be
analyzed?
Ideally, the evaluator uses a
combination of methods. For example, a questionnaire to quickly collect a great
deal of information from a lot of people, and then interviews to get more
in-depth information from certain respondents to the questionnaires. In
addition, case studies could then be used for more in-depth analysis of unique
and notable cases, e.g., those who did or did not benefit from the program,
those who quit the program, etc.
Combining quantitative and
qualitative research methods and approaches in monitoring and evaluation of
development projects has proved to be very effective.
References
Guba, E.G., and
Patton, M.Q. (1990). Qualitative Evaluation and
Research Methods, 2nd
Denzin, N.K., and
Krueger, R.A. (1988). Focus Groups: A Practical
Guide for Applied Research.
LeCompte, M.D., Millroy, W.L., and Preissle, J.
(Eds.). (1992). The Handbook of Qualitative Research in Education.
Miles, M.B., and Huberman, A.M. (1994). Qualitative
Data Analysis: An Expanded Sourcebook.
Morse, J.M. (Ed.). (1994). Critical Issues in
Qualitative Research Methods.
Seidman, I.E. (1991). Interviewing as
Qualitative Research: A Guide for Researchers in Education and Social Sciences.
Yin,
R.K. (1989). Case Study Research: Design and Method.
PASSIA
The Palestinian Academic Society for the Study of International Affairs, Jerusalem
Tel: +972-2-6264426 / 6286566 • Fax: +972-2-6282819
P.O. Box 19545, Jerusalem
Email:
passia@palnet.com
Copyright
©
PASSIA