APPROACH TO ENHANCE
EVALUATION USE
METHODS FOR ENHANCING USE
In order to ensure that my
evaluation with have an impact on the program, I intend to engage primary users
in the entire process, from planning the evaluation, throughout the evaluation
process and in the post-evaluation support in which evaluation questions can be
fully supported.
In planning this approach,
I intend to use the Better Evaluation tools, which reflect on Michael Quinn
Patton’s work on Utilization-Focused Evaluation in which various multi-step
models have been identified to guide the thought process about enhancing
use. I will be using Better Evaluation
17 Steps in Building a Utilization-Focused Evaluation to guide my thoughts:
Step
|
Goal of Step
|
Strategy
|
1
|
Assess
and build program and organizational readiness for utilization-focused
evaluation
|
Invite all agency staff to discuss:
·
-Needs of
local youth in regard to employment
·
-Program
evaluation concerns
·
-Staff roles
in evaluation process
·
-Capacity to
engage in the process
·
-Identify
primary users
·
-SWOT
analysis
|
2
|
Assess
and enhance evaluator readiness and competence to undertake a
utilization-focused evaluation
|
Examine evaluator (my):
·
-Potential
baises
·
-Potential
conflict of interest
·
-Capacity-
time and other resources
·
-Relationship
is conducive to collaboration
·
-Proper
foundation skills
·
-Suitable
framework for conducting the evaluation
|
3
|
Identify, organize, and
engage primary intended users: the personal factor
|
Invite primary users and
potential stakeholders to participate
·
-Offer a
forum to discuss youth employment needs
·
-Introduce
the evaluation and ways stakeholders can engage in the process
·
-Ask for
opinions and ideas about the necessary components of the evaluation
|
4
|
Situation
analysis conducted jointly with primary intended users
|
During the forum, engage
as many perspectives on the issue of youth unemployment as possible.
-Use this opportunity to
present information about the purpose of the program and why evaluation is
important.
-This should include
statistics about the macro view of the situation of Canadian youth
unemployment with a goal of finding out more about the local context of the
issue so that collaboration is encouraged.
|
5
|
Identify
and prioritize primary intended uses by determining priority purposes
|
Pre-determine primary
intended users and double check accuracy after the group consultations have
been held
-communicate intended
uses on the evaluation website
|
6
|
Consider
and build in process uses if and as appropriate
|
Weekly meetings about
the evaluation process, highlighting lessons learned, aha moments, staff
insights. This will be staff lead,
whenever possible.
|
7
|
Focus
priority evaluation questions
|
Evaluation questions
have been determined, base on primary user inquiry needs.
|
8
|
Check
that fundamental areas for evaluation inquiry are being adequately addressed:
implementation, outcomes, and attribution questions
|
Carefully tracked using
Trello for To-Do’s of implementation to ensure that all do dates are met for
gathering data, implementing process and addressing key questions.
|
9
|
Determine
what intervention model or theory of change is being evaluated
|
Theory of Change Model
was used to help examine organizational values, priorities and deliverables
and the Logic Model was used to operationalize the flow of the change
mechanisms.
|
10
|
Negotiate
appropriate methods to generate credible findings that support intended use
by intended users
|
-Staff brainstorming
sessions
-Group consultations
with stakeholders
-Surveys of parents
-Surveys of employers
-Key informant
interviews with the clients
|
11
|
Make
sure intended users understand potential methods controversies and their
implications
|
-Noted to use plain
language for greater clarity
-Informed consent for
clients and parents
-Confidentiality agreements
in place
|
12
|
Simulate
use of findings: evaluation's equivalent of a dress rehearsal
|
-A staff meeting to
discuss the release of information during community consultations
|
13
|
Gather data with ongoing
attention to use
|
-Staff implementing the
program and inquiry methods (interviews, surveys, consultations) will be
asked for feedback as to the appropriateness of the methods used and the data
themes being generated
-We will approach the
task with a flexible mindset, prepared to make changes as needed for the good
of the evaluation
|
14
|
Organize and present the
data for interpretation and use by primary intended users: analysis,
interpretation, judgment, and recommendations
|
Organize data from data
collection methods:
-Cross tabulations
-Thematic Groupings
-Data
Staff to offer potential
interpretations and contextual information to add enhance the evaluation
-recommendations will be
made after data is gathered in order to further enhance usefulness of the
evaluation process and findings
|
15
|
Prepare an evaluation
report to facilitate use and disseminate significant findings to expand
influence
|
-Interim reports will
support process use value extraction
-Report will be prepared
as concisely as possible to make sure that findings are impactful
-Reports will be made
accessible on the program evaluation website
|
16
|
Follow up with primary
intended users to facilitate and enhance use
|
-Q&A periods
post-evaluation
-Recommendations will be
checked and shared
-Post evaluation support
will be made available to primary users to integrate knowledge from process
and findings into use beyond the evaluation
|
17
|
Meta-evaluation of use:
be accountable, learn, and improve
|
Accountability:
Evaluation will be
accountable for results. Any disputes
on results will be followed upon and reported back
Learning:
-Both process use and
findings will be emphasized as a process to be continued with partners. Inquiry into the further learning or engagement
needs will be part of a post-evaluation survey
Improving:
-Community, stakeholder,
primary user and funder feedback about further questions or inquiry will be
used to in a reflective process to inform the project, but also the evaluator’s
experience in the goals of enhancing evaluations
|
REPORTING STRATEGIES AND
METHODS
TO ENHANCE USE
Better Evaluation
Resources have guided my thinking about reporting formats and needs to report
and support use.
Report Strategy
|
Plan of Action
|
Reporting Needs Analysis
|
-During the planning
phase and before the evaluation implementation, discuss reporting needs,
features, details, answers and knowledge sought to better visualize the
future reporting process
-Discuss goals of
primary users and stakeholders in terms of reporting outputs
-Present findings to
staff group and discuss best ways of presenting findings from a reporting
perspective
|
Communications Plan
|
-During the planning
phase, develop a communications plan, set up distribution lists for
broadcasts
-Promote a central
meeting place for the evaluation, which will be the evaluation website
-Drive all social media
to the website
|
Reporting Media
|
Plan of Action
|
Considerations
|
Interim Reports
|
-Updated reports to
track progress of goals and encourage engagement of stakeholders and primary
users
|
-Accessibility on
website
|
Final Report
|
-A simplified layout
-Adhering to 1:3:25
principles for outline, executive summary and findings
-Graphic representations
will be analyzed for ability to simplify for greater understanding
-Final report will
include recommendations about next steps for program and for local employment
for youth
|
-Accessibility on
website
-Plain language
-Adhere to Literacy
standards in text, language and graphics
|
E-Postcards
|
-Send monthly postcards
driving stakeholders to the website for greater information
|
-Interesting, graphical
format to spark interest and curiosity
|
Website
|
-Website will be
maintained weekly with process and findings as they become available
|
-Website will have to
have content management or blog format to allow for regular updates
-AODA standards
|
Social Media
|
-Social media will be
regularly maintained with snip its of information about the process and
findings, linking back to the website
|
-Hashtags
-Brevity
|
Presentation of Process
and Findings
(with available video
and teleconference)
|
-Report findings will be
presented in a community forum
-Recommendations will be
discussed
-Request info on how to
best support future use
-Reflection on lessons
learned
-Discussions about
further research needs and related community concerns
-Challenges to use
|
-Allow for extensive
Q&A period
-Discussion period for
recommendations
|
Recorded Presentation
for Playback
|
-Post community
presentation on website
|
-ensure AODA standards
are met
|
Graphic Considerations
|
-Use of graphics to
create mental models will be used whenever possible
|
-Simplified versions of
graphic will be used to represent core meaning
|
Resources:
Collect and/or retrieve data. Better Evaluation-sharing information to improve evaluation. [Website]. Retrieved from:
Utilization-focused evaluation. Better Evaluation- sharing information to improve evaluation. [Website]. Retrieved from:
http://www.betterevaluation.org/en/plan/approach/utilization_focused_evaluation
Report and support use of findings. Better Evaluation- sharing information to improve evaluation. [Website]. Retrieved from:
https://www.betterevaluation.org/en/plan/reportandsupportuse
No comments:
Post a Comment