Venting at the Virtual
Greetings! I’m Dr. Eval, and this is the first installment of my new research evaluation advice column for L&L. As part of ISTE’s
in-house Research and Evaluation (R&E) Department, I work on federal- and corporate-funded
projects all the time, so I know what they’re looking for and how to design programs and evaluations that meet their requirements. And I’m here to
answer your questions about the process.
Because readers haven’t had the chance to submit
any questions yet, I went looking for some online.
I found “An Open Letter to All Organizations in
Need of a Program Evaluator” by Watson Scott
Swail on the Washington, D.C., Education Policy
Institute’s news site ( http://tinyurl.com/3enaxzn).
The American Evaluation Association’s discussion
list ( email@example.com) picked up the letter,
which elicited a flurry of contributions under the
strand “Things Evaluators Hate to Hear.” Although
I would classify these comments mostly as venting
at the virtual water cooler, they provide a short list
of important questions for educators and evaluators to consider together.
When do you bring in an evaluator?
Ideally, you work with an evaluator when you
design a project. Scroll through grant announcements to see if the funders require evaluation. If
so, try to bring someone in while planning your
proposal. The evaluator can help ensure that the
project is set up to collect the information it needs
to tell its story. The planning process itself often
results in a more focused and compelling proposal.
Most evaluators don’t charge for their contributions to a proposal, because they expect to recover
their investment with an evaluation contract if the
proposal wins. Evaluators can join a project that’s
already under way, but they may identify important studies that will be difficult to carry out if personnel are already committed to other activities.
What do you do with an unwelcome finding?
Consider it an opportunity for growth. No finding should be a total surprise. Evaluators should
update a program whenever they feel that something—technology, professional development,
assessments, whatever—is not working out. You
can question the data and methods behind any
evaluation report; evaluators should expect to back
up their work. However, a negative evaluation may
be entirely valid. Trying to get the evaluator to spin
the findings is a nonstarter. The “
lemonade-from-lemons” approach is to identify why you did not
get the results you wanted. Strong evidence that an
approach does not work could save educators millions of dollars. If you are at a point where you can
modify your program, funding agencies usually
appreciate responsive grantees. And if you are
at the final-report stage, you are now the expert in
what not to do. You should be well positioned to
propose an alternative for your next project.
ASK DR. EVAL
How much is an evaluation going to cost?
In general, it will cost what the funder says it
should cost. Some agencies expect you to commit a
certain percentage of funds to evaluation. If you are
the funder (e.g., your district wants to commission
a needs assessment directly) and you do not know
what the work should cost, tell prospective evaluators that you need help with the scope of work and
can negotiate the final assignment. Some evaluators
will be reluctant to invest time in a project that they
ultimately may not want to bid. However, those
who do respond may end up being long-term research partners.
Find ISTE R&E online at
If you have a question for Dr. Eval, please send it to
Dr. Eval is part of ISTE’s
Research & Evaluation
provides needs assessment, grant development,
and program evaluation
services to school districts, universities, public
agencies, and private
Research Exchange (TRx)
review site and the Ask
Dr. Eval column, R&E
helps ISTE members use
research and evaluation
in their work.