Learning & Leading Through Technology

and in journals. In general, the U.S. Department of Education and researchers funded through its Institute of Education Science (IES) have championed the theory-level studies, while the National Science Foundation (NSF) and the evaluators of its diverse grant programs have backed multiple methods.

In 2011, the U.S. Office of Science and Technology Policy (OSTP) published a report on the nation’s investment in education for science, technology, engineering, and math (STEM). In February

2012, the Government Accountability Office (GAO) published a related report analyzing the overlap of numerous STEM initiatives. The overall conclusion of the report was that the government simply could not say whether and to what extent those investments were paying off. The GAO looked at the numerous grant programs of the NSF, IES, and other agencies and noted that, while many of them included evaluations, the project-level evaluations did not necessarily provide composite data that could inform program-wide decisions. In other words, the OSTP and GAO reviewers (and by extension, the taxpayers of the United States) face the same problem any other consumer of research does when trying to make sense of different orders of evidence.

Both the IES and NSF came under
criticism in the reports. In response, the
two agencies convened a joint commit-
tee to set their research programs in or-
der. In August 2013, the joint committee
released a concise and useful document,
the Common Guidelines for Education
Research and Development ( ies.ed.gov/
pdf/CommonGuidelines.pdf). The doc-
ument offers several potential benefits
to educators and researchers, including
making explicit the fact that there are
different types of research for different
purposes. They also state that program
directors, principal investigators, and
evaluators for U.S. institutions should
not have to switch methodological gears
when submitting proposals to the NSF
or IES. They still have to align their
project and research to the specifica-
tions of each grant program, but they
should be able to dispense with one
layer of second-guessing what the pro-
gram really wants.

Finally, the report limited its taxonomy of research studies to six types, which makes them easier to recall:

Foundational. These types of studies cover basic science, such as studies of how the brain works or how children perceive and organize facts.

Early-Stage or Exploratory. These studies are about basic relationships between educational factors, such as whether certain testing conditions are associated with higher performance. (Note that this is different from testing whether conditions cause higher performance.)

Design and Development. These studies use theory and established relationships to create new lessons, materials, or other interventions. A simulation program pilot might be an example. Grant reviewers at this level typically do not fret about statistical significance or effect size. They are more likely to scrutinize what would happen if kids are bored by the simulation: How would you know? What would you do about it?

Efficacy. These studies are all about testing an intervention in a controlled setting. (Jargon alert: In the research world, efficacy, effectiveness, and impact are not the same thing.) This level and the next two are classified as “impact” studies, where technical research issues come into play, including the similarity of comparison groups and the statistical power of comparisons. A typical efficacy study might involve a software developer providing its simulation to randomly assigned students in closely monitored treatment and control classrooms to quantify outcomes.

Effectiveness. This type of research tests an intervention in field conditions.

Maybe the efficacy study mentioned
above showed that the simulation
software improves learning when per-
fectly implemented by teachers who re-
ceive special training. But what happens
when it is implemented across a district
of diverse students by teachers who get
a standard one-shot training?

Scale-Up. This is effectiveness testing on a large scale, usually with a large budget.

It is tempting to place these six levels on a timeline with an arrow showing the progress of knowledge from basic research to a statewide learning materials adoption, but the NSF/IES authors stress that it does not always work that way. Studies can bridge levels, and an impact study—even with negative results—may instigate new foundational or design projects.

How can we make use of this little gem? If you are seeking U.S. government funding for an educational program, you should try to place your project within the Common Guidelines framework. Look in the guideline tables for the expectations for purpose, policy/practical significance, theoretical/empirical basis, outcomes, and research plan. And then look in the appendixes for examples of funded projects of that type.

If you are an educator or educational policy maker who is involved in a federal initiative, go through the same process so that you understand the implications for your school, district, or agency. For instance, if you sign on to an efficacy study to access new technology, keep in mind that you may be asked to modify student schedules to fit the research sample selection requirements.

Finally, if you wonder whether the guidelines make a difference, pay attention to your own school, district, or agency’s involvement in educational research. At what levels do the programs you care about generate knowledge for improvement? What does the evidence say about how they could reach more or different learners? Forget finding a study to replicate and conduct your own exploratory or developmental research.