Does Technology Enhance Inquiry-Based Learning?

Glenn M. Kleiman, Director, EDC Center for Online Professional Education

• View a PDF version of this document

There is currently a great deal of controversy about a critical issue in educational research: What constitutes valid evidence for determining whether educational innovations are effective? One side of the controversy is reflected in the U.S. Department of Education's current policies and funding programs. This side emphasizes methodological considerations, specifically the use of randomized control trials in which the educational innovation is provided to one group of students, teachers, or classes (the experimental group) while it is not provided to another comparable group (the control group). According to the U.S. Department of Education (2003), "strong" evidence of effectiveness can only be provided when the candidates for the research are randomly assigned to the experimental and control groups. Evidence of "possible" effectiveness may be provided by studies that closely match the groups on relevant factors, such as academic achievement and demographics, but do not make random assignments ("quasi-experimental" studies). According to this side, any other research methods do not provide meaningful evidence of effectiveness.

The controversy is whether this approach is the only type of research that can yield valuable findings and therefore the only type that should be supported by federal funding. Those on the other side of this controversy follow the lead of the Scientific Research in Education report from the National Research Council (2002), which provides the following six guiding principles that underlie all scientific inquiry, including education research:

1. Pose significant questions that can be investigated empirically
2. Link research to relevant theory
3. Use methods that permit direct investigation of the question
4. Provide a coherent and explicit chain of reasoning
5. Replicate and generalize across studies
6. Disclose research to encourage professional scrutiny and critique.

The difference between these two views is significant for those concerned about research into educational uses of technology. The randomized trials view begins research with a fixed methodology; the other view begins with significant questions linked to relevant theory and then looks for appropriate methods to obtain rigorous information about those questions.

However, there is more consensus than might first appear to be the case. All researchers agree that, when feasible, randomized experimental trials provide the strongest data on which to base conclusions that an educational program or innovation is the actual cause of any change in student performance. All researchers also agree that education research needs to become more rigorous, and that the inclusion of more randomized experimental studies is important to the field.

Within the context of this controversy, this is the first of a series of planned articles on research about technology in K-12 education. In each article, we will begin with a broad question about technology in education. We will then describe one or more research studies that address that question, looking at the technology and educational innovations studied and at the research methods employed. We will summarize what's been learned from the research and, equally important, what has not yet been learned. These articles are written for COSN members, and are therefore intended to address the needs and interests of educational leaders and policymakers responsible for decisions about the educational uses of technology.

Broad Research Question: Does Technology Combined with Inquiry-Based Lessons Increase Students' Learning?

Many advocates for technology (e.g. Papert, 1993; Jonassen, Peck & Wilson, 1999) link the value of technology to teaching approaches that are inquiry-based, constructivist, project-based, or student centered. While varying in specifics, these approaches all emphasize the importance of students exploring ideas, conducting "hands-on" investigations, engaging in projects on topics they choose, working collaboratively, discussing their ideas, and gaining conceptual understanding. In general, these approaches view knowledge as something individuals construct for themselves through action, reflection, and discussion; not as something that can be simply transmitted from teachers or books to students. Becker and Riel (2000), in a national survey of the uses of computers for teaching and learning, found that teachers who have a more constructivist orientation are more likely to make use of computers in their classrooms. However, the survey data does not tell us why this relationship occurs, or whether the combination of approach and technology has positive impact on student learning. In this article, we will consider a major project that provides evaluation data on the use of technology-enhanced, inquiry-based approaches to teaching.

The Enhancing Missouri's Instructional Networked Teaching Strategies (eMINTS) program.

The Enhancing Missouri's Instructional Networked Teaching Strategies (eMINTS) program is administered by the Missouri Research and Education Network (MOREnet) in collaboration with the Missouri Department of Elementary and Secondary Education. The mission of eMINTS is "to support Missouri educators as they integrate multimedia technology into inquiry-based, student-centered, interdisciplinary, collaborative teaching practices that result in higher levels of student performance." eMINTS began as an initial demonstration project conducted in 1997 and is now a large scale program involving more than 16,000 students in classrooms across Missouri. Extensive research has been conducted throughout the life of the program by an evaluation team based at Missouri's Office of Social and Economic Data Analysis. A wealth of information about this project is available at www.emints.org and the full set of research reports is available at www.emints.org/evaluation/.

eMINTS is focused at the third and fourth grade levels and is a classroom-based initiative, so that one or more teachers in a school may have an eMINTS classroom while others do not. (More recent work has begun exploring whole-school implementations and the use of the eMINTS approach at other grade levels). eMINTS provides each participating teacher with technology for the classroom, extensive professional development, and ongoing support. Specifically, each classroom receives the following equipment:

• one computer for every two students, with a high speed Internet connection
• teacher workstation computer
• teacher laptop
• interactive whiteboard and projector
• digital camera and scanner
• printers
• Microsoft® Office and Inspiration software

eMINTS provides extensive professional development and ongoing support to help teachers implement the planned innovation. Each teacher in the program receives more than 250 hours of professional development over a two-year period, designed to help them learn new teaching strategies using multimedia technologies. Classroom visits and ongoing support from instructional specialists are also provided to eMINTs teachers as the program designers view a high level of "just-in-time" support as essential to enabling teachers to transform their practices.

Clearly, eMINTS involves much more than providing technology for classrooms; there is a particular educational approach based upon educational theories and there is extensive professional development and classroom coaching. This reflects prior research documenting that significant professional development is necessary to prepare teachers to make effective use of technology (Sandholtz, Ringstaff, & Dwyer, 1997). We know that research needs to consider not just the technology, but rather the educational value of technology-enhanced or technology-enabled instructional practices, in contexts that enable teachers to have the training, support, and resources to successfully implement those practices.

While the full eMINTS research program is too comprehensive to summarize completely in the space available here, some of the major questions, methods, and results are summarized below (see eMINTS 2002 and eMINTS 2003 for the full reports). Four of eMINTS' key research questions, and the answers offered by eMINTS, are presented first. Next, a commentary is provided in the following section.

Research Question 1: Does the eMINTS program increase student learning?

In the eMINTS research, the primary measure of students' learning is students' performance on the Missouri Assessment Program (MAP) state-wide standards-based test, which includes constructed response, performance, and multiple choice items. Third graders are tested in language arts and science and fourth graders are tested in mathematics and social studies, thus the study includes data in all four of the major content areas in the elementary school curriculum. Two eMINTs studies provide evidence of impact on student learning, one from the 2000-2001 school year, the second from the 2001-2002 school year. Combining the two studies, the data are from all of the students in 57 third grade and 103 fourth grade eMINTS classrooms. The results for the eMINTS students were compared to the results for all the grade 3 and 4 students in non-eMINTS classrooms in the same schools. Of the 10,000 students overall, approximately 30% were in the eMINTS group and 70% in the non-eMINTS group. An important aspect of the eMINTS research is that it uses data from far more students, teachers, and schools than most research into the impact of educational technology.

The results from each year show that the fourth grade eMINTS students had, on average, higher MAP scores than the comparison students in both mathematics and social studies, with the differences being statistically and educationally significant. The first year study found a similar but smaller positive effect in favor of eMINTS grade 3 students on both language arts and science, but in the second year study the two grade 3 groups did not show any significant differences. Overall, the data is much stronger showing positive effects on learning for grade 4 students in mathematics and social studies than for grade 3 language arts and science. The data available so far do not enable us to determine whether this is because the program is more effective with grade 4 students than with grade 3 students, because it is more effective with mathematics and social studies content than for language arts and science, some combination of those two factors, or other factors that may have influenced the results.

Research Question 2: Does the impact of the eMINTS program on student learning differ for different groups of students?

The researchers also disaggregated the data to look at test scores of eMINTS students in three categories: special needs students with individual education plans (IEPs), students eligible for Title 1 remedial services, and students from low income families (i.e., those eligible for free and reduced price lunch programs). Comparisons were made to students in these same categories from the comparison, non-eMINTS classrooms. In the first year study, the grade 4 eMINTS students in all three categories did better than the relevant comparison groups, but in the second year study this positive effect was replicated only for the low income students. For the third grade, there were some trends showing positive impact for students in these categories, but the differences were not large enough to be considered statistically reliable.

Research Question 3: Are the increases in student learning in the eMINTS program tied to specific changes in classroom practices?

The data on student learning is encouraging in a number of comparisons, especially at grade 4, but do not show significant positive results on student learning in all comparisons. This leads to another, very important question: Are the increases in student learning tied to specific changes in classroom practices? To consider this question, the researchers needed to determine how well each teacher implemented the recommended practices of the eMINTS approach, and then determine whether there is a relationship between the classroom practices and student learning. To do so, two different approaches were used. In 2000-2001 the eMINTS evaluation team was able to observe classroom lessons and classify them as teacher-centered lessons, student-centered facilitated lessons, or hybrid lessons. Students in classrooms characterized by student-centered facilitated lessons scored higher on the MAP test than did students in classrooms characterized by other types of lessons. In 2001-2002, a Hallmarks of an Effective eMINTS Classroom rubric was used by the classroom trainers to categorize teachers into one of four levels, reflecting how well they thought the teacher conformed to the eMINTS instructional model. The researchers then investigated the student learning results of the teachers at each level, ranging from emerging teachers who showed very little evidence of the eMINTS approach in their classrooms to proficient teachers who fully implemented the approach. The results are very clear: Students of eMINTS teachers who regularly applied the eMINTs instructional practices scored higher than students with teachers who do not apply these practices. Based upon both sets of data, the researchers argue that the recommended practices of the eMINTS program do have significant impact on student learning when they are implemented by the teachers.

This finding is in accord with a principle that has been consistently established in research: It is not the availability of technology or professional development, but actual changes in classroom practices that can result in gains in students' learning.

Research Question 4: Does the approach of the school principal influence the impact of the eMINTS program?

Another analysis looked at categories of leadership styles of the principals in eMINTS schools. Principals in the collaborative leadership category explicitly worked to foster a collaborative environment in their buildings by involving themselves in the overall school community and by acting as catalysts in support of the full implementation of eMINTS. Teachers in schools with these principals showed more success in changing their teaching practices through adopting the eMINTS teaching methods, and students in these schools showed higher test scores, than those whose principals had less collaborative approaches to supporting their teachers and the eMINTs innovations. This finding is related to one in many studies of educational innovations: The context of the implementation, especially the role of the principal, is critical to its impact.

Other research questions

The eMINTS research also includes studies of student, teacher, and parent views about the eMINTS program, collected through focus groups, interviews, and surveys. In all cases many positive views, along with information valuable for improving the program, have been reported. This qualitative information helps support and contextualize the quantitative findings and provide more insight into potential causes of the relationships that were found. The researchers have also explored classroom management approaches within eMINTS classrooms, different ways to categorize classroom practices, and other relevant questions. They are also currently conducting a study of the expansion of the eMINTS program throughout all grades and classrooms of individual schools. See www.emints.org/evaluation/ for more information.

Interpreting the research: What can we conclude?

The eMINTS researchers conclude that:

The results . show the consistent role of inquiry-based instructional practices in supporting student performance. In both cohorts, students of teachers who consistently apply the inquiry-based instructional practices emphasized by the eMINTS professional development program scored higher on the MAP tests than did the students whose teachers used other instructional practices. (eMINTS, 2003, p. 28).

The eMINTS researchers identify a clear and consistent relationship between teachers' use of the eMINTS recommended teaching practices and students' performance on the MAP test at both grades 3 and grade 4. The research also finds a relationship between school principals' leadership styles and the likelihood that teachers in a school will successfully adopt the eMINTS recommendations.

Here is where the controversy about research methodology comes into play. One side of the controversy will cast doubt on the eMINTS findings as valid evidence of the effectiveness of the program, noting that the research design does not involve random assignments of classes to the eMINTS and control conditions and therefore does not eliminate the possibility of confounding variables. For example, perhaps those teachers who successfully implemented the eMINTS practices were the stronger teachers in other ways, and students in their classrooms would have had higher test scores than other students even without the eMINTS program. Perhaps classroom assignment procedures resulted in differences between the students in the eMINTS and control classrooms at the start of the school year, and this contributed to the differences in test scores. Perhaps principals of schools with innovative teachers are better able to adapt a collaborative orientation-that is, the teachers' approach is influencing the principal's leadership style rather than vice versa. Overall, this side-the view of key policymakers at the U.S. Department of Education-will argue that the results are not necessarily due to the eMINTS program itself, since the research design does not eliminate the possibility of other factors causing the differences observed.

Those on the other side of the controversy will argue that while confounding factors are a logical possibility, the overall pattern of results from the data provide a strong argument that eMINTS, when implemented as recommended, is very likely to have positive effects upon students' learning. This side will argue that the eMINTS research uses appropriate methods to address the questions raised while conducting research within the constraints of real schools, classrooms, and research budgets. They will note the advantages of the large number of students, teachers, and schools studied, which makes it more likely that the results are relevant to many contexts and do not occur only in special, difficult-to-replicate, situations. They will also point out that some of the findings require analyses other than experiment and control group comparisons. For example, researchers could not randomly assign principals to experimental and control groups and then ask the first group to use a collaborative style and the second a non-collaborative style. In order to learn about the impact of different leadership styles of principals, they must study the styles that principals actually use-these cannot be changed and controlled for the purpose of a research project.

While practitioners and policymakers would like quick and simple answers, we need to recognize the dangers of over-simplification. If we expect all research studies on education innovations to provide a simple "thumbs-up or thumbs-down", we are likely to avoid research methods that can yield important insights into the complexities of implementing major innovations in our schools. We often need an extended period of observations, surveys, interviews, analyses of naturally occurring data, and explorations of relationships such as those conducted by the eMINTS project in order to establish the potential of an innovative educational program and learn how it can implemented successfully. Then, and only then, are we ready for randomized, experimental tests to insure the validity of causal connections between the innovation and student learning. This view is consistent with the resolution unanimously passed by the Governing Council of the American Educational Research Association (AERA) in January, 2003:

".A fundamental premise of scientific inquiry is that research questions should guide the selection of inquiry methods. Council recognizes randomized trials among the sound methodologies to be used in the conduct of educational research and commends increased attention to their use as is particularly appropriate to intervention and evaluation studies. However, the Council of the Association expresses dismay that the Department of Education through its public statements and programs of funding is devoting singular attention to this one tool of science, jeopardizing a broader range of problems best addressed through other scientific methods. The Council urges the Department of Education to expand its current conception of scientifically-based research." (www.aera.net/meeting/councilresolution03.htm)

In summary, thoughtful research into innovative educational practices usually requires multiple studies and analyses, using a variety of research methods, to delve into the many factors that can influence the success of an educational practice. The eMINTS research makes valuable contributions to the field but could be strengthened by additional randomized group studies on new eMINTS implementations. These studies could provide a stronger level of support for claiming causal connections between the eMINTS program and gains in student achievement. The groundwork for these studies has been laid by the extensive program development and research that has already been conducted.

References

• Becker, H. J. and M. Riel (2000). Teacher Professional Engagement and Constructivist-Compatible Computer Use. www.crito.uci.edu/tlc/findings/report_7/startpage.html
  
  
• eMINTS Evaluation Team (2002). Leadership Orientations of FY00 eMINTS Principals. School Evaluation Report, October, 2002. www.emints.org/evaluation/reports/fy00principals.pdf
  
  
• eMINTS Evaluation Team (2003). Analysis of 2002 MAP Results for eMINTS Students. Policy Brief. January, 2003. www.emints.org/evaluation/reports/map2002.pdf
  
  
• Jonassen, D. H., K. L. Peck, and B.G. Wilson. (1999). Learning with Technology: A Constructivist Perspective. Upper Saddler River, NJ, Prentice Hall, Inc.
  
  
• National Research Council(2002). Scientific Research in Education. (R.J. Shavelson & L. Towne, Eds). Washington, DC: National Academy Press.
  
  
• Papert, S. Mindstorms: Children, computers and powerful ideas. Basic Books, New York, 1980. 1993
  
  
• Sandholtz, J. H., C. Ringstaff, and D.W. Dwyer (1997). Teaching with Technology: Creating Student-centered Classrooms. New York, Teachers College Press.
  
  
• U.S. Department of Education (2003). Institute of Education Sciences. Identifying and Implementing Educational Practices Supported by Rigorous Evidence: A User Friendly Guide. (www.ed.gov/.../rigorousevid.pdf)
  
  

© EDC 2004