Trip Report #2: Assessing ICT's Educational Impact (Editorial)

	US Delegation to Europe, 2002: Internet Technology and Schools Agenda Working Groups Delegation Members London Briefing Materials Paris Briefing Materials Brussels Briefing Materials Bonn Briefing Materials Sponsor Background Information US Education System Review Trip Report #1: Government Demand/Resource for Education Technology/ICT Trip Report #2: Assessing ICT's Educational Impact Trip Report #3: Standards & Accessibility - SIF, SCORM, IMS, GEM, etc. Trip Report #4: As a Member of the European Delegation of CoSN Trip Photos

by Eric Hamilton

The October, 2002, COSN delegation visit to European countries yielded a number of important observations and potential collaborations on assessing ICT's impact on improving school learning environments. This working group paper will summarize impressions about the European countries, adding comments about individual countries, and close with a summary of next steps that COSN may wish to pursue in the context of a discussion of the types of assessment measures now in place.

The approach of this report is comparative: the commentary about the countries and organizations we visited are primarily drawn in contrast to US policy and practice. The report does not give a comprehensive view of ICT impact assessment policies and practices, in either the US or Europe. The intent, instead, is to convey observations drawn by a five-person committee over a five-day visit.

To the extent that generalizations are possible: the European approach to ICT impact assessment is quite different from the US approach, reflecting different dynamics and pressure points.

While the countries and organizations hosting our visit were not uniform in their ICT assessment priorities and resources, there were some important common themes. Broadly speaking, there are fewer political demands imposed on educational entities to assess or to quantify impact of ICT in Europe than in the US, but there more economic competitiveness pressures in Europe associated with increasing the scaled implementation of ICT. There is a lower emphasis in Europe on building a evidentiary basis (typically in the form of test scores) for education innovations in general and ICT innovations in particular. Otherwise put, salutary effects of ICT are more routinely assumed as a given in policy circles in Europe than in the US. To the extent that this generalization is accurate, there are both positive and negative implications on both sides of the Atlantic.

To explore these generalizations a bit further and then proceed to a discussion of opportunities that COSN may wish to pursue, it is important to introduce a critical contextual variable in the education systems here and in Europe. This variable is centralized versus decentralized education policy-making and resource decision-making.

Centralized versus decentralized is not an either-or condition either in the US or in Europe. For example, while the US system vests control of educational systems to state and local entities, with the federal government contributing only 7% of the aggregate K12 budget in the US, the role of the federal government has become highly leveraged under the current administration's No Child Left Behind (NCLB) initiative. NCLB places significant demands on K12 education systems to conduct annual assessments of student progress; additionally, NCLB-related legislation has specified forms of research that have priority standing in providing an evidentiary basis for policymaking. In practice, while the federal contribution averages only 7% of aggregate K12 education budgets, its share in distressed and impoverished areas is significantly higher. Further, while state and local education authorities theoretically can forgo federal funds and thus disregard federal mandates, these resources are too critical to them. They generally have no practical choice in complying with the demands exerted by the testing and assessment legislation and ethos that now characterizes the NCLB approach. That has both positive and negative dimensions, but at the least it certainly changes the dynamics for this comparison. While education decisions, and especially decisions related to ICT, are decentralized in the US, there is a ubiquitous and paradoxical pressure to size up all decisions, and especially decisions related to ICT, in terms of annual student achievement assessments. If an education innovation does not promise a "straight-path" impact on test scores in this environment, it is unlikely to be implemented.

In contrast, European countries span a continuum in terms of education resourcing and decision making. Germany provides no federal education funding, relying on the individual state governments; England provides all funding and a national curriculum. France maintains a system of greater local autonomy but a strong federal policy set. (Centralized features in England and France enable interesting large scale efforts that are important to understand and assess, such as national, digitized curricula, libraries and testbeds.) Belgium has a highly decentralized decision-making. At the least, there is no pan-European political pressure to assess ICT impact.

This is not only an artifact of the political dynamics of centralized national policy, but of a more cautious approach in the US to current generation ICT implementation, as a result of disappointments over previous ICT generations in the 1980s and 1990s. The US education enterprise embraced and adopted ICT sooner, with far greater resources, and with greater abandon than their European counterparts. Schools and school systems bought into educational technology fads and into buggy systems that promised more than they could deliver and that were engineered more for marketing niches and sales than for education. Billions of scarce local education dollars were invested in technology efforts that had little or no proof of impact, that produced little or no added value to classroom learning, or that were implemented haphazardly and inefficiently.

European systems bought into and invested in education technology at a slower pace. By lagging their US counterparts in the early penetration of education technologies, Europeans gained in perhaps an unexpected manner. They not only averted enormous expenses later deemed to be excessive or wasteful, their later investments have come in education technologies that have matured from early generation versions. This is analogous to a powerful phenomenon that is well-known in fields of industrial innovation. Industrial environments that reward innovation tend to produce innovation. Innovation, however, has a significant startup and shakeout cost that is not shared by mid and late-adopters. This phenomenon was often cited as an underlying dynamic in the rapid Japanese industrial growth in the 1970s and 1980s. Innovations formulated in other countries, especially the US, were refined and perfected by Japanese industry, which did not incur the initial costs of innovation. That phenomena perhaps has been replayed here and in a more intense fashion. European experiences in education technology have not been as negatively skewed or as relatively costly as have US experiences. European policy makers not only appeared to be less cautious than US policy-makers in embracing education technology; mindful that they have "lagged" behind in ICT penetration, there is pressure associated with economic competitiveness that drives allocation of resources into ICT implementation without the evidentiary demands that now exist in the US.

These impressions, to the extent that they are true, are nonetheless subordinate on both sides to an overriding reality that the state of the art in ICT impact assessment is poor.

The delegation's first stop was in the UK. The UK Department of Education shared a three level organizing classification for educational ICT analysis that included 1) ICT infrastructure; 2) instructional content; and 3) teacher/administrator professional development. This proved to be a useful trio of categories for discussion throughout the trip, and it is useful in organizing the types of measures and instruments currently in use on both sides.

At each of the three levels, instruments and measures used to assess ICT impact are, in the view of the committee, somewhat helpful but also disappointing and inadequate. ICT has already realized some of its remarkable potential to improve and transform learning, but the language to represent those achievements and potential seems to fall either into the world of exciting marketing videos or relatively uninformative statistics about improving ratios of computers per students or percentage of curriculum that is available in software packaging. Both have their advantages; neither seem to tell the most important ICT stories for either Europe or the US.

While COSN is not organized around developing new assessment instruments, it may exert an important role in articulating a more thoughtful approach to assessment. For example, standardized or norm-referenced paper and pencil tests are widely recognized as problematic measures of learning traditional and non-ICT enabled environments. When used to document effectiveness of ICT-enabled learning, their flaws are not only passed on but they are magnified, because such instruments are unable to capture critical value-added features of ICT-enabled learning. COSN may be in a position to work with European counterparts in developing white papers that articulate deeper pedagogical issues at each of the levels of analysis, explaining how or why existing instruments or measures yield inadequate impact indicators.

The committee believes that a more thorough approach to impact assessment will advantage policy-makers, whether they are responding to accountability mandates or to economic competitiveness pressures, for the simple reason that a clearer set of understandings about assessing learning is critical independent of the frame of reference and is a prerequisite for capitalizing on ICT promise.

Table 1 suggests for each level of analysis the kinds of assessments that the committee observed and suggests some possible opportunities for improving existing measures.

(Different items under third column need narrative specification.)