5.2 Educational Evaluation
Let’s talk about evaluation in education
Educational evaluation is a complex and evolving disciplinary field and has a robust yet complex research tradition behind it. Since the late 1960s, the debate on the concept of educational evaluation has increasingly focused on a number of components such as:
- the context,
- reference systems,
- the programs,
- materials and, in general, the occasions in which teaching-learning processes are situated.
Starting from the definition of Beeby (1997), who states that evaluation consists of the systematic collection and interpretation of data and leads, as an integral part of the process, to a value judgment directed toward action, it is possible to frame an epistemic field of educational evaluation in which we can distinguish certain characteristics of evaluation in the educational sense.
What, then, does it mean to evaluate in the educational sphere?
If, precisely, to evaluate in general means to make a judgment about a given object, to evaluate in education, according to an Italian author, Galliani, means to make judgments about formative and teaching actions (or a merit to a complex of organized actions and here we refer to programs and/or courses), intentionally designed or carried out to guide and develop learning (individual, collaborative, organizational) in the recipients, with effects on the educational, economic and social system. In this sense, the evaluative discipline is founded on the use of methods and tools proper to empirical and experimental research in education (Galliani, 2009).
According to this pedagogical perspective, evaluation, along with planning and communication, is an ontological and methodological component of educational science, in that the educational experience is deeply linked to the evaluative process, which is, therefore, configured as a complex pedagogical fact that touches on the dynamics of teaching/learning and requires teachers to make conscious choices, embedded within specific projects.
Barbier qualifies the evaluative act as “a dual process of representation, the starting point of which consists in a factual representation of an object and the ending point in a codified representation of this same object” (Barbier, 1989, p. 76). When we speak of evaluation we refer to two closely related operations or moments: the measurement stage and the proper evaluation moment (Vertecchi, 1993).
The first moment- that of measurement -privileges a quantitative approach and consists of obtaining data according to criteria of certainty; subsequently, with the evaluation process proper, we proceed to the qualitative interpretation and description of the ascertained data according to the criteria made explicit (Notti, 2010).
Measurement and evaluation are the two moments of verification, one finalized to the other; evaluation, specifically, is rooted in criteria that are autonomous from the methods and tools that are used in measurement: in fact, it is based on the set objectives and their achievement (Coggi and Notti, 2002, p. 23).
Often, however, these two different phases are overlapped and confused. In fact, as early as 1998, another Italian author, Vertecchi (1988) in the text The Words of the New School describes the tendency to confuse the two terms. Verification and evaluation in fact are not to be understood as synonyms because “from a conceptual point of view, verification precedes evaluation […],” since “the priority of verification consists in the fact that it has the task of detecting objective elements, while evaluation is expressed by attributing a value to these elements […]” (Vertecchi, 1988, p. 69).
Evaluation is a process and, at the same time, a tool for producing information describing the correspondence between the characteristics of the educational proposal, individual and contextual needs and social purposes of education. Domenici has repeatedly pointed out that the evaluative act can be considered a procedure aimed at “attributing value to facts, events and the like, in order to be able to reward or sanction, promote or inhibit them in relation to the most important reasons for which the evaluative action is carried out, but above all in relation to the purposes that the evaluator (or those on his behalf) actually wants to pursue” (Domenici, 2014, p. 24).
Dimensions of analysis and evaluative approaches
It is since the 1990s that, at the same time with the emergence of new theories on learning processes, the topic of assessment itself has also been the subject of profound reflections aimed at exploring its forms, functions and meanings, again in the light of innovations in education. Indeed, in this sense, theoretical models and evaluation practices have followed the evolution of the different theoretical paradigms (positivist, pragmatist, constructivist) that characterize the teaching-learning experience.
Educational assessment affects several areas: in addition to grade-giving strategies, the assessment process concerns all the different stakeholders involved in the educational process.
When evaluating we try, first of all, to answer some fundamental questions about educational facts: what to evaluate, why to evaluate, when to evaluate, how to evaluate, questions that help us circumscribe some dimensions of evaluation (Notti, 2010).
From this perspective, evaluation is a complex phenomenon that is not reduced to a single moment, but accompanies the entire educational process and is a substantial part of it (Falcinelli, 2014). In fact, starting from the definition matured from the scientific comparison between Bezzi (2000) and Palumbo (2001), we can consider evaluation as a cognitive activity aimed at providing a judgment about an action (or complex of coordinated actions) intentionally carried out or intended to be carried out, intended to produce external effects, which is based on social science research activities and follows rigorous and codifiable procedures. This definition allows us to specify some elements referring to object, purpose, and procedures.
In relation to the object:
In relation to the object, we can say that evaluation must be about an intentional intervention that has explicit purposes, aimed at affecting the outside world and employing resources and tools designed to achieve it.
In relation to purpose:
With respect to purpose, evaluation is qualified by making a judgment based on explicit criteria and referring to one or more characteristics of the intervention’s object, perpetrators, recipients or beneficiaries.
With respect to procedures:
Finally, with respect to procedures, the formulation of judgment is the result of comparative activities developed within a process, conducted according to specific procedures and methods of data collection, processing and analysis (Palumbo, 2001).
The assessment process generates considerable consequences in the individual’s educational journey: the outcome of an assessment, whether positive or negative, implies a reallocation of resources on the part of the student.
The impact that assessment has with the educational context, which is always changing and inhabited by different individuals, has resulted in a dense web of studies and research in the field of assessment in order to identify a model consonant with social needs.
In the literature, there are numerous contributions available, and the equally conspicuous classifications make, from time to time, one aspect evident rather than another. Trinchero (2014) describes eight approaches, which we reproduce here in a very synthetic manner:
- goal-based,
- goal-free,
- responsive,
- judgment-oriented,
- antagonistic,
- empowerment,
- decision-based
- and success cases.
The goal-oriented, or goal-based, approach draws on a realist epistemology and has as its main purpose to establish the achievement of the goals that the intervention set for itself; the focus is on discrepancies between the observed situation and the expected situation (Tyler, 1942).
The Tylerian evaluative model characterized U.S. evaluative research and practice in the first half of the twentieth century and conceives of evaluation as a procedure through which one is able to determine the effectiveness of an educational program, based on the achievement of objectives.
The goal-free approach draws on an interpretivist epistemology and aims primarily to describe what the effects of the intervention were, regardless of the intent behind the action. The focus is on the observed situation, on the outcomes actually produced and not on the achievement of predetermined goals (Scriven, 1972).
The responsive approach is based on an interpretivist epistemology and aims to bring out instances in the subjects involved. Within the relevant literature it is also known as responsive evaluation (Stake, 1975).
The judgment-oriented approach is based on a realist epistemology and aims to arrive at the formulation of an authoritative judgment by outside experts (Patton, 1997).
The adversarial approach is based on an interpretivist epistemology with the aim of bringing out the pros and cons of the intervention, shifting the focus to value attribution (Wolf, 1979).
The empowerment approach starts from an interpretivist epistemology and aims at the acquisition of awareness (from the learners point of view) about their own potential and limitations during the evaluative moment (Fetterman, 2001).
The decision-based, pragmatist approach aims to support the decision-making process by virtue of evaluative action (Stufflebeam, 1973).
Finally, the success-case approach highlights the potential of the intervention itself, taking as its focus the experiences of individuals who have been particularly successful (Brinkerhoff, 2003; Trinchero, 2014).
It should be emphasized that these distinctions of approach in educational reality turn out to be blurred, since a good evaluative process usually resorts to multi-method strategies with a view to integrating different approaches in a complementary way. The act of evaluating is fundamentally related to the act of knowing (Dewey, 1976): knowledge to be reliable and complete requires the joint use of a plurality of sources. Similarly, evaluative practice should take advantage of the possible synergy of different approaches, each valuable in shedding light on different aspects of educational action.
Let us now look at some of the main approaches that have characterized the evolution of the evaluative field
The positivist-experimental approach
It was born, according to a leading Italian researcher, Nicoletta Stame with the aim of finding a fruitful solution to serious and frequent social problems, such as unemployment or the decay of urban centers.
In this sense, according to a purely rational logic, programs are thought to be articulated in goals to be achieved, means by which to achieve them, and expected results, and to be placed in a policy cycle that goes from decision to implementation to evaluation and new decision (Stame, 2001, p. 26).
From this perspective, then, what is the significance of evaluation?
Evaluation thus pursues the task of monitoring whether project objectives have been achieved, through comparison with the results achieved.
In this sense, the definition of objectives represents a fundamental phase in the implementation of actions aimed at the desired change: in fact, they are an integral part of the program and encompass the very resolution of the problem.
Evaluation according to the positivist-experimental approach also allows the generalizability of an intervention, or a series of interventions, to similar situations. This also determines consistency between the instruments adopted for evaluation and the action to be evaluated. The function of such an approach to evaluation “corresponds to an idea of ‘instrumental’ use of evaluation: evaluation should provide information inputs that policymakers should turn into decisions” (Stame, 2001, p. 28).
The pragmatist-quality approach was born in response to positivism.
According to this approach, evaluation has the task of managing and controlling organizational procedures to see the established training standards realized, that is, how closely the outcomes obtained approach what was the model or reference standard.
A founding father of this evaluation model is Scriven, who calls it the prospectivist or transdisciplinary approach, in which the evaluator assumes as objective a role as possible in relation to the program objectives themselves.
The evaluation must therefore express itself as goal-free evaluation, recognizing the degree to which similar programs of action approach the standard. Such an evaluation refers back to the concept of quality, precisely as the achievement of pre-established standards (Stame, 2001).
In this approach, too, evaluation has an instrumental function, but more strictly it is a function of administrative decision-making.
The constructivist approach — social process
This third model is described as a compendium of a number of models, including references to Guba and Lincoln’s (1992) research around the “fourth generation of evaluation” and Cronbach’s (1980) analysis that sees evaluation as a process of a social nature that takes place in a changing socio-political context.
Evaluators, in this particular type of approach, engage in continuous discussion with stakeholders, and criteria are established from time to time, depending on the specific situation. Using the terminology introduced in the field of educational evaluation by Barbier (1989) and Hadji (1989), the référent, i.e., the objective pursued, the model to which the evaluator must refer, is not established aprioristically, but is constructed in dialogue and confrontation with stakeholders, referring in any case to similar situations, which we might define as best practices. In summary, “evaluation must take into account the fact that each time a program is implemented it changes in contact with the context: the problems felt by various stakeholders will be different and the conclusions reached for one program cannot be generalized to other situations where similar programs are implemented. However, this does not mean that lessons cannot be drawn that can be applied elsewhere” (Stame, 2001, p. 32).
According to the constructivist-social process approach, assessment would have a cognitive use both of the individual situation to which it relates and for the purpose of identifying strategies for the improvement of the beneficiaries of the process itself. Galliani calls the third paradigm constructivist (or also constructivist-social), since he believes “that knowledge is constructed from the experience of the learner, from the cultural tools (symbolic and technological) that he knows how to use, from the contexts he attends and from the forms of social negotiation that are allowed, conceives educational evaluation as an interpretation of educational actions as a process, with the task of regulating its development and producing meaning through the social construction of reference values by all stakeholders” (Notti and Galliani, 2014, p. 58). According to this perspective, the evaluation process would be a negotiated act among stakeholders, and the sharing of meanings would form the basis for initiating and implementing the process itself. The evaluator therefore must be an “initiator of meaning for the other” (Notti, 2014, p. 15).
Toward a mixed method: the combinatorial-creative model
A significant contribution to the research on the evaluation process in the educational field comes to us from Notti (2010; 2014), who summarizes the different approaches into two methods: the subjectivist method that is based on a case study and the objective method that is based on objectives. The first is called qualitative-hermeneutic because it has as its object the study of the educational situation, is based on the contribution of experience, values the role of intuition and has a holistic approach. The second method starts from the belief that evaluation is based on verifying the achievement of planned objectives and emphasizes primarily the aspect of measurement.
Each of the two models has advantages and disadvantages, and there is now a consensus that they are not to be understood as opposed to each other but should be conceived of as complementing each other and should be used in combination, favoring one or the other according to different evaluation needs. Quoting Notti, “a possible common ground between the two models can be found in an approach that distinguishes measurement from evaluation by leaving to the former the quantitative data, that is, those data that meet the criterion of certainty, and to the latter the qualitative interpretation and description of the ascertained data according to criteria […]. What has been said supports the need for a model in which the quantitative experimental and the qualitative hermeneutic methods of investigation are factually linked in a logic that uses their possible overlaps” (Notti, 2014, pp. 18-19).
In conclusion, the preferred way forward in order to be able to improve and innovate professional practices in education is outlined in the creative combinatorial model. These principles, when applied to educational assessment, can promote the regulatory function of assessment aimed at improving learning outcomes, teaching performance, and the quality of education.
Use of indicators in evaluation
When evaluating, it is necessary to decide the object to observe and the way through which to do it. That is, deciding which indicators to use and the method to employ them. Indicator is a “characteristic, or observable or calculable variable (or relationship between variables) that provides indications about a certain phenomenon (Leone, Prezza, 2003, p. 141). The indicator is used to give information, therefore it must be understandable and clear in expressing the phenomenon for which it was created. Indicator must respect methodological, conceptual and feasibility characteristics.
The characteristics of a conceptual nature are:
- Pertinence: an indicator is pertinence if it observes the essential characteristics of the object to be measured.
- Relevance: a relevant indicator is identified in the importance of the characteristic of the phenomenon to be observed.
- Specificity: a specific indicator minimizes the risk of false positive, for example it does not lead to the assumption that a condition is present when it is absent.
- Sensitivity: a sensitive indicator emphasizes the differences in the intensity of the phenomenon visible.
The methodological characteristics that indicators must adhere to pertain to the measurements:
- validity: A measurement is valid if it measures the phenomenon (or its aspects) that one intends to analyze;
- reliability: a measurement is reliable if it is precise and repeatable.
The feasibility characteristics refer to the final choice of the indicators to be used, which is carried out by looking at the properties of the indicators, the purposes, the importance and the available resources (Leone, Prezza, 2003, pp. 142-143).
Moreover, the indicators can be subdivided on the basis of multiple criteria: (a) criterion of proximity of the indicator with the phenomenon it wants to indicate. The indicators are direct and indirect: the direct ones show the phenomenon directly, while the indirect ones show elements that are in a relationship with the object of the analysis. The indicators can be defined according to the objectives for which they are used, for example the evaluation indicators allow to evaluate the achievement of the objectives of a project, while the process indicators evaluate the modalities of realization of the activity (ibid, pp. 143-144).
The fourth-generation assessment
According to Guba and Lincoln (Guba & Lincoln, 1989), evaluation is a fundamentally social, and political activity, oriented by institutional cultures and their values. In the words of these authors: The results of the evaluation are not descriptions about “how things really are” or “how things work,” or about the “truthfulness” of a state of affairs; but they are actually meaningful constructions that individual actors or groups generate to “make sense” of situations in which they find themselves acting. The aspects that are detected are not “facts” in the sense that is generally given to this term, but are, rather, literally created through processes of interaction that include the evaluator (so much for objectivity!) as well as the many stakeholders who get involved (and risk) within the evaluation. What emerges from this process is one of many possible constructions representing the different realities of the case at hand (Guba & Lincoln, 1989, p. 8),
Fourth-generation evaluation or fourth-generation assessment, as the authors call it, emerges from the responsive evaluation perspective of Robert Stake (Stake, 1975); it is a type of evaluation that considers epistemological and ontological principles of reality as constructed from lived experience, where subjectivity obtains a central role.
This approach is based on two important elements:
- (a) responsive focusing-that is, an evaluative focus determined by the questions posed by the evaluation stakeholders themselves, and
- (b) a constructivist methodology based on a hermeneutic approach that adopts the interpretation of sense-generating processes as a fundamental way of understanding (and thus evaluating).
Fourth-generation evaluation thus emerges as a proposition in which evaluation is an accountable relational process (Abma & Widdershoven, 2011). where the evaluating subject no longer looks at the world from the outside but interacts and takes personal responsibility for the developmental process that one’s participation in an evaluative context can trigger. This is indeed, a perspective where autonomy, reflexivity and intersubjectivity are placed at the basis of interactive and dialogic processes of evaluation.
In such processes, the evaluator and participants learn from each other, and understand that they are jointly responsible for the results of the evaluation, which itself becomes a formative process (Abma & Widdershoven, 2011, p. 672).
This perspective moves far beyond the traditional qualitative approach, where the evaluator works through a responsive paradigm, observing and doing interpretive but isolated work. In this new perspective, a process of mutual learning is triggered, which can define transformative forms of one’s reality of practice.
The aim is to develop a dense and holistic understanding about the complexity of social practices, from an insider,. -that is, internal to the evaluation- rather than as an outsider. Our main assumption is that this approach can be part of the evaluative method we adopt in a deep, complex view of quality, where we integrate the evaluation of standards (necessary, in a context of inter-institutional and transnational communication) with a complex, constructed view of what quality means for the actors involved.
