5.4 Student Assessment in Teaching and Learning
Over the past three decades, the topic of student assessment, in other words, the systematic collection of evidence useful for making judgments about student learning (EPPI, 2002), has attracted increasing attention in the national school policy debate (Donaldson, 2013). Underlying this attention are different processes, such as
- increased demands for effectiveness, quality, and equity placed on schools to meet continuing social and economic changes;
- a gradual affirmation of the principles of decentralization and school autonomy that concomitantly requires the activation of mechanisms for monitoring assessment in different local contexts;
- the development of information and communication technologies that has offered the possibility of managing and sharing assessment data, both on a large scale and on an individualized basis;
- a stronger reliance on assessment results for evidence-based decision-making (Organization of Economic Co-operation and Development, OECD, 2013a).
Let us now briefly look at an overview of school learning strategies, based on data from the most recent international comparative surveys, focusing the discussion around a few guiding questions:
1. Who assesses? With what objectives do we assess? What do we assess? How do we assess?
2. Who evaluates?
Responsibility for student assessment in different national assessment systems is generally shared by multiple actors at different levels depending on the purposes for which it is conducted. Supra-school authorities, regional or national, typically have charge of assessment processes at key transitions between cycles of education (e.g., at the conclusion of lower secondary school); while teachers generally have the task of assessing, in both formative and summative terms, their students during their learning journey.
Traditionally, a distinction is made between summative assessment whose goal is to provide a summary judgment of past learning (assessment of learning) (EPPI, 2002), and formative assessment (assessment for learning) whose goal is to provide a useful judgment to guide the future teaching-learning process (Bloom, 1969; Scriven, 1967).
Centralized evaluative models tend to be recognized as highly reliable, although low in validity. Conversely, high validity is attributed to teacher-led assessment, however, in the face of low reliability, although, with respect to this last point in particular, there does not seem to be a consensus literature (Bishop, 2006; Crooks, 2004). In general, from the data of the OECD TALIS 2013 survey (OECD, 2013b) there is a widespread affirmation of the principle of autonomy of schools and teachers in defining and applying student assessment strategies (Figure 1).
In primary education in particular, teachers are given almost exclusive responsibility for student assessment; while at higher levels of the education system this responsibility is shared and/or integrated with that of supra-school authorities. In this distribution of tasks, teachers are typically assigned assessment in the various disciplines of the curriculum, while the central level is more frequently given the task of assessing specific knowledge/competencies in a limited number of disciplines, and only at certain levels of education. On the role of teachers, OECD (2013a) stresses the need to involve teachers (and more generally school stakeholders) in the definition of the assessment system itself, so as to create consensus around it and stimulate the development of assessment skills at all levels, including through shared choices. In some countries (e.g., Northern Ireland, New Zealand, the Czech Republic and the Netherlands) an important role is also played by private agencies/companies that provide models and tools for evaluation. For example, in the Netherlands, the Central Institute for Test Development (CITO) provides a system, called Leerling Volg Systeem (LVS), for monitoring students. Available for both primary and secondary education and, in an adapted version, for students with special educational needs, it consists of a set of standardized (partly adaptive) tests geared primarily for formative assessment, which can be used longitudinally to monitor and track students’ progress in various disciplines, such as (for primary education) language, arithmetic, world orientation, socioemotional development, English, science and technology.
3. Why do we evaluate?
With reference to objectives, it is possible to make an initial general distinction between systems that use evaluation data for reporting purposes (accountability), i.e., to certify/attest to processes and products of the school system (e.g., to the public or the ministry) or for development purposes (development), i.e., to make improvements to the processes/products of the school system (e.g., redistribute economic resources in order to resolve critical issues that have arisen).
In this regard, according to data from the “Education at a Glance 2011” survey (OECD, 2011), Italy would be among the countries that make low use of results for the first objective (accountability) and moderate use for the second (development);
In other words, the large investment made in terms of the collection, management and analysis of school evaluation data would, with respect to these two objectives, be partially wasted due to a lack of or limited use of them. With respect to these goals, countries that place more emphasis on accountability purposes would tend to be characterized by a more frequent high-stakes approach in the use of standardized tests, emphasis on summative purposes, use of external inspectors to evaluate schools, and public reporting of performance results from a comparative perspective. On the other hand, systems that are more oriented toward developmental purposes would tend to be characterized by a low-stake (or no-stake) approach to testing, more frequent use of formative assessment models, use of results on teacher evaluation for the proposal and redefinition of professional training and upgrading pathways, and use of self-evaluation by schools (devising possible external structures as support for them) (OECD, 2013a).
In Europe, according to a survey by the Eurydice Network (2009), nationally based standardized assessments in primary and lower secondary education are predominantly used to make decisions about pupils’ educational careers (e.g., promotions, issuing certifications, and/or subdividing pupils by grade level – this is the case in the French-speaking community of Belgium, Denmark, Germany, Italy, and Luxembourg) and for monitoring and evaluating educational institutions or the educational system as a whole (e.g., comparing results between institutions and/or evaluating system performance-it is the case of the Flemish community of Belgium, Spain, France, Italy, Austria, Portugal, and Finland); while less frequently such assessments are aimed at identifying student learning needs (as in Denmark, Ireland, France, England and Sweden). In spite of this, there is an increasing prevalence of formative assessment principles in the frameworks of different national school systems (Figure 3). In Norway, for example, whose system has traditionally been based on the principles of decentralization and school autonomy (especially in the primary and lower secondary cycles),6 surveying each student’s competencies en route with formative purposes is a key objective of the school system, and providing precise feedback to students on their progress in relation to national competency goals is an obligation for schools (Norwegian Directorate for Education and Training, 2011).
In this sense, the incorporation of the principles of formative assessment into national regulations appears to be a factor of particular interest, given the vast pedagogical-didactic literature that brings evidence on the positive impact of this strategy in improving learning (Black & Wiliam, 1998; Hattie, 2009; 2011; Popham, 2011).8 In this regard, the OECD (2013a) itself stresses the importance of the formative aspect of assessment in the national regulations. In this regard, however, OECD (2013a) itself points out an important criticality: in the face of explicit policies and statements in regulations that emphasize the need to implement and enhance the formative aspect of assessment, there is widespread generality in the specification of effective and systematic practices within schools and/or classrooms. A critical issue, the latter, which is linked to another fundamental issue: the professionalization of school leaders and teachers in assessment strategies. With reference to the former, OECD (2013b) data show how today the role of school leaders is highly unbalanced, placing an increasing burden on them stemming from administrative-bureaucratic management burdens, to the detriment of pedagogical-didactic leadership tasks. Turning our gaze to teachers, the same survey shows that the percentage of Italian lower secondary school teachers expressing a high need for professional training in the area of student assessment (22.9 percent) is significantly higher than the OECD average (11.6 percent).
4. What do we evaluate?
In all OECD countries, students are assessed by their teachers on all subjects in the national curriculum. Centralized standardized assessment, on the other hand, while there is an uneven landscape, is generally centered on a few subject areas that are considered priorities, such as language and mathematics (Figure 4) 9 . Consider, moreover, that national school curricula have undergone radical reforms in the last decade, a period in which there has been an increased emphasis in various education systems on the development of complex skills, often traceable to the principles of lifelong learning, such as critical thinking, problem solving, creativity, information literacy, and active citizenship, which appear to be cross-cutting to more traditional disciplines (OECD, 2013a). In Europe, for example, since 2006 there has been a gradual introduction into the curricula of school systems of eight key competencies: communication in one’s own language, communication in foreign languages, competence in mathematics, science and technology, digital competence, learning to learn, social and civic competencies, sense of initiative and entrepreneurship, and cultural awareness and expression (Recommendation 2006/962/EC). In the United States, as many as 16 countries have introduced into their curricula a set of competencies deemed critical to the challenges of the 21st century (Partnership for 21st Century Skills), including learning and innovation skills, ICT skills, and life and work skills. UNESCO, in turn, has promoted a reform of school systems, adopted for example in Mexico, directed toward the development of “life skills,” such as critical thinking skills, creativity, social and communication skills (Singh, 2003).
The introduction of such competencies into the national curricula requires the development of assessment models in line with them, which are often referred to as “performance-based” (Linn, Baker & Dunbar, 1991), the characteristic feature of which is to assess simultaneously an integrated set of knowledge and skills through the requirement to perform tasks, such as oral presentations, research, experiments, projects, collaborative tasks, problem solving tasks, portfolios, etc. However, in this regard, OECD critically notes how, in the face of increased emphasis on such complex transversal competencies, the use of innovative assessment models is rather limited and how they are often limited to the detection of basic knowledge and skills (OECD, 2013a). Obstacles to their widespread adoption may be attributable to several factors, such as the difficulty in defining valid and reliable instruments, the difficulty in translating complex competencies into observable and measurable processes, and the increased burden of development. Beyond these objective difficulties, the risk is that there is a misalignment between competency-centered curriculum and assessment centered on basic knowledge and skills: instead, there is a need to develop a consistent and logically coherent framework in which there is timely alignment between objectives, curriculum and assessment (ibid.).
5. How do we evaluate?
The main tools used in standardized assessment can be traced to four macro-categories (Looney, 2011):
multiple-choice tests: require the selection of one answer from several options, generally useful for assessing knowledge, including high-level knowledge, but less appropriate for complex skills (they have, in addition, the advantage that they can be corrected automatically and result in comparable structured data);
adaptive tests: sequences of questions administered by a computerized system that can adapt the difficulty of the questions as a result of the answers given by the student (since not all students answer the same questions, there is less opportunity for comparison);
performance tests: allow the assessment of complex skills, through the proposal of tasks (typically require expert evaluation and may have limitations in reliability, due to the subjectivity of the assessment);
computer-based performance tests: allow for the assessment of complex skills through the use of ICT for task administration and assessment. Closed-ended questions and written tasks requiring predetermined sequences of steps, probably due to the possibility of automatic correction and greater comparability and reliability, are among the most popular in centralized assessments (Figure 5) (OECD, 2013a).
However, in recent years, following the more frequent reference to complex transversal competencies, more sophisticated approaches to assessment are being introduced in some countries. For example, in Austria, the 2008 structural reform of the New Secondary School was accompanied by a pedagogical reform, aimed at the individualization of teaching, social learning, and the integration of children with special educational needs, and in line with these intentions, schools are required to adopt formative assessment strategies, focused on the evaluation of complex performance, through self-observation, self-evaluation, and portfolios (Specht & Sobanski, 2012)12. In Belgium (Flemish community), assessment tools have been developed by institutions/organizations outside the school for the assessment of noncognitive performance, related for example to attitudes and personal well-being (among them, the most widely used is the SAM-scale – Scale for Attitude Measurement – aimed at the detection of attitudes, such as diligence, flexibility and responsibility, and used for guidance purposes) (Flemish Ministry of Education and Training, 2010). In addition, in support of formative assessment promoted by teachers and schools, ICT tools and on-demand digital databases are now available in some countries, which can be freely accessed, taking responsibility for the choice of the most appropriate tools, timing and objectives of the assessment themselves. In this regard, OECD (2013a) suggests the adoption of multi-approach systems, adaptable to individual, local and special needs, with availability of different tools and methods so as to collect and integrate evidence of different nature for student assessment. In New Zealand,13 for example, the Ministry of Education has set up a web environment, called the Assessment Tool Selector, where some 80 different assessment tools are made available for teachers to try out and use free of charge, selectable on the basis of subject area, skills, school level, correspondence with the New Zealand curriculum; and so on. Similarly, in Australia, the government has made available to teachers an environment, called Online Diagnostic Tools, which provides access to useful resources for monitoring student progress, linked to digital learning resources; and in Norway, the Directorate for Education and Training has developed a special web environment where programs, sample materials, videos, and tools for training in formative assessment practices are available.
6. Instructional design and evaluation
6a) Instructional system and evaluations
In this last section of the MOOC we will approach the topic of instructional design from a new point of view, the evaluative one. Given the impossibility of an unambiguous and undisputed definition of instructional design, Molenda (2009) suggests the use and accompaniment of the adjective system to instructional. In fact, the acronym ISD (Instructional Systems Design Model) is used to indicate a more generic, and therefore more global, idea of the design and development of didactic and pedagogical systems. (Sannicandro p. 271). The construction of a system around design, moreover, allows several aspects to be observed and taken into consideration simultaneously.
The instructional design (ID) try to create a framework in which to choose instructions to get changes and improvements according to knowledge, experience and abilities. There are many perspective and different point of view about how face the instructional design process. Some designers focus on the process as the main drivers, some authors adopt a prescriptive approach to ID in which the process is guided by rules to get the combination for an optimal education; others have more explorative models in which ID can find creative and alternative solutions to different problems. (Weston, McAlpine, Bordonaro, 1995).
The different instructional design models are united by a number of recurring elements, such as the definition of learning objectives, content structure, teaching strategies, the way of presenting study material and the evaluation. What distinguishes them is the type of approach used in dosing and assembling these elements (Trentin, 2014).
Like ISD, evaluation includes multiple aspects; therefore, it would be more correct to speak of evaluations instead of “evaluation”; the objects taken into consideration are different, as are the modalities and perspectives. The first thing to do, therefore, is to include and associate the evaluation with a specific pedagogical perspective and framework. That is why we will focus on the meaning of evaluation practice in relation to the context and perspective of instructional design.
Inside this system, in fact, we can consider the set of practices that follow the project processes related to design, development, implementation, management, analysis and, finally, evaluation. (Reiser, 2011b). The combination of these elements and the meaning attributed to each of them can generate different procedures and effects for the imagined and proposed design (Sannicandro, De Santinis, Bellini, Minerva, 2021).
For Raiser (2011b), in fact, the field of instructional design includes “the analysis of learning and performance problems, and the design, development, implementation, evaluation and management of instructional and non-instructional processes and resources aimed at improving learning and performance in a variety of contexts, particularly in educational institutions and workplaces” (Reiser, 2011b, p. 57).
6b) The evaluating process in his phases
People often imagine the evaluation process as a final step; this thought is erroneous because the evaluation phase, in fact, is transversal to all the others and concerns both the overall planning and delivery process. In the first case we refer to the on-going evaluation of what is implemented and developed while presenting the course; in the second case to the level of effectiveness and satisfaction of the course completed in its various parts; assuming particular attention to the results and learning produced (Trentin, 2014).
“Formative evaluation can be conceived of in a variety of ways: as a continuous iterative process that occurs throughout the design of instruction (e.g., McAlpine, 1992); ad a process that occurs at a specific time later in the development of the instruction, such as when trying out instruction that has reached draft stage (e.g., Dick & Carey, 1990); or as a process that continues beyond the design and development of instruction to include verification and revision throughout the life of the instruction (e.g., Komoski & Woodward, 1985)” (Weston, McAlpine, Bordonaro, p. 31, 1995).
In this process, importance is given to the analysis and the research of the aspects that create an educational experience; the analysis of the problems, the set-up, the design thinking, the development and realization that results in an evaluative focus within the instructional design model. This investment in evaluative processes is necessary and functional to the improvement, growth and adaptability of an educational experience to the context and target audience.
Instructional evaluation has to do with the understanding and the monitoring of the impact and the effectiveness of a process and his designed activities. Existes methods that can evaluate how well an instructional program was designed, developed, implemented and managed. It is the process in which it’s possible to recognize and describe weakness, consequences and recommendations for improvements thanks to specific and optimal techniques that are the main theme of this section. The discipline of instructional evaluation produces knowledge about techniques, models and perspectives trying to find the best combination for different situations in which each of those evaluations models is optional. (Reigeluth, 1983)
“Formative evaluation is an essential part of instructional design, the idea of formative evaluation is not unique to ID; it has been seen as central in many educational endeavors for years (…). However, the term formative evaluation was first coined by Scriven (1967) to refer to the evaluation of educational programs during their development phase for the purpose of improvement. (Weston, McAlpine, Bordonaro, p. 30, 1995). This way of understanding evaluation it’s nowadays similar to the way in which the educational community evaluates curricula, materials, programs and teaching.
By the way, in every model and reflection proposed in the years, formative evaluation is seen as a necessary component of the instructional design process, so formative evaluation can be intended by all these ways (Weston, McAlpine, Bordonaro, 1995).
6c) Goal of the evaluation in ID
The main objective of a formative and instructional evaluation is to validate and analyze the achievement of the settled goals, to observe and monitor the process, to evaluate the impact and efficiency of the course designed, planned and proposed with its instruments and material. Final results of this process is the acquisition of different feedback, important for the improvements, changes and new perspective of design.
The evaluation focus within instructional design, therefore, is on the process inscribed in the design of the course, the products obtained and the learning generated in those who use it. The choice and definition of evaluation criteria in an ex-ante phase is therefore necessary in order to know what and how to evaluate while going and to give linearity to the evaluation process that will lead to the analysis of results in an ex-post phase.
As a consequence, the same course design, delivery and evaluation strategies will have to integrate to a greater extent targeted actions for the collection and analysis of the learners’ needs, preferences and learning styles also with respect to the levels of success recorded both in relation to the achievement of the learning objectives set by the teachers and in terms of the skills developed. (Sanniccardo, De Santis, Bellini, Minerva, 2021).
The instructional design intends to pay attention to the processes of inclusion, differentiation, accessibility, participation and personalisation also in the evaluation phase (Sannicandro, De Santis, Bellini, Minerva, 2021). In fact, “it is not just a matter of collecting data on learning and teaching processes in online environments and courses, but a reflection on the possibilities offered by digital tools and resources to promote greater use of activities related to interaction processes, evaluation methodologies with different feedback (Vegliante & Sannicandro, 2020), the choice of digital resources and tools, the presence of case studies, simulations or small group work, etc.” (Sannicandro, De Santis, Bellini, Minerva, p. 272, 2021). In this way also the users will have a central and active role in the evaluation phase and in the monitoring actions of the training course, giving voice to an analysis expressed not only in terms of attendance, participation and in the collection of quantitative data, but expressing a judgement on the effects, learning and quality of the contents and resources proposed by the instructional designer.
Observation and monitoring in the evaluation phase, in fact, produce a further effect: the identification of aspects of functioning and possible margins for improvement. The analysis of the course within the established criteria and objectives makes it possible to identify strengths and weaknesses, thus highlighting a possible direction for growth and improvement of practices. That’s why evaluating is so important and connected to professional growth. The effects and the impacts of the evaluation’s process are more visible thanks to research and studies, but it still needs to be practiced and should be made compulsory and not desired.
6d) A model for a formative evaluation in instructional design
In this section we will present a relevant model in the construction of a formative evaluation’s process in the field of instructional design.
This model was developed by Cynthia Weston, Lynn McAlpine and Tino Bordonaro and presented in the scientific paper “A Model for Understanding Formative Evaluation in Instructional Design”, in Educational Technology Research and Development , 1995, Vol. 43, No. 3, pp. 29-48, 1995.
The model was validated by analyzing 11 instructional design texts and provides a decision-making model for designing effective formative assessment.
The model elaborated here provides a common language for analyzing and understanding the literature on formative assessment, identifying four categories inside two directions and perspectives of evaluation.
Weston, McAlpine and Bordonaro ask “What kind of formative evaluation of instruction is the most effective?” Many researchers have tried to answer this question by elaborating reflections and models around effective and high-impact evaluation strategies but, in fact, we can state that there is no absolute most correct and effective evaluation model; there is the one that, for certain contextual and temporal conditions and criteria, is most appropriate. The three authors, following a lunge and a theoretical reconstruction of evaluation in instructional design, propose and illustrate their model, built on the basis of study, field analysis and research carried out. The model elaborated in fact, provides a guide in the design of evaluation and in the choice of evaluation criteria according to the objectives that each designer sets for it to be efficient. “The model elaborated here provides a common language for analysing and understanding the literature on formative evaluation by identifying four components: who participates, what roles can be taken, what techniques can be used, and in what situations these can occur” (Weston, McAlpine, Bordonaro, p. 29, 1995).
In the model, formative evaluation is like a process conducted by the goals of the instructional project. “To generate an effective strategy we believe one must think about formative evaluation as a design process; thus the essential first step is clarifying the goals of the instruction. Actions or formative evaluation strategies are selected based upon the goals, and limited by any particular constraints. Further, since each formative evaluation is an ill-defined problem, the evaluator does not mix the problem solutions from components, but must look for the appropriate assemblies” (Weston, McAlpine, Bordonaro, p. 47, 1995). That’s why setting general and specific goals is important for the project’s success.
Establishing and clarifying goals is a fundamental and priority step in the instructional design and in the evaluation’s process. Goals can be measured in terms of satisfaction, learning, transfer and organizational impact (Weston, McAlpine, Bordonaro, 1995).
“The model presents formative evaluation in two stages: data collection and revision. The data collection stage is when draft instruction is tried out and feedback is collected in a variety of ways. The revision stage is when the instruction is modified to improve it” (Weston, McAlpine, Bordonaro, p. 32, 1995).
In addition, there are four components that can be used to understand the analysis of the data and the revision; these components are: participants, roles, methods, and situations.
- participants: This component focuses on the intrinsic knowledge that individuals bring to the task; the categories of the knowledge are novice and expert according to a specific knowledge domain, such as subject matter or instructional design. Novices are those who have not experience in a target domain, while experts are those who have developed knowledge in a specific field through both training and experience. So, the distinction among participants is the level and domain knowledge they have.
- roles: This component focuses on the tasks that are given to participants during formative evaluation; there are four types of role: evaluator, learner, critic, and reviser. “The evaluator develops and conducts the formative evalua- tion strategy (…). The most common is evaluator, probably because this is the individual who is responsible for the formative evaluation. The learner is the one who makes meaning of and comes to understand the instruction, the critic verifies and makes judgments about the instruction. Finally, the reviser decides what changes to make and modifies those aspects of the instruction that are identified as problematic” (Weston, McAlpine, Bordonaro, p. 32, 1995).
- methods: “are defined as the procedures, techniques, and instruments used in formative evaluation. The first category, procedures, includes the steps that the evaluator takes in constructing and carrying out the formative evaluation strategy, such as selecting participants and roles, and making decisions about which techniques and instruments to use. Techniques are loosely structured heuristic tools, such as interviews, that may vary in the ways they are used with a number of participants since they are dependent upon the interaction between the evaluator and the participant. Instruments are highly structured algorithmic tools, such as posttests, that are prepared in advance and can be used consistently with any number of participants. Techniques and instruments provide information related to the goals of the formative evaluation and lead to changes that influence the effectiveness, efficiency and appeal of the instruction” (Weston, McAlpine, Bordonaro, p. 32, 1995).
- situations: “refers to the context in which the formative evaluation activity occurs. There are five categories of situations: individual, one-to-one, dyad, small group, and large group. In the individual situation, a single participant acts in isolation. In the other four situations an evaluator works with different numbers of novices and experts” (Weston, McAlpine, Bordonaro, p. 33, 1995).
“The components constitute the basis for possible actions in the design of any formative evaluation with a particular selection from each of the components for both data collection and revision stages representing a formative evaluation strategy” (Weston, McAlpine, Bordonaro, p. 46, 1995).
According to the study and the approach proposed there is no answer to the initial question “What kind of formative evaluation is most effective?” because formative evaluation is a design process and every situation is different from the participants, the roles, the methods and the situations. That means that there isn’t a one single approach that fixes for every evaluation’s project but exists the one that, in that moment and for the categories analyzed, fix better.
In fact, “the model elaborated assists in generating appropriate formative evaluation strategies for specific goals and contexts because it forces consideration of all aspects of formative evaluation and forces one to explicitly address assumptions that previously may have been unexamined. (…) The intent of this model is that each formative evaluation strategy has to be seen as a unique design process consisting of specific goals which drive choices, and constraints which limit the ultimate choice of actions. (…) Using the model as a template to work through the four components at each stage will ensure that the formative evaluation strategy is complete and help to reveal any tacit assumptions that are being made. A desirable evaluation approach then would feature an explicit goal, and clear definition of participants, roles, methods and situations for data collection and revision, all decided in advance and in reference to this goal” (Weston, McAlpine, Bordonaro, p. 46, 1995).
Following these steps, the model proposed shows that it is possible to create a language and a framework that help to guide and create the evaluation’s process; a process that leads to a formative evaluation inside the instructional design’s field.
