- Pedagogy
- AcademicSkills Assessment Exam LearningObjectives
- Content
- ChemistryEducation STEM ScienceEducation
- Context/Topic
- PedagogicalIssues QualityAssurance UnitedKingdom
JL Kiappes
Department of Chemistry, University College London, United Kingdom
Challenge and goal
In course design, many instructors focus on subject-specific knowledge and skills (e.g. chemistry concepts). In both assessment and learning activities, further skills (e.g. providing clear and persuasive explanations) can be required, but these are not explicitly highlighted as learning objectives and can be underemphasised in feedback. In this user case, such skills are called 'covert' assessment criteria or learning outcomes.
Without these 'covert' skills being purposefully developed, there is a breakdown in communication between students and teachers. Assessment outcomes can then cause frustration for both groups. Students feel their answers were correct and unfairly penalised, while teachers misinterpret that students have misunderstood subject content. Because of the misinterpretation, attempted improvements can miss the covert criteria and misplace resources into changing other aspects that are already well done.
The goal is to identify these 'covert' skills, decide how to incorporate them (or not), and support colleagues in this revision of modules.
Topic of the user case
Identifying criteria that are evaluated by assessment but are not explicitly highlighted in intended learning outcomes.
Local context (specific)
Chemistry Department modules at University College London (UCL) typically have an on-site, timed, written exam as the major form of summative assessment. In several cases, a single exam of this type forms the entire basis of the grade for the module.
Given the pedagogical training required by UCL (see below), staff are generally proactive about constructive alignment, and there is a strong correlation between ILOs and the subject content on exams. At the same time, some assessment questions/tasks require broader skills (e.g. constructing a logical written argument) that are not highlighted in the ILOs or learning activities. Often these skills appear in learning activities, but feedback focuses on the chemistry concepts rather than the form of the answer. Particularly as there is a significant proportion of international students who are studying in a non-native language, these skills can require further support.
Local educational scope (science, technology, engineering, math, other?)
While potentially relevant to all disciplines, the CPD's initial scope will be for the Chemistry Deparment, across all sections (Inorganic, Organic Chemistry and Chemical Biology, Physical Chemistry, Theoretical and Computational Chemistry).
Pre-knowledge / Background of the participating local teaching staff
All new academic staff are required to complete a course about pedagogy/didactics which is run by the Arena Centre. The Arena Centre aims to improve the quality of teaching across UCL. Building upon this introductory course and their own experiences, members of staff then produce a portfolio demonstrating Descriptor 2 (Fellow of the Higher Education Academy) as defined by AdvanceHE. Because these are organized at the institutional and national levels, they have no content-specific materials.
The Arena Centre also offers CPD courses centered on a variety of topics including technology (e.g. Moodle, social annotation), education styles (e.g. problem-based learning), or supporting specific groups (overseas students, neurodivergent students, etc).
In the Chemistry Department, there are weekly education-focused meetings (Connected Learning Sessions), where academic staff can share experiences, questions, and examples of practice. These are sometimes open discussions, and other times have a specific theme or include a presentation.
Local CPD goals
- Identify skills required by assessment beyond those in ILOs.
- Support staff in updating modules (ILOs, learning activities, and assessment) to address these covert criteria by:
- Adding them as ILOs and providing appropriate feedback about them in learning activities.
- Making clear that they are expected pre-requisites and providing appropriate feedback about them in learning activities.
- Redesigning assessment so the criteria is no longer required.
Needs defined in STEM-CPD Roadmap
Competences
Attitudes
Activities
CPD activities at the local university
Stage 1 - Identify 'covert' assessment criteria
Initially work will be conducted with members of the teaching team and others who actively participate in the Connected Learning sessions.
An initial synchronous meeting will be held to introduce the problem and brainstorm anticipated common 'covert assessment criteria.' Participants will be allocated to pairs or teams based on content area.
Asynchronously, they will reflect individually on relevant exams from the last two years, both looking for the brainstormed criteria and identifying any others. Each team will then discuss their findings (either synchronously or by email) to prepare an updated list and examples.
A second synchronous meeting of all participants will take place to share findings from all teams and construct a list of identified covert criteria. This will be developed into a document and slide deck to allow wider dissemination within the Department.
Stage 2 - Categorising and prioritising 'covert' criteria
Participants from stage one will act as ambassadors to their sections within the Department to present the initial findings and canvas broader teacher experiences by facilitating similar reflection on assessment from other modules.
For the covert criteria, it will be discussed whether these skills are ones that are (a) expected to already be possessed by students; (b) worth including as explicit ILOs (and if so which modules are most appropriate); (c) niche skills that should be removed by modification of assessment.
Stage 3 - Supporting change
Once covert criteria have been prioritise to remain in the degree programme, resources will be collected appropriate to the specific criteria (see example in teaching resources below). These will look at learning activities that emphasize these skills, and provide information to colleagues who might feel less experienced or comfortable teaching these types of skills. These will be announced an All-Staff meeting and presented in full detail at a Connected Learning meeting.
Teaching and learning materials
Past exam papers
Once 'covert' criteria are identified, specific resources (papers and videos) will be selected. For example, 'Write-to-learn' articles by Shultz et al. for the development of clear explanations.
Sustainable implementation
The document produced in Stage 1 (and further developed by subsequent stages) can be provided to new members of academic staff by their mentors to help guide assessment design. It can also be incoporated into the exam scrutiny process to quality assess exams each year.
The identified 'covert' skills and resources for supporting their incorporation into modules will be collected into a Moodle page.
Expected impact of the CPD User Case
- Improve the constructive alignment of modules, in particular identifying and supporting skills as well as subject-based content.
- STEM colleagues would feel more confident in teaching broader, transferable skills (e.g. constructing an argument).
- Students will feel more prepared for assessment and that outcomes are more fair.
Plans for eventual continuation of the CPD within the same topic
- After identifying common covert criteria, the documents and Moodle page can be shared more widely within UCL.
- Building on the experience of the identification of these criteria, there would also be a possibility to eventually develop this into a μMOOC to support similar reflections in other departments and institutions.
Constructive alignment, Deep learning (type P1-1, P1-3c)
This CPD scenario describes a User case in which lecturers develop their competence in sound course design and how to facilitate student’s deep learning and development of higher cognitive skills and develop attitudes in supporting student development and enabling students’ well-being in a learning process and inclusivity and reflecting on own teaching practice and knowledge sharing.
The approximate duration of a User case that follows this scenario is several months.
In this CPD scenario the participants professionalize in a close connection to their own teaching practice (at their workplace) and meet in person on location with the training staff and with other participants.
Learning environment
Time
Several months
Creative Commons 3.0 BY SA applies to all content on Starfish.
Starfish-education support for the publishing on Starfish-education.eu does not constitute an endorsement of the contents, which reflect the views only of the authors and Starfish-education cannot be held responsible for any use which may be made of the information contained therein. Starfish-education cannot be held responsible for the content published by authors that is not conform with Creative Commons 3.0 BY SA.