OPINION:Optimism about greater focus on aural and oral testing has waned as old exam techniques remain.
REPORTS ON dumbing down in Irish education and the recent disappointing results from the Programme for International Student Assessment have brought academic testing and assessment into the public domain. Phrases such as “grade inflation” are now common currency and the time has come to examine the reliability of examinations at all levels of our educational system. The Leaving Certificate Irish exam would seem an appropriate place to start the debate.
The starting point for any review is to establish validity: does the exam test the skills it is supposed to test? We can only judge the exam in light of the aims laid out in the Leaving Certificate syllabus. Let us take, for example, the higher level components of the Irish aural and oral tests. Clear aims are set out for the aural component: candidates are expected to understand normal conversations delivered at normal speeds, television and radio reports, dramas and so on. Indeed, the test tends to cover a pleasingly diverse range of materials, although some passages are recorded at an unnaturally sluggish pace. The questions are more problematical.
First, comprehension tasks are pitched at an elementary level. These overwhelmingly involve processing highly literal “who”, “what” and “how much” questions. For example, the candidate hears a speaker called Sinéad say she intends to do a degree in French and law at UL and is later asked what degree Sinéad will do. There is an absence of what language testers refer to as “distractors”, in this case, other courses that Sinéad might have done and which the candidate would have to understand, process and discount before giving the correct answer.
A further disturbing feature is that candidates are often required to state one fact relating to a piece they’ve just heard, a simple task of information retrieval which is a world away from the inferencing and contextualising of real communication. An interesting contrast is to be found in the aural section of the Leaving Certificate German paper, in which candidates are asked to give their impression of the relationship of certain speakers to each other, or to supply the word that best sums up their emotional state. This is cognitive processing far beyond the word recognition level of the Irish exam.
Worst of all are the “general knowledge” questions that break a cardinal rule of testing – that answers should only be reached through an understanding of the text. One recent example: “State one piece of information about Mícheál Ó Muircheartaigh”. Most Irish people already know that he is an authority on Gaelic games, a Kerryman, a broadcaster and so on. The desire to present students with contemporary materials about real-life situations serves no purpose in this instance, as this task tells us nothing about listening comprehension ability.
According to the State Examination Commission’s website, the weighting for the current listening comprehension is divided between 90 per cent for comprehension and 10 per cent for ability in Irish. It is not clear what is meant by “ability in Irish” and so its application would seem further to undermine the validity of the exam. The marking scheme contains one example in which the fully correct answer to a comprehension question is “Gael-Linn”, the well-known music publisher. For this response the candidate will receive two marks. If, however, the candidate writes “Gaol-Linn” he or she will receive no marks. Again the German exam, with its acceptance of “phonetic spellings”, is a better reflection of best practice.
Much publicity and no little optimism surrounded the decision to increase the percentage mark for the oral Irish examination from 25 per cent to 40 per cent with effect from 2012. That optimism has waned somewhat given that half an academic year has passed without the Department of Education giving any direction to schools as to how to go about teaching for this change. In the absence of such direction we can only assume that the aims of the existing syllabus remain those of the assessment. The syllabus states that candidates are expected to play a full interactive and confident role in personal and formal conversation and be able to read aloud with good pronunciation. Again, we are entitled to ask if the assessment matches the aims; again, the answer is no. The tasks and marking scheme to be used for the new oral exam remain the subject of speculation, much of which suggests that almost 15 per cent of the marks will be awarded for reading aloud 10 or 12 lines of poetry. The idea of recitation is absolutely at variance with the notion of interactive communication and other buzzwords used in the syllabus. “Reading aloud as part of an oral test makes the test invalid to the point of absurdity,” writes the testing expert YP Lee. It would be bad enough if the text were given to the candidate on the day of the exam. How would Lee react to the knowledge that students have two years to learn their few lines off by heart?
It seems likely that candidates will also be obliged to describe a series of pictures during the oral exam – a perfectly valid testing activity rendered meaningless by the fact that they will be distributed to schools well in advance of the exam. If speculation about the marking scheme is correct, as much as 48 per cent will be allocated to tasks that can be learned off prior to the examination. The commission has already noted the tendency of candidates to lapse into rehearsed rigmaroles. Providing so much of the testing materials well in advance of the exam is nothing more than an open invitation to learn by rote.
All of the above and many other flaws can be easily put right. But that will not happen without a systematic review of syllabuses and examinations. We must aim for a situation where the outcome of any given examination can be interpreted as a genuine reflection of the candidate’s ability. We must make use of best international practice to ensure that our assessments are valid and reliable. At the moment we are failing very badly indeed.
Siuán Ní Mhaonaigh is testing consultant to the Language Centre at NUI Maynooth