A Canadian company linked to coding errors which led to thousands of Leaving Cert students receiving incorrect grades last year has defended the quality of its work.
About 14,000 Leaving Cert grades were incorrectly issued last year due to errors in the standardisation process used to adjust teachers’ estimates grades for consistency and fairness.
Students who were wrongly downgraded were subsequently upgraded, while those who received higher grades than they should have were allowed to retain their grades.
Fernando Cartwright, chief executive of Polymetrika Inc, said independent reviews into the calculated grades process characterised its work as "remarkable".
"Ireland is noteworthy in that not only did we complete our work on calculated grades in less time than neighbouring jurisdictions, but ours was the only process that successfully achieved its goals," said Mr Cartwright.
“In any scientific process, a correct result is produced through trial and error. It is extremely unfortunate that the CAO timelines required a release of the results prior to the completion of the review of the work, but it is better to seek out and remedy errors, even when it is inconvenient, than to bury them for expediency.”
Mr Cartwright said while he was bothered by “misleading” characterisations last autumn about its work, he was more concerned with ensuring the accuracy of results and fair outcomes for students.
“It was incredibly frustrating to bear silent witness to ill-informed judgments about the process and myself at the time, but I am proud of the quality of the work and my integrity in supporting it,” he said.
Minister for Education Norma Foley said on September 30th last that Polymetrika had first identified an error which affected the way candidates' Junior Cycle results were included in the standardisation process. A total of four errors subsequently found in the standardisation process.
Mr Cartwright said any depiction placing sole responsibility for success or failure of any aspect of the Leaving Cert 2020 project on Polymetrika would be inaccurate.
‘Political context’
“I was responsible for advising stakeholders on methodological options and designing and implementing their corresponding algorithms, but transparency and the political context required that I have no responsibility for any actual decisions regarding the input data, choice of specific methods, timing of releases, and use of the results,” he said.
“While I certainly would have appreciated a more active defence of Polymetrika and the Leaving Cert 2020 work, I can understand that, in the delicate political context of a new minority government, an even newer Minister, the various Covid emergencies, and almost-weekly Brexit crises, there would not be much political appetite to stick one’s neck out for any reason.
“To Ms Foley’s credit, she maintained support for the process and the work, which resulted in a fairer transition and allocation of students than in the neighbouring jurisdictions, where the calculated grades processes were abandoned almost immediately after the initial release of results.
“The successful defence of the process in the face of a formal legal challenge is also a good indicator of the Government’s faith in the work and, ultimately, the legitimacy of the process.”
He said headlines and opinion articles by pundits and politicians were “incendiary and speculative” and a visit to his home and registered address of Polymetrika Inc by a reporter seeking comment on the controversy was “intrusive”.
Opposition politicians at the time questioned Polymetrika’s track record in the area and the nature of the contract signed with the firm.
It was reported at the time that Polymetrika had been awarded a contract without a tendering process on the basis that there was “insufficient time” to do so. The company has since been paid about €200,000 for its work.
Mr Cartwright said he was the only person from Polymetrika engaged in the calculated grades process.
When asked if the company had the capacity to take on the work, he said it did, as reflected by the fact that Polymetrika completed the work.
“In my experience with projects involving iterative development and implementation of statistical methodologies, large teams tend to take longer, or produce more errors in a fixed time, than smaller teams, simply due to issues with integration and communication,” he said.
When asked if Polymetrika had been involved previously in devising a system of calculated grades for any other jurisdiction, he said the system used for last year’s Leaving Cert was not common practice in psychometrics or educational testing.
“However, the practice of computing or predicting a test score from partial or proxy data and then using that estimated test score to support decision-making is routine,” he said.
"Essentially, all modern psychometric techniques are variations of this general approach. I have co-authored a book for the World Bank describing common applications of this method in the context of national assessments in education, and I have assisted over two dozen educational systems, private assessment providers, and various large-scale surveys around the world in implementing related methods."
