Irish theatre was once brilliant or terrible. Now the extremes are rarer. It is a tentative, cautious time, writes Fintan O'Toole.

In the theatre, every year is the best of times and the worst of times. In the fabric of drama, failure is always interwoven with success. Yet the relationship between the two does undergo subtle changes. For much of the 1980s and 1990s, the gulf between Irish theatre's triumphs and its disasters was an unfathomable abyss. When it was good it was very, very good, and when it was bad it was rotten.

Now there has been a gradual but discernible shift. Looking back on 2003, both the accomplishments and the disappointments seem more modest than before. There are few fiascos and few sensations. The mood is tentative, cautious, questioning.

This was a year, for example, when three of the great veterans of Irish playwriting - Tom Murphy, Brian Friel and Thomas Kilroy - all had new plays to offer. But none was all that interested in writing the great Irish play. All three, strikingly, gave us plays that were largely concerned with questions of life's relationship to art.

Murphy's The Drunkard, a virtuoso version for B*spoke of the 19th-century temperance melodrama attributed to W. H. Smith and A Gentleman, was a delightful homage to a much-abused form that was also a sly exploration of his own artistic autobiography. Friel's Performances, at the Gate, and Kilroy's The Shape Of Metal were both, in remarkably similar ways, metaphorical explorations of the writers' own situations.

Both plays centred on aged artists. Both eschewed overt drama in favour of a downbeat, rather abstract contemplation of the relationship between life and art. Both were haunted by notions of failure. Both took their language from other art forms: music in Friel's case, sculpture in Kilroy's.

Intriguing as all of this is, it also points to the fact that the dominant older generation of Irish dramatists is not, for the time being at least, especially engaged with contemporary Irish society.

There was, admittedly, a certain pleasure in the fact that none of three plays had anything specifically Irish about it. The Drunkard was set in a universal 19th-century imaginative space of bucolic countryside and darkly satanic city. Performances unfolded in an imagined Czechoslovakia and deliberately disrupted the sense of when the action was happening. The Shape Of Metal was notionally set in Ireland, but it could have been anywhere. Liberating as this might be for the playwrights, however, it raised the question of who's left at home, minding the house.

The answer from the playwrights seems to be that the house burned down a long time ago. The year's two most intriguing new Irish plays - Mannix Flynn's James X at Project and Mark O'Rowe's Crestfall at the Gate - were marked by a profound bleakness. O'Rowe's hypnotic, hyper-violent tale of everyday mass murder in an unnamed suburb of an unnamed Irish city was, as revealed in Garry Hynes's superb production, as much about the dislocation of narratives in a media-saturated world as it was an attempt at social realism. It questioned, indeed, whether realism is still a useful concept in a culture where people act as if they are in a movie.

Flynn's extraordinary show, which told his own story of institutional abuse, raised in its form the question of whether theatre as we have understood it is in fact adequate to the kind of stories that need to be told now. The sheer power of the author and actor confronting his past with the help of the documents that constitute the official record of his existence would be hard to match in any more elaborate and supposedly "artistic" form.

If art filters and transforms experience, maybe much of the raw material that Irish society currently provides for drama demands something else: a directness and simplicity that go against the grain of our usual evasive elaborations.

A few other strands reinforce this sense that basic assumptions are up for grabs. It is notable that the traditional Irish repertoire was mostly conspicuous by its absence. I can't remember a year in which there was so little Synge, O'Casey and Yeats. The solid but unexciting Abbey production of Friel's Aristocrats suggested that more needs to be done than simply returning to the repertoire to fill in the gaps. There was in fact just one outstanding revival from the mainstream repertoire - Hynes's brilliant Druid production of Keane's Sharon's Grave - and it reminded us what a strange stream it really is.

The other straw in the wind is the relative success of work adapted from prose texts. Joe O'Byrne led the way with powerful stage versions of Roddy Doyle's The Woman Who Walked Into Doors (in which Hilda Fay was breathtaking) and, in Kilkenny, of Oscar Wilde's De Profundis.

Again, the energy generated by what might seem unpromising projects suggests Irish theatre is casting around for new forms, testing possibilities rather than blazing ahead with full confidence in its own relationship to the fragmented society in which it exists.

That the problem is not necessarily theatre itself was shown by some thrilling visitors. Lia Williams's performance in Tennessee Williams's Eccentricities Of A Nightingale was an astonishing reminder that actors are creative artists too. The Chicago company Steppenwolf showed us in Bruce Norris's Purple Heart, at Galway Arts Festival, that there is life still in the old-fashioned notion of an ensemble company working with a naturalistic text, albeit one with a streak of strangeness.

And Yves Jacques's stunning performance of Robert Lepage's The Far Side Of The Moon - the highlight of the year - in which the universal and the particular were perfectly fused, showed that the exploration of new forms can eventually arrive back at the point where beauty becomes possible again.