"I didn't see it, my mum didn't see it, and my mum's the biggest critic of the lot." The Great British Bake Off judge Paul Hollywood seems surprised by suggestions that he and Ruby Tandoh, one of the finalists, flirted on screen during filming of the BBC Two series, the final of which was watched by a record 9.1 millions viewers on Tuesday night. "I don't normally watch the programme, but I watched the quarter-finals with four of my mates in a pub, to see what everyone was talking about."

Tandoh, a 21-year-old student of philosophy, who was doing her finals at the same time as filming the programme, decried the accusations of flirting and favouritism as “boring and predictable”.

She also had a feisty response for the chef Raymond Blanc, whose Twitter comment on the morning of the final about "female tears" – Tandoh was often filmed in an emotional state as the competition progressed – and "a winner so thin who makes me doubt her love for great cooking, baking", led fans of the programme to believe that the result was a foregone conclusion in favour of the former model.

Hollywood and his co-judge, Mary Berry, said on screen that they had "someone in mind" for the title, going into the final task, in which the three finalists had to create a "show-stopper" wedding cake in six hours. Three less than perfect cakes later, did they change their minds?

"We did have to adjust, slightly," Hollywood says. So did they pick a new winner at the last minute? "Maybe", he says, suggesting that Tandoh's not-so-show-stopping wedding cake may have handed the honours to the eventual winner, children's clothes designer Frances Quinn. "We judge it on the way it looks and the way it tastes, and Frances won on both. She just got better and better," Hollywood says.

“You could win star baker every week and then fail in the final and you will lose, because each one stands on its own. You remember who’s won and who the strong ones are. If it’s close, then we look back to see. There is a benefit to winning star baker, and Ruby and Kimberley would have been fighting it out. The same thing happened last year with John and Brendan. James and Brendan were seeded higher, but John came through and won it.”

Hollywood watched the televised final in London with the crew and co-presenters, including Berry. “Chris Evans came and joined us, too, and we had some drinks and all sat down and watched it together.” The gentile Berry seems an unlikely drinking partner for the baker, who describes himself, rather harshly, as “just a bit of rough from Liverpool”. But their double act is one of the linchpins of the show, which moves to BBC One next year in recognition of its move to mainstream rather than niche broadcasting. “She’s just like my mum; we are a good partnership,” Hollywood says of his co-judge.

They must have had some cosy chats over the four seasons of GBBO, sequestered in their little judges' tent while the drama simmered in the big marquee as the contestants grappled with the technical challenges? "Not really. It's recorded 'as live', so it's quite interactive, and there's always something going on. We film our 'hero' thing, then go back to the green room and put our feet up for about half an hour. Then we go back to the tent and judge theirs."

Hollywood was in Dublin on Thursday to promote his latest book, Paul Hollywood's Pies & Puds, which is dedicated to his son Josh, who is a keen baker at the age of 12. But despite his success, Hollywood isn't keen for his son to follow in his father's and grandfather's footsteps. "If he wants to go into food he can, but I hope he doesn't. It's hard work, and I was just lucky. But James Martin is his godfather, so he's got me and James, and his mum's a great cook, so he's probably knackered, isn't he?"

Hollywood and Martin, the food TV presenter, are great friends who share a love of fast cars. “But James is a Ferrari man, and I like British cars; Aston Martin, that’s my passion.” It’s good to learn that he has a racy side to go with the film-star looks, because life chez Hollywood sounds a little pedestrian. “I’m pretty simple. I like having a bath, putting a dressing gown on and watching movies.” He used to do martial arts but finds that his schedule no longer allows time for it, so he relies on eating “small portions” to stay trim and telly fit.

It has been a busy year for the baker, who says he hasn't cooked at home in a year, and has been on the road constantly, filming, and doing demonstrations and voiceovers, as well as writing. Next year he's going on the road with a live show, at the same time as filming the next series of GBBO. "I won't have a day off for six weeks."

But he's taking a month off at Christmas. "I think I may be going skiing in the Alps with Alex and Josh . . . It's what I'd like to do, but it depends, whether I get the chance," he says. Alex is the 47-year-old's wife of 15 years, from who he was reported to have split earlier this year amid suggestions that he had become romantically involved with Marcela Valladolid, his co-star in the US version of Bake Off.

Hollywood attracts plenty of female attention, and it’s definitely something he has capitalised on. “Food porn,” I say, pointing to one of many images of him in his new book, smouldering at the camera with those piercing blue eyes. “Well, if Nigella can do it, why can’t I?” he says with a laugh that suggests he doesn’t take himself too seriously.

Just don't challenge him on the eyes, though. When I check them out in the dim light of the hotel lounge where we meet, and suggest they're maybe not quite as startlingly blue as on the book cover, he disagrees. "It's the light. If you stick the light on, you'll see. There were 20 women lined up today in Kitchen Complements" – where he did a book signing – "and they all went, 'Yep, his eyes are that blue.' "

Paul Hollywood's Pies & Puds is published by Bloomsbury, £20