Graham Bayne had given the home side the lead after 18 minutes and it looked like they would record their third cup win over the Glasgow side in seven seasons.
But with a minute to go Pressley headed in from close range to stun Caley whose misery was complete in injury-time when Miller fired home from the edge of the box to complete a remarkable comeback.
Gordon Strachan's side go into the hat for the semi-final draw with Hibernian, Dunfermline and the winner of the Motherwell versus St Johnstone game but they will wonder just how they managed to pull this game out of the bag.
Even with the Hoops' history of late fightbacks, this was one match that looked out of their reach as the game entered its final stages.
Despite narrowly losing 2-1 to Celtic in the SPL just a month ago, the Highlanders approached thus match with history on their side.
Caley had beaten the Glasgow side in their only two Scottish Cup meetings to date, 3-1 at Parkhead in 2000 and 1-0 at home three years later.
The visitors began in a slightly unusual 4-3-3 formation, Shunsuke Nakamura flanking Miller on the right with Jiri Jarosik on the left and it almost paid immediate dividends.
Jarosik picked up a Neil Lennon pass on the left and sent the former Wolves striker clear through the middle with a well-weighted pass.
Miller moved in on goal at speed but hesitated getting his shot in and when he finally unleashed 12 yards out, Caley keeper Michael Fraser blocked.
Celtic were made to pay for that miss in the 18th minute when the Highlanders scored in a classic counter-attack.
Right-back Ross Tokely fed the unmarked Ian Black who turned and sent Barry Wilson away down the left.
Wilson cleverly turned Steven Pressley inside out inside the Celtic penalty area before slamming a cross-shot across the Hoops' six-yards box which Bayne gleefully tucked away.
The SPL champions responded positively, Nakamura's curling shot from 25 yards pushed around the post by Fraser for a corner but Caley, again, defended stoutly.
With the Parkhead side deploying a narrow midfield, at times Thistle were finding joy down the flanks.
And in the 28th minute Bayne had the ball in the net again after Celtic keeper Artur Boruc had spilled a long-range shot from Black but, to the consternation of the home fans, the Caley striker was ruled offside.
Richie Hart replaced Roy McBain at the start of the second half and it was the home side who had the first effort after three minutes, John Rankin working a neat one-two with Bayne before firing well wide of the target from distance.
However, in the 54th minute, after Miller had cleverly set up Nakamura with a flick close to the Caley goal, the Japan international thundered an angled shot with his left foot off the cross bar from 10 yards out, the ball rebounding to the far side of the box and safety.
Most of the traffic flowed towards the Caley goal but often stopping short ofFraser, who watched his team mates defend with increasing desperation.
In the 65th minute, moments after Hart had tested Boruc with a decent left-footed drive from distance, Hoops substitute Aiden McGeady replaced Jarosik as manager Gordon Strachan looked to rescue the cup tie.
But the equalising goal looked no nearer arriving and with a little over 15 minutes to go 17-year-old Irishman Cillian Sheridan was thrown on for his Celtic debut, coming on for Mark Wilson, shortly before Thomas Gravesen replaced Evander Sno.
However, despite the changes, Celtic's attacks were losing conviction as the match entered its final stages.
But in the 89th minute the visitors grabbed a dramatic equaliser.
Nakamura's corner from the left was met at the back post by Stephen McManus whose header across the goal was tucked away by Pressley.
It was a body blow to the Highlanders who must have thought they had booked their semi-final place.
But more misery was to follow. A minute into injury-time Miller worked a one-two with Sheridan and slotted away from the edge of the box and Caley's nightmare was complete.
