Pontefract, who ended the game with nine men after playing the majority with 10, won the shoot-out 5-3 having converted all of their penalties.
The away side took the lead from the spot in the 14th minute before going a man down just a minute later.
The Hornets levelled in the 24th minute thanks to Cedwyn Scott before going a goal down again 20 minutes into the second half.
Substitute Graeme Armstrong got the equaliser from the spot in the 88th minute, two minutes after the Colls went down to nine men.
It was set up to be a tough FA Cup tie against a very different Pontefract side to the one that knocked Hebburn out at the same stage last season.
But Hebburn went into the game full of confidence on the back of three victories while they came up against a step four side who were playing their first competitive game of the season.
Tom Potter looked lively early on and saw his shot from an acute angle tipped onto the bar following a good run.
Mark Foden was called into action just a few minutes later when he had to race Gavin Rothery to a weak backpass.
It was a warning to Hebburn and in the 12th minute they would go on to concede a penalty on the back of three consecutive Pontefract corners when Luke Salmon was deemed to have adjudged to have brought down Damian Reeves.
Reeves stepped up to take and put it calmly down the middle.
Former Leeds United academy player, Gavin Rother was sent off just minutes later for an uncontrolled challenge on Lewis Suddick.
Scott had Hebburn level just after the halfway point of the first half when he was slid through by Robbie Spence on the end of a good move and turned the defender to make space of a nice finish.
Hebburn had a couple of good chances from crosses, first Suddick's cross narrowly evaded the bodies in the centre before Salmon's cross found Taylor at the far post who could not add the required connection.
There was a fantastic chance for Scott at the end of the half when the ball fell to him a few yards out after a bit of a scramble but his ambitious backheel didn't find the target.
The Hornets continued on the front foot at the start of the second half but were rocked when Pontefract put themselves back in the lead.
Hebburn lost the ball in their defensive third and Fabian Bailey crossed for Dan Keane who found the left corner with his header.
Armstrong hit just wide from outside the box not long after coming on before Tom Potter curled a great shot against the post in the 78th minute.
The Colls were down to nine men in the 86th minute when Dom Claxton pulled Scott who was racing onto the end of a lobbed pass.
And two minutes later Hebburn were awarded a penalty when Potter advanced into the box and scythed down by Sam Tingle as he tried to hit the byline.
Armstrong stepped up and fired into the bottom left corner.
Hebburn smelled blood and were on top but the Pontefract defence held firm and the away side even managed to carve out a good chance of their own when Keane fired wide from inside the box.
So it was the lottery of penalties again, but the Hornets couldn't repeat last week's heroics, the away side converting all their penalties to win 5-3.