Bradford took the lead on 49 minutes with a goal from Oliver Forsyth before an equaliser for the Lilywhites on 62 minutes by Samuel Ade Osobe seven minutes after coming on as a sub.
In the penalty shootout Bradford keeper Chris Cheetham saved the first penalty from Scott Leather with Bradford captain Alex Flett scoring to give the Banthams a 1-0 lead.
PNE levelled the score at 1-1 when Will Hayhurst scored and PNE keeper Dominic Comrie saved Adam Baker's spot kick.
The score was then 2-2 as both PNE's Michael McLellan and Bradford's Nathan Lawless scored.
The Lilywhites took a 3-2 lead when Daniel Devine scored and Comrie saved his second penalty as Bradford's Joe Mitchell missed.
It was left to PNE goalscorer Osobe to score Preston's fourth penalty and win the tie 4-2.
The Fifth Round will see Preston play at either Blackburn or Leicester.