The best drift punches are High Speed Drill Blanks. I make odd ball punches out of those in my shop.
When I was assembling my Hand Presses I had to make a Roll Pin Punch and did so from a 3/16 Drill blank and turned the pilot and nub on the lathe with a carbide insert.
Then I found out that I needed a punch to knock out pins I had screwed up. So I got a #31 Drill blank .120 in dia and that's my pin removal punch.
The only thing that is a problem with using drill blanks is that they are harder than a hammers face. So you end up with some dings in your hammer.
Randy