As soon as the bullet starts accelerating there is an opposite force back through the rigid firearm frame. That frame is held in your hand that is not rigid. Therefore according to your grip and wrist strength the pistol is moving while the bullet is accelerating. So POI is different from POA if the sights are aligned with the bore. Sights are used to align the POA to the POI.
Your calculations would be true if you could fire the pistol without the sights moving. Also the hammer fall may contribute to firearm movement.