How long is your barrel?
It probably has to do with follow through, the time it takes from when the trigger is released to when the bullet exits the barrel.
Perhaps you were anticipating the shot and the rifle moved before the projectile exited the barrel.
"Do not anticipate the shot and move before and during the time the bullet is still in the barrel." --->>>
https://www.ssusa.org/articles/2018/...ollow-through/
There's detailed explanation with a few tips here: --->>>
https://www.outdoorlife.com/blogs/gu...when-shooting/
Basically keep your finger on the trigger and your head on the stock for 2 full seconds after the shot.
I suppose that a sling could always help, that's what a lot of match shooters use with non-muzzle loading guns.