I vote for the bullets being pushed too hard. Doesn't take much. For lubed .30 cal bullets a general rule of thumb is 140,000rpm is the 'limit'. This does depend a lot on alloy and your bullet quality, but, if you stay below that you will usually have decent groups. My most accurate groups are between 1800 and 2000fps with cast, both lubed and PC.
With lubed bullets I had 1" groups at 2000fps. At 2200fps they were 6". At 2400fps they were mostly off the paper.
Powder coated they can go a bit faster but work up in steps to see where yours work best.