Are you using any kind of sanding block ?
I found that a Pink Rubber Ereaser works very good as a sanding block.
It is firm enough , small enough , and can get into areas you need to apply pressure but still have control of the sanding direction.
Sandpaper just held in your hand will just follow the contour of the flat surface you are trying to restore