I saw an interesting Youtube video yesterday using .22lr 40 gr standard velocity ammo.
From a 10/22 the bullet mushroomed perfectly but only penitrated some 12".
From a Ruger semi auto with 3.5" barrel 3 shots all penitrated 18" but did not mushroom.
They did leave very nice large permanent cavity's which were all 3-5 inches deeper than fired from the rifle.
Speed is not always an aid for penetration!
Now those little .32acp's are moving a third slower, but bullets will have close to or over twice the mass.
I would not count them out.
I don't often carry but if I did it would be a .32 of some flavor. Simply because I know I can be much more accurate with it.
And the muzzle blast is not going to mess with my head near as much as even a 9mm. So IMO, they have a place. YMMV. And that is fine.