This is kind of what I was thinking... it's not like the other guns are more offensive because I'm shooting something small.
Besides, who goes all the way to the range with JUST rimfires, unless that's all they have? We alternated between quiet little 22s, pistols, medium-power carbines like the SKS, and the hilarious muzzle blast (and accompanying fireball) of a m91/44 Mosin Nagant. Sure, I've been pelted with brass cases, but I'm sure not gonna tell the other person that they need to move or can't use the range at certain times now because that's inconvenient for me... and they gave me the same courtesy. Some guns chuck brass into the next county, but that's just part of the game - you could say it's part of the appeal, the different functionality and diversity. It would have been appallingly boring to go the range and see all the same kind of gun all the time, a range isn't complete without an ancient single-shot 22LR, a big revolver, a tactical magdump machine, a surplus rifle equally useful for clubbing and shooting, and a scoped 300 Ultra-Whizz-Mag on the same line.
Of course, that's just my opinion. YMMV.