Black has carbon black and the other colors are using titanium dioxides.
A quick check shows carbon black to be 2. - 2.9 on the Mohs scale and titanium dioxides to be 3.82 to 5.13 Mohs. <--- subject to change by people who know or find more than I did.