Ahhh... That good old debate resurfaces...
Well, the short version is that I love brass, it casts amazingly well with smaller bullets. That said, here is the slightly longer version....
I started with Iron molds, Ideal and Lyman. I do love iron molds, and if custom iron molds were a bit cheaper I might still buy some of them. I actually like the fact that they weigh less than the brass molds, while also holding heat well during casting of smaller bullets. Iron seems to be the best compromise between brass and aluminum where it concerns it's advantages vs disadvantages. There is the rust issue. But I'll be honest with you, I haven't had a rust issue on my own molds. The only rusty molds I have owned have come from other people. Once I get them cleaned up they tend to do just fine. But that said, I have several iron molds still that I love, and occassionally use. The biggest problem I have with Iron molds is that they're not really making new ones with 5-6 cavities and with new designs. I use my 4-cav Lyman for staple bullets like the 358311. But unless I just need something specifically in a iron mold, I typically use my brass or aluminum molds that cast a mountain of bullets so much faster.
As for aluminum, it depends... Lee, Accurate, MP, Arsenal, etc, they all use different alloys with different properties. But the big difference between them is the custom mold makers use alloys that are a bit more durable than the alloy used by Lee. Aluminum also casts very well, usually producing quality boolits after one or two poors. To be honest, I also disliked aluminum for a long time due to bad experiences with Lee molds. But now, I guess you could say I have come around to them. I'm actually planning on ordering a new aluminum mold before too long. And I have recently ordered a few more of the Lee 6cav molds. Though, I will also maintain my position that Lee 2-cavity molds are complete trash, and nobody should waste their money buying them.
If I had to choose one material, it would be brass. I truly love casting with a well made mold made of brass. But that said, aluminum is also nice when it comes to the larger molds. You can get a 6-8 cavity mold that isn't going to weigh a ton when you're handling it. Although I do also REALLY like Mihec's 6cavity Brass Molds....