I thought so too, but my experience has been different.
I normally do my casting (clean lead, no smelting) inside a small enclosed building, with windows open and a fan directly blowing the fumes away from me. I never had a problem until one time I didn't use the fan. That night I had the worse migraine headache that lasted for many many hours.
So, my experience tells me that there are some fumes being released at normal casting temperatures.