i agree, I definitely don't reload the rifle from the powder flask but into the measure from the flask. You don't know how long it can smolder down the barrel. I was watching an episode with RLee Ermey a few months ago and he had a guy shooting either a flint or cap and the guy was quickly loading and shooting fast as he could as part of a rate of fire thing and had one of the charges flash as he dropped it in the barrel. His face was away and the barrel tilted away so didn't get hurt.
Personally, I swab between shots because I'm target shooting but if I was hunting and trying to get off another shot quickly, the odds are obviously better for that to happen.
Lymans book does say to leave the hammer down on fired cap to deprive anything still smoldering in the barrel of air. Just common sense but some like to tempt fate.