Antique Southern Slang

In Tennessee Williams' play, "Spring Storm", the disgraceful behavior of Heavenly Critchfield is being described by one of the older women, and she says something I've never been able to fully define; as she's listing Heavenly's perceived sins, one of them is "going around with Drummers". Is she using that to mean blacks? Is "drummer" an idiom for a partyer? Could it be as literal as fooling around with drummers in bands?
The play is set in fictitious Port Tyler, Miss. in 1937, and I would love to hear from anyone who knows exactly what this means.

Thanks,

Jamie