Without doing any research, I would say that "vague exclamation" comes closest to it, that it probably comes from the jazz culture. I certainly heard it used a lot in the 1950s and '60s. Used as a mild oath: "Oh, man, that's the dumbest thing I've ever heard!" rather than "Oh, my god,…", or "Man, that was loud!" rather than "Holy s**t!…"; also used as an expression of dismay: "Look at that:you got a parking ticket." "Oh, Maaaaaaaan!" I suppose you could also call it a genderless vocative: "You shouldn't have done that, man." could be addressed to any person of any gender, but not in a formal situation.
That's my contribution at well past my bedtime.
Edit: Not related to feminist movement.
Reminds me of a classic standup routine by Rob Schneider on using the vocative "dude." As with "man" it can mean different things depending on context and inflection. Check out this performance from YouTube and you'll see/hear exactly what I mean. Runs only 1:46. Poor video quality, but it's the audio that's important. Enjoy.
http://www.youtube.com/watch?v=77v_Q0mhbZU
I agree with Tromboniator that in current slang, "man" is often applied to either gender. I've even used it on my dog, as in "Man, look at that mess you made." I do not believe the gender-neutral use is specific to any subset of American society. I've heard it used too often by too many people. That said, it is most definitely slang, or at best, informal English.
Seems to me there's a difference between the exclamation of dismay ("oh, maaan!") and the address ("how're you doing, man?"). The former is not vocative, the latter is. I thought at first that tromboniator wasn't making that distinction adequately, but now that I reread his post, maybe he didn't mean it that way.
No comment on whether the vocative form should ever be used to females. I never have done it that way, but I'm willing to stretch a point and say "you guys" when addressing a mixed crowd or even an all-female one, and I can't say why one should be allowed and the other not. If y'all say you've heard "man" used to women, I don't doubt it.
Bob Bridges said
I thought at first that tromboniator wasn't making that distinction adequately
That's my "forum-condensed" writing style. I was acknowledging the distinction, but not clearly. In point of fact, I think that while the distinction is real, it isn't clear exactly where the boundary is. For example, "Man, you really shouldn't do that." can be, to my ear, a kind of mixture of "Sir (or Ma'am),…" and "Holy cow,…".
I wasn't arguing for (nor against) using man in addressing women, merely that I have known it to be used, generally by men, rarely women, in whom it is habitual, and probably unconscious and unanalyzed, and applied universally. I know what I would do, but I'm loath to dictate what should be done!
Peter