One distinction that might explain why  clothing departments are labeled with a plural (Men, Women, Boys, and Girls) but "Baby" is not is that the Baby department is the only one where the people for whom the  items are  intended  are incapable of shopping for themselves.  "Baby" may  induce a parent to feel special about shopping for the helpless one.
Another reason is that we speak of men's clothing and women's clothing, but we don't speak of babies' clothing. Â We speak of baby clothing. Â Maybe it's just the convention--what sounds odd or familiar--and not an issue of correctness.
Finally, "Men's Department" sounds more inclusive and impersonal than "Man Department" which might suggest a more narrow definition and cause some, for example a 15-year old, to question if they belong there. Â (Do I have to be 18 to be a man?)
To me it seems that it's 'baby clothing' because they, the clothing, are all the same form and can be used almost for all babies, but men's or women's are not.
BTW, can anyone tell me why the computer flags men's and women's?!!!