I break it down like this...firstly when people throw around the word unmanly with regard to sport, I take it to be somewhat of an insult to the sport and moreso an underhanded remark to the "man" who plays the sport in question....which is messed up to begin with.....
If baseball were to be played by women they would have to decrease the size of the outfield, infield, and decrease the distance from the mound to the hitter, decrease the distance between bases etc because girls just aren't as strong and athletic as guys and the game would just not be the same ....(I was told this by a pro baseball player...I know nothing about baseball)...so in that regard, you can argue that baseball is more manly that football (soccer)....
Then again, you could argue that for women's soccer the goal & fields should be a bit smaller because the women's game is alot slower, plays take abit longer to be made and female goalkeepers are not as agile and athletic yadda yadda....those adjustments are however not made.....I'm the biggest football fan, and I don't consider myself to be chauvinist, but I do not find women's football (soccer) to be nearly as interesting or entertaining as the men's game.....it's a newer sport and it lacks the sophistication, physical nature and other elements of history & passion that make the men's game what it is.....so in that regard you can argue that football (soccer) is definitely as manly a sport as any...add to that the fact that football, unlike baseball is a contact sport....
End of the day, "manly" should not be taken to mean "good"...There is nothing inferior about sports such as tennis, swimming, track & field, volleyball, gymnastics, field hockey etc...where the women's version appears to be as competitive and entertaining as the men's...(to the ignorant non-follower such as myself)... and therefore calling football/soccer an "unmanly" sport should only be regarded as an insult if one endorses beliefs of female inferiority...and that's a whole other conversation.
In sum, does it really matter which sport is more "manly" ?