You don't eat trees and gathering is about gathering food. That's what hunter-gatherers do. They hunt food and they gather food, and once there's too little food left, they take their stuff and go off to an another location where they can hunt and gather stuff.
If "most" of what women did would have been looking after babies, then humanity would have been long extinct, because men would have never been able to provide enough food to feed everyone.
Also, don't forget the fact that a good chunk of women in any hunter-gatherer tribe would be past the menopause. It's tricky to be a baby-factory when you aren't making babies anymore. Here's a quote for you:
Some evidence suggests that hunters contribute less than half the total food budget of most hunter-gatherer societies, and often much less than half, so that foraging grandmothers can contribute substantially to the survival of grandchildren at times when mothers and fathers are unable to gather enough food for all of their children. In general, selection operates most powerfully during times of famine or other privation. So although grandmothers might not be necessary during good times, many grandchildren cannot survive without them during times of famine.
Besides, children don't need anywhere near as much "looking after" as people tend to imagine. My mom was always so busy weeding the fields and washing the clothes (no running water and no washing machine makes this an extremely time-consuming task) and making food, that she never had the time to look after me or my brother. We just ran around doing whatever.
Finally, it's all also about specialization. One of the hypotheses that explains why Homo Sapiens out-competed the Neaderthals points to the fact that Neanderthals had no specialization between the sexes. Neanderthal women both hunted and gathered, and Neanderthal men both hunted and gathered, and that's inefficient. Human men and women specialised with men hunting and women gathering, both evolved to be good at those specific tasks, and that contributed to the success of human species.
Here's a video for you:
http://www.youtube.com/watch?v=OLHh9E5ilZ4