That is only true in peak times, and by necessity, not deliberation.
Think again. The majority of stories I heard from women who attended hajj or umrah involved seemingly deliberate sexual harassment, the most memorable being of my friend who had to make the tawaf with someone's erection pressed against her ass the entire time. It was crowded, of course, as there would be no other way he could have had the opportunity to be so close behind her.
But it seems public sexual harassment is something to be expected in that culture where women are rarely around in public, and the holy site is not immune. There were a handful of times when I would be walking through a semi-crowded market in Kuwait and get my ass or boobs groped by a man in passing. And I was a very modestly dressed Muslim woman, not one of those fashionable Kuwaiti gals.
It is a dreadful feeling to see a crowd of Arab men in a market and realize you must walk through them, or even nearby. My Muslim friends cited that as one of the reasons they were fine with the Islamic standard that women remain in the home.