Maybe the origins of the hajj or pilgrimage can be found somewhere else, especially if there were more than one Ka'ba.
The Guide-book: A Pictorial Pilgrimage to Mecca and Medina.
by Richard F. Burton
Stage I: The Pilgrims Land at Jeddah.
We are about to describe one of the most important scenes in the Mohammedan's life. A pilgrimage to Mecca, followed by a visitation to Medina, are, under certain limitations, obligatory upon all true believers, and many who have led evil lives date their reformation from the first sight of the holy shrines.
There is little doubt that this pilgrimage, like all others, began with a mixture of commerce and religion: the latter element now predominates. In former years, when travelling was more difficult, the hadgee (Haji), or pilgrim, wore, after his return home, a green turban. The custom is now obsolete in the more civilized lands. Maids, wives, and widows go through the ceremonial enactments, and "O pilgrimess!" is the civil address to women of the lower orders in Egypt and other Moslem lands.
From a brilliant link here!