Many were looking in anticipation for the fulfillment of Jeremiah's prophesy. They longed and wept by the waters of Babylon as they remembered Zion. Those of devout faith and their ancestors awaited God's call to return. Maybe they were a remnant chosen by God.
As others have mentioned, those who did not return may have been too old or may have become complacent...or just comfortable in their "new" life. Or, maybe they drifted from their faith in God all together...assimilating into a pagan faith.
Those who follow Jesus' call into a radically different lifestyle or mission are those who have surrendered all and been obedient to God's leading. They "walk with God", they hear His voice and they follow.