James Jory, Senior, was born in the village of St. Cleer, in Cornwall, England on August 7, 1787. His father was gamekeeper and gardener on an English estate, and James became a carpenter and mechanic. His wife, Mary Stephens Jory, was born October 28, 1792. They were married on September 28, 1814, and all of their eight children — two daughters and six sons — save their youngest child, Hugh, were born in England.
They were a working class family, which limited their prospects in England. Partly in search of opportunity and partly to escape an apprenticeship law which could take young boys away from their families and set them to work at the age of nine, the family left England for Canada in 1830, and settled about 40 miles upriver from St. Johns, New Brunswick.
The family farmed in Canada for six years but were frustrated by the poor soil. They moved to St. Johns and the men worked in the shipyards, where they heard that there was good land to be had in central Canada. The family booked passage to New York with the intention of taking the Erie Canal inland, but soon after arriving on October 31, 1836, they met a man from Missouri who convinced them that his home was in every way superior to Canada. They left New York a week later aboard a sailing vessel bound for New Orleans. From there, they took a sidewheel steamer up the Mississippi River to St. Louis.