The Poe family were Fayette County, Alabama farmers. According to the Federal Census of 1860, James Poe (b. ca. 1796) and Mary McMillian Poe (b. ca. 1792) were both from North Carolina. By 1825, the Poes were living in Alabama, where all their children were born. Three of their sons, James R. Poe (b. ca. 1825), William S. Poe (b. ca. 1837), and George W. Poe (b. ca. 1840), served in the Confederate Army. James R. Poe served in the 12th Alabama Cavalry (Malones) and later in the 9th Alabama Cavalry. He was taken prisoner and died at Camp Morton prison camp in Indiana on August 18, 1864, leaving behind a wife, Lemiley, and eight children. William S. Poe and George W. Poe served as privates in Co. I, 32nd Alabama Infantry. George died of measles on June 26, 1862 in Mobile County, Alabama. William was taken prisoner at the Battle of Missionary Ridge in November 1863 and spent the remainder of the war at Rock Island prison camp in Illinois.