It was second wife Maria Annunciata who bore Karl Ludwig four children: Franz Ferdinand (Graz, 1863), Otto (Graz, 1865), Ferdinand (Vienna, 1868) and Marguerite (Artstetten, 1870). Unfortunately, she died in 1871, so Karl Ludwig's third wife, Maria-Theresia, was principal 'mother' to the four children, in addition to the two – Maria Annunziata (1876) and Elisabeth (1878) – she gave birth to herself. She never exhibited favouritism in her affections toward her own daughters or her step-children. On the contrary, a woman of outstanding character, Maria-Theresia resolutely stood by Franz Ferdinand during and after his controversial marriage, along with the facetious cold-shouldering that followed. Although young Franz Ferdinand was often brooding and introspective, a bit of an enigma to his doting parents and siblings, all of Karl Ludwig's children enjoyed a happy childhood. Whether in Vienna or Graz in winter, or Schloß Artstetten above the Danube near Melk or Schloß Wartholz near Reichenau at the foot of the Raxalpe in the Höllental in summer, the entire family was always together.

As a nephew of the Allerhöchsten 'Emperor of Emperors', Franz Josef I of Habsburg, the young Erzherzog Franz Ferdinand was far down the list of future Emperors of Austria. He inherited many titles and estates, including the lovely d'Este estates of the Duke of Modena at the age of twelve. Being the scions of aristocracy, however, meant a pretty strict education, supervised by Graf Degenfeld. Among the series of tutors were Bohuslaw Graf Aichelburg, Karl Graf Coreth, Johann Graf Nostitz-Rieneck, Georg Graf Wallis, Major Johann Witek, and Otto Klopp. The same curriculum was followed as was taught throughout the Empire's schools, but with a few changes and additions, necessary to both condition and prepare the children for the various, and sometimes onerous, Imperial/Royal and public duties later on in life. The Arts and art history were taught personally by their father Karl Ludwig.

Franz Ferdinand took to all subjects well – except to languages. French and English he could managed well enough, but there were also the three main languages of the Empire: Magyar (Hungarian), Czech and Italian. These he found difficult – particularly Magyar, which would cause him problems later on, both in the media and in Hungary itself. Religious education – a very serious subject in Catholic Austria – was charged to Prelate Dr. Gottfried Marschall.

The body physical was also deemed important (en vogue throughout the German-speaking world at the time, eg. the outdoor 'health & efficiency' movement), and so sports – particularly gymnastics, swimming and ice-skating – was emphasized. It was perhaps during some of these outdoor activities that Franz Ferdinand developed his interest of the hunt – a leisure activity that would later evolve into something perhaps a little more than a passion. Evenings were usually devoted to the Arts, meeting artists, singers, musicians, thespians and poets, but also scientists, inventors and explorers.