The formation of well-controlled interfaces between materials of different structure and bonding is a key requirement when developing new devices and functionalities. Of particular importance are epitaxial or low defect density interfaces between two-dimensional materials and three-dimensional semiconductors or metals, where an interfacial structure influences electrical conductivity in field effect and optoelectronic devices, charge transfer for spintronics and catalysis, and proximity-induced superconductivity. Epitaxy and hence well-defined interfacial structure has been demonstrated for several metals on van der Waals-bonded substrates. Semiconductor epitaxy on such substrates has been harder to control, for example during chemical vapor deposition of Si and Ge on graphene. Here, we demonstrate a catalytically mediated heteroepitaxy approach to achieve epitaxial growth of three-dimensional semiconductors such as Ge and Si on van der Waals-bonded materials such as graphene and hexagonal boron nitride. Epitaxy is "transferred"from the substrate to semiconductor nanocrystals via solid metal nanocrystals that readily align on the substrate and catalyze the formation of aligned nuclei of the semiconductor. In situ transmission electron microscopy allows us to elucidate the reaction pathway for this process and to show that solid metal nanocrystals can catalyze semiconductor growth at a significantly lower temperature than direct chemical vapor deposition or deposition mediated by liquid catalyst droplets. We discuss Ge and Si growth as a model system to explore the details of such hetero-interfacing and its applicability to a broader range of materials.