The growth of Co-substituted BaTiO3 (BTO) films on Ge(001) substrates by molecular beam epitaxy is demonstrated. Energy-dispersive x-ray spectroscopy and transmission electron microscopy images confirm the uniform Co distribution. However, no evidence of magnetic ordering is observed in samples grown for Co concentrations between 2% and 40%. Piezoresponse force microscopy measurements show that a 5% Co-substituted BTO sample exhibits ferroelectric behavior. First-principles calculations indicate that while Co atoms couple ferromagnetically in the absence of oxygen vacancies, the occurrence of oxygen vacancies leads to locally antiferromagnetically coupled complexes with relatively strong spin coupling. The presence of a significant amount of oxygen vacancies is suggested by x-ray photoelectron spectroscopy measurements.