Dynamic nuclear spin polarization (DNP) mediated by paramagnetic point defects in semiconductors is a key resource for both initializing nuclear quantum memories and producing nuclear hyperpolarization. DNP is therefore an important process in the field of quantum-information processing, sensitivity-enhanced nuclear magnetic resonance, and nuclear-spin-based spintronics. DNP based on optical pumping of point defects has been demonstrated by using the electron spin of nitrogen-vacancy (NV) center in diamond, and more recently, by using divacancy and related defect spins in hexagonal silicon carbide (SiC). Here, we describe a general model for these optical DNP processes that allows the effects of many microscopic processes to be integrated. Applying this theory, we gain a deeper insight into dynamic nuclear spin polarization and the physics of diamond and SiC defects. Our results are in good agreement with experimental observations and provide a detailed and unified understanding. In particular, our findings show that the defect electron spin coherence times and excited state lifetimes are crucial factors in the entire DNP process.