Deformation of the pattern or the increase in line roughness during plasma etching becomes more significant with the shrink of complementary metal-oxide-semiconductor patterns. For aggressively scaled patternings, an organic underlayer (UL) is often used under the photoresist and a thin layer of a Si-containing hardmask. In this work, the effect of ultraviolet/vacuum ultraviolet (VUV) photons on UL parameters such as wavelength, photon dose, and process order was investigated using synchrotron radiation. First, the index of refraction and extinction coefficients of mask materials such as e-beam resist [hydrogen silsesquioxane (HSQ)] and organic UL (NFC-1400; NFC) were measured by utilizing the Kramers-Kronig relations and/or ellipsometry measurements depending on the wavelength involved. Second, VUV photons at specific wavelengths, corresponding to absorption maxima of HSQ and NFC at 54 nm (HSQ), 62 nm (HSQ and NFC), 88 nm (HSQ), 112 nm (NFC), 138 nm (HSQ), 155 nm (NFC), 194 nm (NFC), and 238 nm (NFC), were exposed before or/and after etching of NFC. The authors continued to etch into poly-Si, and the resulting line-edge roughness (LER) was measured. The improvement of the LER was seen at the wavelengths corresponding to the absorption maxima of NFC, and the degree of LER improvement was better at the higher photon dose up to 3 × 1017 photons/cm2. The LER reduced from 6.7 to 4.2 nm and to 3.6 nm when the VUV photons at wavelengths of 62 and 155 nm, respectively, were used for exposure. In contrast, pattern degradation was observed at the wavelengths corresponding to the absorption maxima of HSQ. These findings indicate that optimizing VUV radiation corresponding to the UL materials greatly influences the LER.