Defects are ubiquitous in the semiconductor industry and detection, classification at various levels is a challenge and requires rigorous sampling. Every technology offers a new challenge in this area and requires numerous hours of setup, debug time for semiconductor integrated chip manufacturers. Intentional defects on patterns at various sizes in comparison to the critical dimension of a particular technology are used to preview their modulation and assess inspection tool performances. Figure 1 shows a few types of intentional defects as designed and their respective manifestation on a photomask. The chart also shows detection and printability limit and it varies by each defect type.1, 2 In recent years deep learning has made tremendous progress and has substantially improved the state-of-art in many applications that include object detection, speech recognition, and image classification3The application and design of these deep learning tools in semiconductor defect recognition can be very useful, as it has the potential to reduce the setup time, cost and improving the overall (defect) detection and classification accuracy. In this work we demonstrate the application of convolution neural networks on e-beam images of intentional defects on various types of patterns. We assess various filters and their outputs at each layer of a custom convolutional neural network (Figure 2) to ultimately improve the architecture and its accuracy. We also demonstrate the relative effectiveness of transfer learning in this application with limited availability of labelled data. Using the architecture illustrated in Figure 2 we are able to achieve a classification accuracy of 95%.