Speech data has the potential to become a powerful tool to provide quantitative information about emotion beyond that achieved by subjective assessments. Based on this concept, we investigate the use of speech to identify effects in subjects under the influence of two different drugs: Oxytocin (OT) and 3, 4-methylenedioxymethamphetamine (MDMA), also known as ecstasy. We extract a set of informative phonological features that can characterize emotion. Then, we perform classification to detect if the subject is under the influence of a drug. Our best results show low error rates of 13% and 17% for the subject classification of OT and MDMA vs. placebo, respectively. We also analyze the performance of the features to differentiate the two levels of MDMA doses, obtaining an error rate of 19%. The results indicate that subtle emotional changes can be detected in the context of drug use.