Biomedical data, particularly in the field of genomics, has many characteristics which make it challenging for machine learning applications – it is sparse, and high dimensional, and the targets associated with the data are not clean or noise-free measurements. Biomedical applications also present challenges to model selection – whilst powerful, accurate predictions are necessary, they alone are not sufficient for a model to be deemed useful in this environment. Due to the nature of the predictions, a model must also be trustworthy and transparent, empowering a practitioner with confidence that its use is appropriate and reliable. In this paper, we propose that this can be achieved through the use of judiciously built feature sets coupled with Bayesian models, specifically Gaussian processes. We apply Gaussian processes to drug discovery, using inexpensive transcriptomic profiles from human cell lines to predict animal kidney and liver toxicity after treatment with specific chemical compounds. This approach has the potential to reduce invasive and expensive animal testing during clinical trials if in vitro human cell line analysis can accurately predict model animal phenotypes. We compare results across a range of feature sets and models, to illustrate that – for medical applications – the model is of great importance.