Deep learning can serve as bridge between data and physics-based methods when applied to drug discovery. Graph-based networks are a physically intuitive choice to generate representations of molecular systems. We have developed modular, graph-based neural network architectures that allow for flexible representations, ranging from chemically bonded ligand-based to three-dimensional representations of ligand-protein binding (see schematic below). Our networks yield useful visualizations that highlight molecule substructures relevant to activity and have been validated by the medicinal chemistry literature. These networks have also been applied to classification of ligand-protein structures obtained from molecular docking and have shown improved performance over baseline results in the task of choosing accurate binding modes. We also show how our networks are used to build classification models from large-scale datasets, including molecular dynamics trajectories.