Knowledge Graph Question Answering (KGQA) has become a prominent area in natural language processing due to the emergence of large scale Knowledge Graphs (KGs). Semantic parsing based approach is the predominant direction to solve the KGQA task where natural language question is translated into a logic form such as SPARQL query. Recently Neural Machine Translation based approaches are gaining momentum in order to translate natural language query to structured query languages thereby solving the KGQA task. However, most of these methods struggle with out-of-vocabulary words where test entities and relations are not seen during training time. In this work, we propose a modular two stage neural architecture to solve the KGQA task. Stage-I of our approach comprises a NMT-based seq2seq module that translates a question into a sketch of the desired SPARQL query called a SPARQL silhouette. Stage-II of our approach comprises a Neural Graph Search (NGS) module which aims to improve the quality of the SPARQL silhouette by detecting the right relations in the underlying knowledge graph. Experimental results show that we achieve substantial improvements and obtain state-of-the-art performance or comparable results to the best performing systems on two benchmark datasets. We believe, our proposed approach is novel and will lead to dynamic KGQA solutions that are well-suited for practical applications.