COVID-19 is threatening the health of the entire human population. In order to control the spread of the disease, epidemiological investigations should be conducted, to trace the infection source of each confirmed patient and isolate their close contacts. However, the analysis on a mass of case reports in epidemiological investigation is extremely time-consuming and labor-intensive. This paper presents an end-to-end framework for automatic epidemiological case report analysis and inference, in which a Tuple-based Multi-Task Neural Network (TMT-NN) is designed and implemented for jointly recognizing epidemiological entities and relations from case reports, and an epidemiological knowledge graph and its corresponding inference engine are built to uncover the infection modes, sources and pathways. Preliminary experiments demonstrate the promising results, and we published a real data set of COVID-19 epidemiological investigation corpora at Github, as well as contributing our COVID-19 epidemiological knowledge graph to the open community OpenKG.cn.