Fully Homomorphic Encryption (FHE) has become a key enabler for secure computation with end-to-end quantum-safe encryption. Unlike previous cryptographic paradigms which necessitate decryption of the data in order to carry out computations, FHE makes it possible to compute in the encrypted space, thus eliminating potential vulnerabilities that may arise from storing unencrypted data on untrusted third-party cloud servers. However, the aforementioned encrypted computation operations, while quantum-safe, come with substantial performance and energy overheads, in comparison to the equivalent unencrypted operations on plaintext. Several algorithmic improvements have significantly reduced this overhead, and yet encrypted computations remain 3–4 orders of magnitude slower and less efficient than unencrypted operations. This has resulted in the emergence of several novel architecture proposals, software libraries and runtimes with the aim of bridging this gap. In this paper, we carry out a detailed characterization of popular FHE libraries on a server-class CPU and GPU across key FHE primitives and a credit card fraud detection application. We study the effect of both algorithmic parameters such as the security level, degree of packing and polynomial degree, as well as system-level parameters, such as allocated cores and memory resources, on the performance and energy efficiency of execution. We hope that this characterization can help bridge the gap between cryptographers and system architects in order to realize a commercially-viable system with end-to-end FHE.