Quantum Gibbs state sampling algorithms generally suffer from either scaling exponentially with system size or requiring specific knowledge of spectral properties a priori. These algorithms also require a large overhead of bath or scratch or ancilla qubits. We propose a method, termed the minimal effective Gibbs ansatz (MEGA), which uses a quantum computer to determine a small effective ensemble of pure states that accurately reproduce thermal averages of an objective dynamic correlation function. This technique employs properties of correlation functions that can be split into lesser and greater parts; here, we primarily focus on single-particle Green's functions and density-density correlators. When properly measured, these correlation functions provide a simple test to indicate how close a given pure state or ensemble of pure states is to providing accurate thermal expectation values. Further, we show that, when properties such as the eigenstate thermalization hypothesis hold, this approach leads to accurate results with a sparse ensemble of pure states; sometimes only one suffices. We illustrate the ansatz using exact diagonalization simulations on small clusters for the Fermi-Hubbard and Hubbard-like models. Even if the MEGA becomes as computationally complex as other Gibbs state samplers, it still gains an advantage due to its ease of implementation without any a priori information about the Hamiltonian and in the efficient allocation of available qubits by eliminating bath qubits and using a minimal number of ancillas.