Development of heuristic policies is a common solution approach for stochastic inventory control problems that are computationally intractable. In the inventory control literature, Monte Carlo simulation has been widely used to measure the performance of heuristic policies. In this paper, we propose control variate methods for the estimation of heuristic inventory control policies. Our methods construct control variates using the optimal control policies of relaxed optimization problems. We apply the methods to two inventory control problems. Numerical experiments demonstrate the effectiveness of our methods.