Uncertainty is a major challenge in reservoir management. To take the uncertainty into consideration, optimization can be carried out over a set of scenarios. Most approaches on reservoir management under uncertainty optimize a sequence of control inputs applied to all scenarios over the prediction horizon; hence, they are open-loop predictions. In this paper, we optimize over control policies, as opposed to a sequence of control inputs, to obtain closed-loop predictions. The policies are specified as a set of implicit algebraic equations, allowing for efficient gradient calculation by an adjoint simulation. The method is compared with the more traditional open-loop approach in a case study, indicating a significant potential for reservoir optimization by use of closed-loop predictions.