Devices in Internet of Things (IoT) often offer services that allow tenants to access data of different metrics collected from sensors. These sensors can be built-in or remotely connected to such devices. Given that such monitoring services are usually invoked within devices that have limited IT resource capacities, it is impossible to collect data of all metrics in the application’s context with a very high frequency. In this paper, we propose a framework that determines which metrics to monitor, monitoring start times, the optimal allocation of metrics to devices, and the optimal monitoring frequency of these metrics, without exceeding different device-specific time-varying resource capacities. Our approach is also adaptive; it gives updated solutions whenever a trigger happens in the system necessitating the need for a change in the previous optimal decisions. We provide an implementation of our approach and present numerical results showing its usage and limitations. At the heart of our approach is an integer programming optimization model that might be hard to solve for large-sized IoT systems. Thus, we present another predictive model that predicts for the user whether our optimization-based approach would be appropriate for her system or not. That is, whether the optimization model is predicted to give optimal solutions within some user-given optimality gaps in a time less than or equal to some user-given maximum allowed time.