We introduce a novel approach of addressing inter-dataset variability in the context of speaker recognition in a mismatched condition under the JHU-2013 domain adaptation challenge (DAC) framework. Previously, we took a subspace removal approach for inter-dataset variability compensation (IDVC) of within speaker variability. In this work we substitute subspace removal with incorporation of the variability into the Probabilistic Linear Discriminant Analysis (PLDA) model. We do that by introducing a novel optimality criterion which is minimizing the expected square error in estimation of the log-likelihood ratio of target trials when dataset-dependent PLDA models are replaced by a dataset independent PLDA model. The result we obtain is a correction term for the commonly estimated within speaker variability matrix. The correction term represents the normalized inter-dataset variability of the within speaker variability matrices. The proposed method outperforms the extended IDVC method on the DAC.