Evaluation

Regret

pyepo.metric.regret evaluates the decision quality of a prediction model. Regret is defined as \(l_{Regret}(\hat{\mathbf{c}}, \mathbf{c}) = \mathbf{c}^T \mathbf{w}^*(\hat{\mathbf{c}}) - z^*(\mathbf{c})\), which measures the excess cost of the predicted solution over the true optimum.

pyepo.metric.regret(predmodel: nn.Module, optmodel: optModel, dataloader: DataLoader) → float

Normalized true regret (SPO loss) of a trained predictor.

Solves the optimization problem on the predicted cost vector \(\hat{\mathbf{c}}\), then measures the excess true objective incurred by that decision: \(\sum_i (\mathbf{c}_i^\top \mathbf{w}^*(\hat{\mathbf{c}}_i) - z^*(\mathbf{c}_i)) / \sum_i |z^*(\mathbf{c}_i)|\). The result is dimensionless and comparable across problem scales. The predictor is automatically put into eval() for the call and restored to train() afterwards.

Parameters:

  • predmodel – a regression neural network for cost prediction

  • optmodel – a PyEPO optimization model

  • dataloader – PyTorch DataLoader over an optDataset (yielding (x, c, w, z) tuples)

Returns:

normalized true regret

Return type:

float

import pyepo

regret = pyepo.metric.regret(predmodel, optmodel, testloader)

Unambiguous Regret

When a predicted cost vector \(\hat{c}\) yields multiple optimal solutions for \(\underset{\mathbf{w} \in S}{\min}\;\hat{\mathbf{c}}^T \mathbf{w}\), the unambiguous regret considers the worst case: \(l_{URegret}(\hat{\mathbf{c}}, \mathbf{c}) = \underset{\mathbf{w} \in W^*(\hat{\mathbf{c}})}{\max} \mathbf{w}^T \mathbf{c} - z^*(\mathbf{c})\).

learning curves

As the figure shows, regret and unambiguous regret are nearly identical across all training procedures. While the unambiguous regret is more theoretically rigorous, it is rarely necessary in practice.

pyepo.metric.unambRegret(predmodel: nn.Module, optmodel: optModel, dataloader: DataLoader, tolerance: float = 1e-05) → float

Normalized unambiguous regret (worst-case-tie SPO loss).

When the predicted cost vector \(\hat{\mathbf{c}}\) yields multiple optimal solutions, regret reports the realized one while unambRegret reports the worst case over all optima: \(l_{\mathrm{URegret}}(\hat{\mathbf{c}}, \mathbf{c}) = \max_{\mathbf{w} \in W^*(\hat{\mathbf{c}})} \mathbf{c}^\top \mathbf{w} - z^*(\mathbf{c})\). More theoretically rigorous than regret, but in practice the two are nearly identical and unambRegret is rarely required.

Parameters:

  • predmodel – a regression neural network for cost prediction

  • optmodel – a PyEPO optimization model

  • dataloader – PyTorch DataLoader over an optDataset

  • tolerance – precision used when rounding predicted costs to find ties

Returns:

normalized unambiguous regret

Return type:

float

import pyepo

regret = pyepo.metric.unambRegret(predmodel, optmodel, testloader)