Evaluation

Regret

pyepo.metric.regret evaluates the decision quality of a prediction model, usually on a held-out test set. Regret is \(l_{Regret}(\hat{\mathbf{c}}, \mathbf{c}) = \mathbf{c}^\top \mathbf{w}^*(\hat{\mathbf{c}}) - \mathbf{c}^\top \mathbf{w}^*(\mathbf{c})\), the excess cost of the predicted solution over the true optimum. By default, instances are aggregated as normalized regret, \(\sum_i l_i \, / \, \sum_i |\mathbf{c}_i^\top \mathbf{w}^*(\mathbf{c}_i)|\), which is dimensionless and comparable across problem scales. Use reduction to switch to "sum", "mean", or "none" (a per-instance array). processes parallelizes the solves, as in training. Set processes=0 to use all available cores.

pyepo.metric.regret(predmodel: nn.Module | Callable, optmodel: optModel, dataloader: DataLoader, processes: int = 1, reduction: str = 'normalized') → float | np.ndarray

True regret (SPO loss) of a trained predictor.

Solves the optimization problem on the predicted cost vector \(\hat{\mathbf{c}}\), then measures the excess true objective incurred by that decision: \(l_i = \mathbf{c}_i^\top \mathbf{w}^*(\hat{\mathbf{c}}_i) - z^*(\mathbf{c}_i)\). With the default reduction="normalized" the result is \(\sum_i l_i / \sum_i |z^*(\mathbf{c}_i)|\), dimensionless and comparable across problem scales; instances with near-zero true optima inflate the ratio. PyTorch predictors are evaluated under eval(); the original mode is restored afterwards.

predmodel may also be a plain callable f(x: np.ndarray) -> array-like for JAX/Flax models; pass a functools.partial that closes over the current parameter pytree, e.g. functools.partial(model.apply, params).

Parameters:

  • predmodel – a PyTorch nn.Module for cost prediction, or a JAX callable f(x_numpy) -> cost_array

  • optmodel – a PyEPO optimization model

  • dataloader – PyTorch DataLoader over an optDataset (yielding (x, c, w, z, ...) tuples; fields beyond the first four, e.g. CaVE tight constraints, are ignored)

  • processes – number of processors, 1 for single-core, 0 for all of cores; a fresh worker pool is spawned per call, each worker rebuilding the model from its constructor args

  • reduction – “normalized” (sum of regrets over sum of absolute true optima), “sum”, “mean”, or “none” (per-instance array)

Returns:

aggregated regret, or per-instance regrets when reduction="none"

Return type:

float or np.ndarray

import pyepo

regret = pyepo.metric.regret(predmodel, optmodel, testloader)

Unambiguous Regret

When a predicted cost vector \(\hat{\mathbf{c}}\) yields multiple optimal solutions for \(\underset{\mathbf{w} \in \mathcal{S}}{\min}\;\hat{\mathbf{c}}^\top \mathbf{w}\), the regret depends on which optimum the solver happens to return. The unambiguous regret removes this ambiguity by scoring the worst case: \(l_{URegret}(\hat{\mathbf{c}}, \mathbf{c}) = \underset{\mathbf{w} \in W^*(\hat{\mathbf{c}})}{\max} \mathbf{w}^\top \mathbf{c} - \mathbf{c}^\top \mathbf{w}^*(\mathbf{c})\).

learning curves

unambRegret returns only the normalized value. It does not expose a reduction option. For each instance, it adds a constraint that restricts the feasible region to the tie set and then re-solves for the worst case. This makes it slower than regret and requires a backend that implements addConstr.

pyepo.metric.unambRegret(predmodel: nn.Module, optmodel: optModel, dataloader: DataLoader, tolerance: float = 1e-05, max_iter: int = 10) → float

Normalized unambiguous regret (worst-case-tie SPO loss).

When the predicted cost vector \(\hat{\mathbf{c}}\) yields multiple optimal solutions, regret reports the realized one while unambRegret reports the worst case over all optima: \(l_{\mathrm{URegret}}(\hat{\mathbf{c}}, \mathbf{c}) = \max_{\mathbf{w} \in W^*(\hat{\mathbf{c}})} \mathbf{c}^\top \mathbf{w} - z^*(\mathbf{c})\). More theoretically rigorous than regret, but in practice the two are nearly identical and unambRegret is rarely required. The result is normalized by \(\sum_i |z^*(\mathbf{c}_i)|\); instances with near-zero true optima inflate the ratio.

Parameters:

  • predmodel – a regression neural network for cost prediction

  • optmodel – a PyEPO optimization model

  • dataloader – PyTorch DataLoader over an optDataset (fields beyond (x, c, w, z) are ignored)

  • tolerance – precision used when rounding predicted costs to find ties

  • max_iter – maximum number of solve retries with relaxed tolerance

Returns:

normalized unambiguous regret

Return type:

float

import pyepo

regret = pyepo.metric.unambRegret(predmodel, optmodel, testloader)