Evaluation

Regret

pyepo.metric is to evaluate model performance. Regret \(l_{Regret}(\hat{\mathbf{c}}, \mathbf{c}) = \mathbf{c}^T \mathbf{w}^*(\hat{\mathbf{c}}) - z^*(\mathbf{c})\) aims to measure the error in decision-making. It evaluates the distance between the objective value of the solution from predicted cost \(\hat{c}\) and the true optimal objective value \(z^*(c)\).

pyepo.metric.regret(predmodel, optmodel, dataloader)

A function to evaluate model performance with normalized true regret

Parameters:

  • predmodel (nn) – a regression neural network for cost prediction

  • optmodel (optModel) – an PyEPO optimization model

  • dataloader (DataLoader) – Torch dataloader from optDataSet

Returns:

true regret loss

Return type:

float

import pyepo

regret = pyepo.metric.regret(predmodel, optmodel, testloader)

Unambiguous Regret

Given a cost vector \(\hat{c}\), there may be multiple optimal solutions of \(\underset{\mathbf{w} \in S}{\min}\;\hat{\mathbf{c}}^T \mathbf{w}\). Unambiguous Regret Loss \(l_{URegret}(\hat{\mathbf{c}}, \mathbf{c}) = \underset{\mathbf{w} \in W^*(\mathbf{c})}{\max} \mathbf{w}^T \mathbf{c} - z^*(\mathbf{c})\) considers the worst cases.

learning curves

However, as figure shows, the regret and the unambiguous regret are almost same in all training procedures. Therefore, although the unambiguous regret is more theoretically rigorous, it is not necessary to consider it in practice.

pyepo.metric.unambRegret(predmodel, optmodel, dataloader, tolerance=1e-05)

A function to evaluate model performance with normalized unambiguous regret

Parameters:

  • predmodel (nn) – a regression neural network for cost prediction

  • optmodel (optModel) – an PyEPO optimization model

  • dataloader (DataLoader) – Torch dataloader from optDataSet

Returns:

unambiguous regret loss

Return type:

float

import pyepo

regret = pyepo.metric.unambRegret(predmodel, optmodel, testloader)