pyepo.metric

Performance evaluation

Submodules

Package Contents

Functions

MSE(predmodel, dataloader)

A function to evaluate model performance with MSE

calRegret(optmodel, pred_cost, true_cost, true_obj)

A function to calculate normalized true regret for a batch

regret(predmodel, optmodel, dataloader)

A function to evaluate model performance with normalized true regret

calUnambRegret(optmodel, pred_cost, true_cost, true_obj)

A function to calculate normalized unambiguous regret for a batch

unambRegret(predmodel, optmodel, dataloader[, tolerance])

A function to evaluate model performance with normalized unambiguous regret

SPOError(pred_cost, true_cost, model_type, args)

A function to calculate normalized true regret

makeSkScorer(optmodel)

A function to create sklearn scorer

makeAutoSkScorer(optmodel)

A function to create Auto-SKlearn scorer
pyepo.metric.MSE(predmodel, dataloader)

A function to evaluate model performance with MSE

Parameters:

  • predmodel (nn) – a regression neural network for cost prediction

  • dataloader (DataLoader) – Torch dataloader from optDataSet

Returns:

MSE loss

Return type:

float

pyepo.metric.calRegret(optmodel, pred_cost, true_cost, true_obj)

A function to calculate normalized true regret for a batch

Parameters:

  • optmodel (optModel) – optimization model

  • pred_cost (torch.tensor) – predicted costs

  • true_cost (torch.tensor) – true costs

  • true_obj (torch.tensor) – true optimal objective values

Returns:predmodel

float: true regret losses

pyepo.metric.regret(predmodel, optmodel, dataloader)

A function to evaluate model performance with normalized true regret

Parameters:

  • predmodel (nn) – a regression neural network for cost prediction

  • optmodel (optModel) – an PyEPO optimization model

  • dataloader (DataLoader) – Torch dataloader from optDataSet

Returns:

true regret loss

Return type:

float

pyepo.metric.calUnambRegret(optmodel, pred_cost, true_cost, true_obj, tolerance=1e-05)

A function to calculate normalized unambiguous regret for a batch

Parameters:

  • optmodel (optModel) – optimization model

  • pred_cost (torch.tensor) – predicted costs

  • true_cost (torch.tensor) – true costs

  • true_obj (torch.tensor) – true optimal objective values

Returns:

unambiguous regret losses

Return type:

float

pyepo.metric.unambRegret(predmodel, optmodel, dataloader, tolerance=1e-05)

A function to evaluate model performance with normalized unambiguous regret

Parameters:

  • predmodel (nn) – a regression neural network for cost prediction

  • optmodel (optModel) – an PyEPO optimization model

  • dataloader (DataLoader) – Torch dataloader from optDataSet

Returns:

unambiguous regret loss

Return type:

float

pyepo.metric.SPOError(pred_cost, true_cost, model_type, args)

A function to calculate normalized true regret

Parameters:

  • pred_cost (numpy.array) – predicted costs

  • true_cost (numpy.array) – true costs

  • model_type (ABCMeta) – optModel class type

  • args (dict) – optModel args

Returns:

regret loss

Return type:

float

pyepo.metric.makeSkScorer(optmodel)

A function to create sklearn scorer

Parameters:

optmodel (optModel) – optimization model

Returns:

callable object that returns a scalar score; less is better.

Return type:

scorer

pyepo.metric.makeAutoSkScorer(optmodel)

A function to create Auto-SKlearn scorer

Parameters:

optmodel (optModel) – optimization model

Returns:

callable object that returns a scalar score; less is better.

Return type:

scorer