Inconsistent prediction: pred in logger vs pred from .predict function

[chpoonag]

I have got a GAE model trained with data pyg_graph_train.
Then, I use pyg_graph_test for model prediction.

I tried this:
pred, score = model.predict( pyg_graph_test, label = pyg_graph_test.label, return_score=True )
And I got "Recall 0.7490 | Precision 0.7490 | AP 0.6226 | F1 0.7490"

But when I check the pred and score:
f1_score(y_true=pyg_graph_test.label, y_pred=pred)
I got 0.34680888045878483, which is inconsistent.

I found that the returned pred from the predict function is not the same as that in the logger function (pygod.utls.utility), because of different threshold values.
In the logger function:
contamination = sum(target) / len(target)
threshold = np.percentile(score, 100 * (1 - contamination))
pred = (score > threshold).long()

In contrast, in the predict function (pygod.detector.base):
if return_pred:
pred = (score > self.threshold_).long()
The "self.threshold_" is determined in _process_decision_score as:
self.threshold_ = np.percentile(self.decision_score_, 100 * (1 - self.contamination))

So, which prediction (i.e. which threshold value) is correct? Or is there something I may have missed/overlooked instead?

[kayzliu]

Sorry for the confusion.

If you do have the label or you know exactly how many outliers are in the dataset, e.g., 15%, you can specify the contamination in the initialization of the detector, for example model = DOMINANT(contamination=0.15). The model will make the binary prediction pred based on this contamination.

However, in many cases, our user does not have any label. We set a default contamination to 0.1. The threshold is changed correspondingly. That's why you got ~0.3 for F1. The ~0.7 F1 is evaluated with labels, which means the contamination is set to an ideal value.

To avoid setting the threshold, we also provide AUC, AP, and Recall@k for easier evaluation.

[withMoonstar]

Hello. I've also been using GAE for anomaly detection recently. However, errors have been constantly reported during the import process. Could I refer to your usage code?

The following is my error message. Thank you very much.

RuntimeError: pyg::neighbor_sample() Expected a value of type 'Optional[Tensor]' for argument 'edge_weight' but instead found type 'bool'.

The following is my code:

from pygod.detector import GAE
from pygod.utils import load_data
from sklearn.metrics import roc_auc_score, average_precision_score

Function to train the anomaly detector

def train_anomaly_detector(model, graph):
return model.fit(graph)

Function to evaluate the anomaly detector

def eval_anomaly_detector(model, graph):
outlier_scores = model.decision_function(graph)
auc = roc_auc_score(graph.y.numpy(), outlier_scores)
ap = average_precision_score(graph.y.numpy(), outlier_scores)
print(f'AUC Score: {auc:.3f}')
print(f'AP Score: {ap:.3f}')

graph = load_data('weibo')

Initialize and evaluate the model

graph.y = graph.y.bool()

if hasattr(graph, 'edge_weight'):
graph.edge_weight = None

model = GAE(epoch=100)
model = train_anomaly_detector(model, graph)
eval_anomaly_detector(model, graph)