timeout_per_row not working in fit_catalogue

[tristen-shields]

The 'timeout_seconds_per_row' argument in the 'fit_catalgoue' method of 'SBI_Fitter' doesn't seem to be working, with some rows of my catalog stalling indefinitely despite setting the argument to 10 seconds. I think the function that actually samples the posterior with the timeout is commented out in line 6445 of sbi_runner.py

[tHarvey303]

Hi Tristen, thanks for the bug report! You are correct that the argument doesn't currently work. Interrupting code in Python requires running it in a seperate thread, and I found issues with this behaviour when running in a Jupyter notebook. It can also lead to memory leaks, file locks etc.

The more stable way to do this is the new max_sampling_time argument built into the posterior object (see https://sbi.readthedocs.io/en/latest/api_reference/_autosummary/sbi.inference.DirectPosterior.html) - however this isn't currently supported by LtU-ILI (and may require you to update your version of the sbi package). I've opened a pull request to enable support for this (see maho3/ltu-ili#207), which should hopefully be merged soon. I've just added support for this to synference, so this should work again when the PR is merged.

If you are running into issues with your catalog not sampling, then either your model doesn't work that well, then most likely there is some amount of model missspecifiation (i.e. your input doesn't look like anything in your training data). There are some methods for outlier detection, which might help. Have you trained a new model, or are you trying to use the existing model?

[tristen-shields]

Hi Thomas,

Sorry for my late reply and thank you for making that pull request! I have not trained a new model yet, I am using the one you wrote about in the synference paper. I was just running it on a general catalog of photometry, and I wanted to skip the ones that were out of distribution for now and figure out what went wrong with them later.