Gain solver stops too early, results in bad calibration

[baron-de-montblanc]

I've identified an issue in fhd_core/calibration/vis_calibrate_subroutine.pro. Specifically, several of my observations (MWA Fall 2016 season) are triggering the following breakpoint (lines 257-266):

; Halt if the strict convergence is worse than most of the recent iterations
divergence_test_1 = convergence_strict ge median(conv_test[fii, i - divergence_history - 1 : i - 1])
; Also halt if the convergence gets significantly worse in one iteration
divergence_test_2 = convergence_strict ge min(conv_test[fii, 0 : i - 1]) * divergence_factor
if divergence_test_1 or divergence_test_2 then begin
     ; If both measures of convergence are getting worse, we need to stop.
     print, string(format = '("Calibration diverged at iteration ", A, " for pol_i: ", A, " freq_i: ", A,". Convergence was: ", A, " threshold was: ", A)', $
                  Strn(i), Strn(pol_i), Strn(fi), Strn(conv_test[fii, i - 1], format = '(e12.2)'), Strn(conv_thresh, format = '(e12.2)'))
     divergence_flag = 1
     break

This is happening early on, at iteration ~10, for all tiles, but only certain frequencies. Here is an example of the resulting cal_amp.png:

I investigated the gain vs. iteration history for the same OBSID (1161091512):

Now if I comment out the divergence_flag = 1 and break on lines 265-266, and run the same calibration job, we see that frequency 111 is actually able to converge:

divergence_test_1 and divergence_test_2 are entirely dependent on parameters hard-coded in lines 22-23:

divergence_history = 3 ; halt if the strict convergence is worse than most of the last x iterations
divergence_factor = 1.5 ; halt if the convergence gets significantly worse by a factor of x in one iteration

So maybe a solution would be to lower these thresholds. Happy to test different values and open to any other suggestions!

[bhazelton]

@baron-de-montblanc nice find! Love the plots, they really tell the story. We discussed this on the telecon today and our vote is to start by making it so these divergence checks are only done after a certain number of iterations. That number should be settable, but we think you should default it to 20 based on your plots.

Another idea that Miguel had, which we think might be a good thing to try is to have the starting gain be set to the value from the previous frequency (if it converged happily). This is not instead of the fix above, but it might just make things happier and might result in faster convergence. The underlying assumption here is that gains tend to be fairly smooth in frequency, which is generally true.