ZCU102 delay (in)stability

[jaycedowell]

This has been going on for a while so I thought I would open an issue for it and document what we've learned/where we're at.

When we did testing in the server room before deployment of the NDP servers to LWA1 we did a series of tests on the system. One of those was to take the broadband noise source, amplify it, and split it eight ways. Those signals were then injected into to ports on each of the four ZCU102 boards that we were testing with. Those tests showed that the relative delays between the inputs were not zero but they were stable across ADC initialization. That was ok since all we needed was stability. We then deployed slight different hardware (different ZCU102 boards) to LWA1 toward the end of 2025.

For the past few months I've been struggling to get good sensitivity with LWA1¹. After a lot of flailing around and a few dead ends (phantom PPS pulses, noise on the ADC control ribbon cables, noise in the rack) I came back to this idea that there was a delay stability problem since that was a good way to get reasonable results with re-processes TBT data but have problems with the online system. The test that clinched it was a Cyg A calibration sequence of:

• N captures before transit for calibration
• 2 captures for verification
• Run an INI
• 2 more captures for verification

The calibration from the first N captures improve the image quality of the pre-INI verification data but not that for the post-INI data. Testing since then has shifted to what's happening now where I take data without re-initializing the system, beam form the TBT data to get an offline measure of the SEFD, and leave things running as long as possible to see if the instability is only related to the startup.

Footnotes

1. "Good" here means a reasonable SEFD for a full station, about ~14 kJy, and not the ~100 kJy stuff we've been seeing. ↩

[jaycedowell]

"Where does this instability coming from? and "Why didn't we see it during testing?" are two things that keep coming up.

• For the first we always point to the SNAP2 setup at North Arm and how that doesn't seem to show this behavior. It's a slightly different system with SNAP2 boards but about the same control software (more on that later) and the same digitizer boards. If that works why don't the ZCUs?
• For the second question of why we didn't see it in testing I think that comes down to software churn from the co-development of the firmware and the control software.

The firmware/software development was "messy". On the software side there were two branches on caltech-lwa, more_general and more_general_take2. On the ADC specifically we have a history of changes at https://github.com/realtimeradio/caltech-lwa/commits/more_general/control_sw/src/blocks/adc.py ¹ with realtimeradio/caltech-lwa@c56ced3 probably being the biggest change to the behavior. Looking at a diff of what's in more_general_take2 vs master we see:

37a38,44
>     
>     :param n_boards_per_fmc: Number of digitizer boards connected to each FMC port.
>     :type n_boards_per_fmc: int
>     
>     :param cal_step_size: Number of IDELAY tap steps between error counts to use
>                           during calibration.
>     :type cal_step_size: int
40c47,48
<     def __init__(self, host, name, logger=None, passive=False):
---
>     def __init__(self, host, name, logger=None, passive=False, n_boards_per_fmc=NBOARDS,
>                        cal_step_size=TAP_STEP_SIZE):
49a58,59
>         self.n_boards_per_fmc = n_boards_per_fmc
>         self.cal_step_size = cal_step_size
117c127
<             for board in range(NBOARDS):
---
>             for board in range(self.n_boards_per_fmc):
129c139
<             for board in range(NBOARDS):
---
>             for board in range(self.n_boards_per_fmc):
133c143
<             for board in range(NBOARDS):
---
>             for board in range(self.n_boards_per_fmc):
162c172
<         ok, delays, slacks = self.calibrate(fail_hard=fail_hard)
---
>         ok, delays, slacks = self.calibrate(fail_hard=fail_hard, step_size=self.cal_step_size)
178c188
<             for board in range(2):
---
>             for board in range(self.n_boards_per_fmc):
224c234
<         out = np.zeros([8,8,NSAMPLES])
---
>         out = np.zeros([4*self.n_boards_per_fmc,8,NSAMPLES])
228c238
<         for i in range(8):     
---
>         for i in range(4*self.n_boards_per_fmc):     
264c274
<         out = np.zeros([32,2*NSAMPLES])
---
>         out = np.zeros([16*self.n_boards_per_fmc,2*NSAMPLES])
267c277
<         for i in range(8):     
---
>         for i in range(4*self.n_boards_per_fmc):     
300c310
<         for i in range(8):
---
>         for i in range(4*self.n_boards_per_fmc):
302c312
<         NTAPS=512
---
>         NTAPS = 512
304,306c314,316
<         d = np.zeros([NSTEPS, 8, 8, NSAMPLES]) # taps x chips x lanes x samples
<         errs = np.zeros([NSTEPS, 8, 8]) # taps x chips x lanes
<         for cs in range(8):
---
>         d = np.zeros([NSTEPS, 4*self.n_boards_per_fmc, 8, NSAMPLES]) # taps x chips x lanes x samples
>         errs = np.zeros([NSTEPS, 4*self.n_boards_per_fmc, 8]) # taps x chips x lanes
>         for cs in range(4*self.n_boards_per_fmc):
315c325
<             for cs in range(8):
---
>             for cs in range(4*self.n_boards_per_fmc):
319c329
<             for c in range(8):
---
>             for c in range(4*self.n_boards_per_fmc):
347c357
<         for i in range(8):
---
>         for i in range(4*self.n_boards_per_fmc):
352c362
<         errs = np.zeros([8, 8]) # taps x chips x lanes
---
>         errs = np.zeros([4*self.n_boards_per_fmc, 8]) # taps x chips x lanes
354c364
<         for c in range(8):
---
>         for c in range(4*self.n_boards_per_fmc):
432c442
<         for cs in range(8):
---
>         for cs in range(4*self.n_boards_per_fmc):
439c449
<         for cs in range(8):
---
>         for cs in range(4*self.n_boards_per_fmc):
489c499
<             for i in range(8):
---
>             for i in range(4*self.n_boards_per_fmc):
536c546
<             for i in range(8):
---
>             for i in range(4*self.n_boards_per_fmc):
546c556
<             for i in range(8):
---
>             for i in range(4*self.n_boards_per_fmc):
555c565
<             for i in range(8):
---
>             for i in range(4*self.n_boards_per_fmc):
558c568
<     def calibrate(self, use_ramp=False, fail_hard=True, step_size=TAP_STEP_SIZE, verbose=False):
---
>     def calibrate(self, use_ramp=False, fail_hard=True, step_size=None, verbose=False):
577c587
<         :type step_size: int
---
>         :type step_size: int or None
587a598,600
>         if step_size is None:
>             step_size = self.cal_step_size
>             
595c608
<             for board in range(2):
---
>             for board in range(self.n_boards_per_fmc):
597,598c610,616
<                 adc.decrement_bitslip_index(board) # empirically optimized
<                 adc.decrement_bitslip_index(board) # empirically optimized
---
>                 if self.n_boards_per_fmc == 2:
>                     adc.decrement_bitslip_index(board) # empirically optimized for the SNAP2
>                     adc.decrement_bitslip_index(board) # empirically optimized for the SNAP2
>                 elif self.n_boards_per_fmc == 1:
>                     adc.increment_bitslip_index(board) # empirically optimized for the ZCU102
>                     adc.increment_bitslip_index(board) # empirically optimized for the ZCU102
>                     adc.increment_bitslip_index(board) # empirically optimized for the ZCU102
601c619,620
<             for slip in range(2):
---
>             
>             for slip in range(2 if self.n_boards_per_fmc == 2 else 0):
603c622
<                 for board in range(2):
---
>                 for board in range(self.n_boards_per_fmc):
608,609c627,631
<                         adc.decrement_bitslip_index(board)
<                         adc.decrement_bitslip_index(board)
---
>                         if self.n_boards_per_fmc == 2:
>                             adc.decrement_bitslip_index(board)
>                             adc.decrement_bitslip_index(board)
>                         elif self.n_boards_per_fmc == 1:
>                             pass
617c639
<                 for board in range(2):
---
>                 for board in range(self.n_boards_per_fmc):
629c651
<                         if np.any(errs[-5:-1,4*board + chip:4*board + chip + 1,:] == 0):
---
>                         if np.any(errs[-5:-1, 4*board + chip:4*board + chip + 1,:] == 0):

Most of those changes are related to two digitizer boards on the ZCU102 vs four on the SNAP2 but lines 558, 597, 601, and 608 touch the actual procedure.

Footnotes

1. more_general_take2 doesn't have any additional changes to adc.py beyond these`. ↩

[jaycedowell]

As a slight tangent the SEFD at LWA1 from this morning's stress tests is 20 kJy. That's reasonable. LWA-SV is still struggling at ~130 kJy but that's related to some analog signal problems (low gains and high frequency noise).

[jaycedowell]

In addition to looking at stability and calibratability I'm also trying to see if reverting any of these changes to adc.py will move the ball.

[jaycedowell]

So, a few updates:

• If you don't INI NDP it seems to be stable. You can take take calibration data, run the calibration, apply it back, and get what you expect. It also seems stable over time on the ~few day timescale I've looked at.
• If you do INI then you randomize, even after removing all of the conditional branches up above that differentiate SNAP2 vs ZCU102 (I think I did all of that right).'

The plan for today is to do some INI/TBT capture sequences as quickly as possible at LWA1 to see if I can see any kind of pattern is what is changing and how.

[jaycedowell]

I've spent the afternoon looking at plots from the INI/capture tests. I've been focused on redundant-ish baselines in the array where I can get a few measurements of the same sky at ~40 MHz. I'm beginning to wonder if the problem isn't widespread but ZCU102-board specific. Here's what I'm seeing:

Board	Delay Error Mean [ns]	Delay Error Standard Deviation [ns]
zcu08	15.141222	105.40845
zcu03	2.6569507	103.63942
zcu07	-8.145986	91.180466
zcu02	-10.01008	87.699905
zcu15	7.3937006	76.33647
zcu10	-1.0519713	71.536285
zcu09	6.4649124	66.415985
zcu13	5.604452	65.34268
zcu01	-4.073661	63.887856
zcu04	3.8073769	62.234474
zcu11	-18.063217	61.879242
zcu16	0.30341882	55.879364
zcu12	-4.983568	53.072193
zcu14	17.547009	52.313667
zcu06	4.4	51.895702
zcu05	3.0168777	48.287292

The middle column isn't all that useful but the last column seems to be something like a "goodness of calibration metric", and 03 and 08 are the worst. If I drop all redundant-ish baselines that involve either of those two boards I get:

Board	Delay Error Mean [ns]	Delay Error Standard Deviation [ns]
zcu02	-9.786585	85.45304
zcu07	-4.986842	80.05982
zcu15	11.575758	75.61208
zcu10	-1.4963768	70.87161
zcu13	4.6938405	62.08302
zcu04	4.4789476	59.534683
zcu09	1.830292	56.418587
zcu11	-11.545267	53.330177
zcu01	2.2454545	52.04742
zcu06	-0.7650376	45.43441
zcu16	2.652968	44.958298
zcu14	15.932558	44.486282
zcu12	2.0159574	44.244682
zcu05	0.27608696	43.84575

The standard deviation seems to drop overall, not by a lot but by by some. Dropping 02 as well doesn't seem to move the needle much further from these "03 and 08"-less results

I'm not sure what to make of this right now. On one hand it could explain part of the "Why didn't we see this during testing?" question. On the other hand it's 2 out of 16 boards and I don't know if that's enough to explain the variation in sensitivity I'm seeing. In the morning another stress test will run at LWA1 and we'll see what happens there. The SEFD of LWA1 was at 21 kJy at 74 MHz this morning.

[jaycedowell]

SEFD this morning was 26 kJy. That seems consistent with "only a few board are off". Next step would be to try this again tomorrow (without an INI) to see if this holds. If it does then we can move on to INI+stress test every day.

[jaycedowell]

This morning was 26.5 kJy so that's holding.

I've done another INI this evening and we'll see what happens in the morning.

[jaycedowell]

This morning was 21.5 kJy. I'll run another INI.

[jaycedowell]

This morning was 37.5 kJy so still not stable. There'll be another INI this afternoon and test tomorrow.

[jaycedowell]

The stress tests didn't run over the weekend but the one this morning is 38 kJy.

[jaycedowell]

Here's a similar table to the above but for LWA-SV:

Board	Delay Error Mean [ns]	Delay Error Standard Deviation [ns]
zcu13	-58.782608	136.66893
zcu15	77.0	135.29758
zcu11	-60.57377	119.48701
zcu16	-44.712765	119.322075
zcu08	5.955357	115.44397
zcu10	126.63265	112.449
zcu14	41.962265	111.60704
zcu12	-12.363013	109.64281
zcu07	-59.023075	96.74951
zcu09	0.4177215	94.38068
zcu04	2.8116884	81.6718
zcu06	14.5	71.67296
zcu03	-11.049587	58.932632
zcu02	-2.0882354	37.56435
zcu05	8.658334	30.722277
zcu01	14.733945	25.552479

It's much worse but so is Sevilleta SEFD (~400 kJy this morning).

[jaycedowell]

I'm back on to the noise theory now that I'm looking at LWA1 vs. SV.

For phantom PPS ticks I see:
[0, 0, 0, 0, 0, 0, 12, 3, 0, 0, 0, 0, 0, 0, 0, 252] for zcu01 - 16 at LWA1
[0, 0, 0, 0, 0, 0, 74, 70, 0, 0, 0, 0, 9, 0, 0, 0] at SV

Both of those are post-INI from tonight. For those boards with non-zero ticks I see what looks like period variations in the sync pulse when a board registers another phantom PPS (or vice versa since it's hard to say which comes first).

[jaycedowell]

After running a bit:
LWA1 @ [0, 0, 0, 0, 0, 3, 12, 3, 0, 0, 0, 0, 3, 0, 0, 349]
SV @ [0, 0, 0, 0, 0, 0, 237, 156, 0, 0, 0, 0, 15, 0, 0, 0]

I'll do an INI as SV now to see what that does.

[jaycedowell]

Post-INI at SV is [0, 0, 0, 0, 0, 0, 59, 102, 0, 0, 0, 0, 43, 0, 0, 0]. So same three boards are showing the phantom ticks.