Skip to content

perf: Numba JIT compute_pt2_scores Python inner loop #44

Description

@thc1006

Context

_pt2_helpers.py:33-114 compute_pt2_scores has a Python loop over candidate configs (typically 5000-15000 per iter):

```python
for idx in range(n_cand):
cand_h = cand_hash_list[idx]
if cand_h in basis_hash_set:
continue
config = candidates[idx]
h_xx = float(hamiltonian.diagonal_element(config))
...
connected, h_elements = hamiltonian.get_connections(config)
...
for j in range(len(connected)):
c_y = basis_coeff_map.get(conn_hashes[j], 0.0)
if c_y != 0.0:
coupling += float(h_elements[j]) * c_y
scores[idx] = coupling**2 / denom
```

For 5000 candidates × ~hundreds of connections each = millions of dict lookups and float multiplications, all in Python.

Observed wall: 3-5 min per iter at pt2_top_k=2000 (5000 candidates).
With pt2_top_k=5000 (CHEAP-KNOBS variant), candidate count grows → ~5-8 min.

Proposal

JIT the inner loop with Numba. The bottleneck is:

  1. Per-candidate Slater-Condon connection enumeration (already Numba in numba_get_connections)
  2. Per-candidate dot product over connected configs vs basis coefficients

The dict-based hash lookup is the slow part. Replace with array-based hash table or sorted-search:

  • Convert basis_coeff_map: dict[int, float] to (sorted_keys: np.ndarray, sorted_vals: np.ndarray)
  • Use np.searchsorted for O(log n) lookup vectorized
  • Wrap the per-candidate loop in @njit

Expected impact

  • 3-5 min per iter → ~1 min per iter (~70% PT2 speedup)
  • Total iter: 40 min → ~38 min (5-8% wall savings)
  • 30-iter job: ~1-1.5h saved

Risk

  • Numba JIT compile overhead first call (~30s, paid once)
  • Hash collision handling: integer hashes via config_integer_hash, already collision-free for n_qubits ≤ 64
  • API compatibility: hamiltonian.get_connections returns torch tensors → need .cpu().numpy() conversion inside JIT

Implementation sketch

```python
from numba import njit

@njit(cache=True)
def _pt2_inner_loop(connected_int_array, h_elements_array, basis_keys_sorted, basis_vals, e0, h_xx):
coupling = 0.0
for j in range(len(connected_int_array)):
idx = np.searchsorted(basis_keys_sorted, connected_int_array[j])
if idx < len(basis_keys_sorted) and basis_keys_sorted[idx] == connected_int_array[j]:
coupling += h_elements_array[j] * basis_vals[idx]
denom = max(abs(e0 - h_xx), 1e-14)
return coupling**2 / denom
```

Loop body becomes: get connections (already Numba'd) → call `_pt2_inner_loop`.

Verification

  • Add test `test_pt2_helpers.py::test_compute_pt2_scores_numba_matches_python`: identical scores within 1e-12 across 100 random configs.
  • Benchmark: report time for 5000 candidates baseline vs JIT.

Effort estimate

  • 1-2 days dev + testing

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions