BUG: Division results in 0 values for large dfs with numpy==2.3.5

[tilyevsky12]

Pandas version checks

• I have checked that this issue has not already been reported.

• I have confirmed this bug exists on the latest version of pandas.

• I have confirmed this bug exists on the main branch of pandas.

Reproducible Example

import pandas as pd
import numpy as np

# Set dimensions
n_rows = 5000
n_cols = 4097

# Create column names
cols = [f"col_{i}" for i in range(1, n_cols + 1)]

# Generate random DataFrame
df = pd.DataFrame(np.random.randn(n_rows, n_cols), columns=cols)

# Generate Series with same column index
series = pd.Series(np.ones(n_cols), index=cols)

# Divide DataFrame by Series (broadcasting across rows)
result = df.div(series)

df_from_np = pd.DataFrame(df.values/series.values, index=df.index, columns=df.columns)

print (df.equals(result))# -> False when n_cols > 4096
print (df.equals(df_from_np))# -> True

Issue Description

If you have a df with more than 4096 columns, dividing it by a series (row/element wise division), you can get 0s in the result for random entries in the df. However, doing this in numpy directly and then converting back into a df works no problem. I have confirmed that downgrading numpy to 1.26.4 resolves the issue about equals statements will print True.

Expected Behavior

After dividing a df by a series of ones, it should equal itself and not have any 0s in the output.

Installed Versions

Details

INSTALLED VERSIONS

commit : 9c8bc3e
python : 3.11.7
python-bits : 64
OS : Linux
OS-release : 5.14.0-503.40.1.el9_5.x86_64
Version : #1 SMP PREEMPT_DYNAMIC Wed Apr 30 17:38:54 UTC 2025
machine : x86_64
processor : x86_64
byteorder : little
LC_ALL : None
LANG : C.UTF-8
LOCALE : en_US.UTF-8

pandas : 2.3.3
numpy : 2.3.5
pytz : 2023.4
dateutil : 2.9.0.post0
pip : 25.0.1
Cython : 3.0.12
sphinx : None
IPython : 9.0.2
adbc-driver-postgresql: None
adbc-driver-sqlite : None
bs4 : 4.13.5
blosc : None
bottleneck : None
dataframe-api-compat : None
fastparquet : 2024.11.0
fsspec : 2025.3.2
html5lib : None
hypothesis : None
gcsfs : None
jinja2 : 3.1.6
lxml.etree : None
matplotlib : 3.10.1
numba : 0.63.0
numexpr : 2.10.2
odfpy : None
openpyxl : None
pandas_gbq : None
psycopg2 : None
pymysql : 1.4.6
pyarrow : 19.0.1
pyreadstat : None
pytest : 8.3.5
python-calamine : None
pyxlsb : None
s3fs : None
scipy : 1.15.3
sqlalchemy : 2.0.44
tables : 3.10.2
tabulate : 0.9.0
xarray : None
xlrd : None
xlsxwriter : None
zstandard : 0.23.0
tzdata : 2025.2
qtpy : None
pyqt5 : None

[rhshadrach]

Thanks for the report, however can not reproduce with the same version of NumPy on main. You mention "can get", how often do you see the issue appear? Nearly every run?

[jorisvandenbossche]

Given that it is triggered by a certain size, that might indicate it has to do with numexpr being used or not.

To rule that out, could you re-run the code on your end with setting pd.options.compute.use_numexpr to True vs False and see if that gives a difference?

[jorisvandenbossche]

(but also can't reproduce myself, on Linux (ubuntu) with those versions of numpy, pandas and numexpr)

[tilyevsky12]

Thank you for the responses. So I did try setting pd.options.compute.use_numexpr to False and that did the trick (was True by default). Also when it is set to True, sometimes my kernel crashes (running in a python notebook). When False, the results are correct and the kernel does not crash. Any ideas why this is causing the issue?

[vkmenon]

(but also can't reproduce myself, on Linux (ubuntu) with those versions of numpy, pandas and numexpr)

@jorisvandenbossche
the numpy version he specified was the downgrade that fixed the issue. i believe it is broken on numpy 2.3.5

[jorisvandenbossche]

Yes, that is the version that I tested with

[jorisvandenbossche]

@tilyevsky12 how did you install the python packages? If using numexpr results in crashes, then that might be some installation issue with that package.

[tilyevsky12]

I installed it with uv.

[tilyevsky12]

We ran some additional tests with varying numpy and numexpr version and made the following observations:

• numpy 2.2.4 + numexpr 2.10.2 → division works correctly
• numpy 2.3.5 + numexpr 2.10.2 → incorrect division results when pd.options.compute.use_numexpr = True
• numpy 2.3.5 + numexpr 2.14.1 → division works correctly again

[jorisvandenbossche]

With uv I can now also reproduce it (uv run --with pandas==2.3.3 --with numpy==2.3.5 --with numexpr==2.10.2 python and then run your script above).

And I remember now that I also ran into a correctness issue with numpy 2.3+ and older numexpr: pydata/numexpr#515

In the end this is a numexpr bug, so if you could boil it down to an example with just numpy and numexpr, you could report it there.
Now, given that numexpr 2.10 is already more than a year old and there have been more recent releases that do work with recent numpy, I am going to assume that the answer of the numexpr developers will be: ensure to use a newer numexpr when you also want a newer numpy (which is essentially also what is being said in pydata/numexpr#522)

Given the issues, though, I am wondering if we detect this version combo and in that case automatically switch to not using numexpr