PyPI binary wheels with IBM MQ C redistributables

[adiroiban]

Thanks for working on this project.
I am happy to see an IBM MQ Python project on GitHub with the GitHub Issues enabled.

I read the Design Notes documentation, but I am not sure why not use GitHub Action to build and publish wheel packages for Linux, Windows and MacOS .

I am working on a fork of pymqi with the goal of creating a self-contained wheel package.
That is the pymqi C extension + IBM MQ redistributables

You can see the work in progress here https://github.com/chevah/pymqi/blob/1.12.11-chevah/README.md#linux

I didn't had time to read the full IBM MQ C redistributables licence... but since they are "redistributables" I guess that you are allow to distribute.

---

I am happy to migrate my code to the mq-mqi-python and send a pull request.

Are you happy to accept a PR that makes it easier to build self-contained wheel packages ?
At least for Windows and Linux.

Thanks

[ibmmqmet]

Thanks. I've taken a look at this.

I think that this repo already has many of the pieces to make it easier to build "private" binary wheels. For example, the pyReleash.sh script can build it, even though it doesn't upload it to the central PyPI server. But it can work with private servers (see testServer,sh), if you uncomment the appropriate upload line. And the setup.py script did get substantial rework to make it easier to use the MQ_FILE_PATH reference.

Putting a binary wheel on PyPI is certainly possible, but the various version/compatibility constraints and lack-of-guarantees made me reluctant. At least for now. At worst, you could end up with a wheel per platform per version of MQ per version of Python ... which gets unmaintainable. I've not looked at the current "multliinux" specification to verify it does work with MQ, but there's a danger that it might get too old as a baseline for future versions of MQ if OS prereq levels change.

So the binary upload is something that I feel belongs more in the domain of private servers for a particular enterprise, which has more control of the deployed versions of Python and MQ. One (of the many) change I made to the original C code was to restrict it to the Limited API. That does at least give some degree of forward compatibility with newer versions of Python. So that's part of why the pyRelease script exists.

I do not like the idea of embedding a copy of the MQ client code, even if the license allows it. It bloats the package enormously. Rather, my preferred way of doing it would be to "automatically" download the client at an appropriate time. In the equivalent Node.js package, we do it as part of the postinstall processing inside npm install. There's also a sample here of doing it in a container for an application deployment. Look at the Dockerfile in code/examples for how that can be done. It would be relatively simple to change that to be a "build" container instead of a "runtime" environment.

And a related idea - again already part of the Node.js MQ packaging - would mean that compiling the C extension might not even need the MQ C client library at build time. It would be required only at runtime.

Using GitHub Actions is something else I'd look at in the future, but might be overkill given the likely frequency of its being needed.

As this is a new package though, I'm ready to see what feedback comes in. And then see what can improve both the code and the processes.

[adiroiban]

Thanks for the info.

With pymqi being maintained by Zato, I can understand why binary wheels are not available ... since Zato prefer end users to get the IBM MQ library via their services.
And I guess that Zato have now moved away from open source development.

My understanding is that this new repo is supported and maintained by IBM... without any SLAs and contracts ... but still, the main developer effort is paid directly by IBM

Putting a binary wheel on PyPI is certainly possible, but the various version/compatibility constraints and lack-of-guarantees made me reluctant.

I am new to IBM MQ and I don't know how IBM C client compatibility works.
Also I am not very committed to the IBM MQ platfrom.
But this was the biggers battle I had to fight to get Python working with IBM MQ.

A wheel binary that only has the mq-mqi-python C extension compiled it not very helpful to new developers.
It might be ok for devs that are already familiar with IBM MQ product...but when you are new to the game and a junior Python developer having to compile the project youself, then create an IBM account to download the IBM MQ C distributables then putting it all together... can be very complicated... especially since IBM MQ C error reporting is minimal without any hints of what went wrong.

I wheel binary that is distributed with the IBM MQ C distributables shared libararies is what is needed for easy onboarding.

I can imagine having a "thin" version and a "thick" version.

---

For me GitHub Action is critical. It can be used to demonstrate how to use all the scripts and can be used to make sure that the scripts continue to work... and to make sure that the code from a branch works with an actual IBM MQ server.

For my production project, using GitHub action to run automated tests is critical.

With a wheel package that if self contained and the IBM MQ server container it's easy to setup automated tests that can run end to end tests to make sure that the final product works.

---

This is just some feedback, from a new independent developer.

I am working on a legacy project that is tied to IBM MQ... and comparing with the experince that I had when first working with Python Pika and RabbitMQ.

In the end, I think that this is up to IBM to decide if they want to make it easy for new
developers to build projects with IBM MQ... or IBM onlt wants to deal with "corporate" customers that will arrange separate consultancy contract...similar to what we have with Zato.

The fact that we have the IBM MQ server container, is a huge step forward.

Next, having a Python library that just works, would be perfect :)

Thanks for your work on this!

[adiroiban]

I was thinking at this overnight... I think that it is worth trying to publish binary wheels , without the IBM MQ C client distributables..

There is cibuildwheel which makes it easy to build the wheels for all the platforms available via GitHub Actions ...

Here is a somehow simple example to build on windows + Linux, and then publish to PyPI Trusted Publishers mechanism.

I can help with a PR for this

---

Regarding the IBM MQ C redistributables wrapped in a wheel, this can be a separate 3rd party package/repo
The thing is that Python Packaging already handles versionsion, caching, upgrade, downgrade... etc... so if you your project already uses pip/uv it's convenient to use a wheel package to distribute versioned OS specific binaries ... otherwise , you end up reinventing the wheel

[adiroiban]

Another option is to do static library link with the Python C extension... but I don't think that this is feasable.

It looks like the IBM MQ C redistributables don't allow static linking :(

---

This is what Python Cryptography project uses to include the OpenSSL library together with the Python C/Rust extension ... all delivered in a single Python wheel package.

[adiroiban]

As a proof of concept I have created a separate Python package to help distribute the IBM C libraries

The code is here
https://github.com/chevah/mq-mqi-python/pull/1/files#diff-b5cec2a04d07c31889f8b99ab4f8866d44f16bb41251d71ed9759a965ee8d6e3

The idea is to help distribute only files that are really needed at runtime.

For example, for Linux the IBM MQ C redistributables package is 86 MB (archived), and the wheel is only 15MB