lz4: unbloc compressed stream using cython code when input data are on the cpu#4601
lz4: unbloc compressed stream using cython code when input data are on the cpu#4601kif wants to merge 13 commits into
Conversation
|
With this patch, azimuthal integration as described in https://pyfai.readthedocs.io/en/stable/usage/tutorial/Parallelization/GPU-decompression.html goes from 1.76ms for processing an Eiger4M images to 1.09ms, on the same computer. |
|
There is likely still a bug somewhere ... OpenCL get randomly killed. |
| ## This is for developing: | ||
| ##cython: profile=True, warn.undeclared=True, warn.unused=True, warn.unused_result=False, warn.unused_arg=True | ||
|
|
||
| """This module provides :func:`_unblock_lz4` which scans a dataset for different blocks. |
There was a problem hiding this comment.
Why not putting unblock_lz4.pyx directly in silx.opencl.codec?
Regarding naming, it should be a private module _unblock_l4.pyx providing a public function unblock_lz4.
There was a problem hiding this comment.
I tend to dislike mixing python and cython code into the same directory, same for headers and c-code ... especially in the development phase where .c and .html files are likely to be generated in place and could end up polluting the repo.
There was a problem hiding this comment.
For the naming, it is correct that the module should be private.
| return result | ||
|
|
||
|
|
||
| def _unblock_lz4(bytes src): |
There was a problem hiding this comment.
this is more unlock_bitshuffle_lz4 rather than bare LZ4 code block
There was a problem hiding this comment.
Indeed, it looks to be specific to bitshuffle-LZ4
2 participants
<Module or Topic>: <Action> <Summary>(see contributing guidelines)PR summary
AI Disclosure
Close #4598
The performance variation: was 1.2ms, down to 650µs for a 4M Eiger image.