Python Bindings

[jianshu93]

Hi simd-minimizer team,

I am wondering whether it is possible to also add python bindings for the crate. Many of the LLM models for genomes are written in Python and it is highly interesting to use minimizers to train the model.

Thanks,
Jianshu

[imartayan]

Hi @jianshu93,
Great suggestion! I think this should be fairly straightforward using PyO3, I'll let @RagnarGrootKoerkamp have a look since he has more experience than me with it.

[RagnarGrootKoerkamp]

What part of the API should we expose as python?
Something like this could be sufficient maybe:

import simd_minimizers as sm
seq = b"ACTGCAC"
packed = sm.pack(seq)
positions = sm.{canonical}_minimizer_positions(packed, k, w)
(minimizer_positions, super_kmer_positions) =
    sm.{canonical}_minimizer_and_superkmer_positions(packed, k, w)
values = sm.{canonical}_minimizer_values(packed, k, positions)

Would that work for you?

In the rust version we also do

• ascii and ascii-encoded-DNA inputs
• scalar versions
• mul-hash versions
  but I think non of those are particularly important.

Should we also expose one_{canonical}_minimizer?

[RagnarGrootKoerkamp]

@jianshu93 does the above looks good for your usecase?

One thing I kinda worder about: typically LLM models are kinda slow, right? so in that case, is minimizer computation really the bottleneck?

[kokyriakidis]

@RagnarGrootKoerkamp Is it possible to have some C++ bindings? I am working on the development of a genome assembler and I would really want to incorporate it.

[jianshu93]

Hi @RagnarGrootKoerkamp, sorry for the late response, transformer models are faster but require both the positional information and minimizer extracted. Yeah the training is slow but we want to be fast whenever possible, we are going to training using 2 million genomes, this step can be slow if we are using the naive randomized algorithms. --Jianshu

[jianshu93]

I think all I need is the position of the minimizer and the 32-bit hash. Your proposed functions actually looks good. Let me know if it is ready. Great thanks for this. I think this will be building block for the models we are developing. --Jianshu