Small documentation change for Python Bindings

[sistemicorp]

Describe the bug
It was not clear how to build Python bindings from the instructions.

To Reproduce
Current Build instructions:

  mkdir bin
  cd bin
  cmake .. 
  cmake --build . --config Release

Additional context
Suggestion:

cd bindings/python
mkdir bin
cd bin
cmake .. 
cmake --build . --config Release

Has this been tested on Windows lately? I got the build to work, and Python bindings, and I am testing it with a STM32 target that I have, and I don't get any bytes in or out of it. I modified the test program slightly, added some prints,

>python main.py
Version: https://github.com/itas109/CSerialPort - V4.3.1.240204

Available Friendly Ports:
1 - COM3 STMicroelectronics serial
2 - COM4 STMicroelectronics Virtual COM Port
Please Input The Index Of Port(1 - 2)
2
Port Name: COM4
Open COM4 Success
send: len 16 - 000e03a1616668636d645f70696e6700
waiting...

OS Name Microsoft Windows 11 Pro
Version 10.0.22631 Build 22631

[itas109]

Thanks for your suggestions.
I will test it with ESP32.

[itas109]

I tested the master code of Python. It works fine.

Could you test the master code?

Open the following comments at python\CMakeLists.txt

# add_definitions(-DCSERIALPORT_DEBUG) # CSerialPort Debug Mode

This way, the application will output debug info to console and log.

CSerialPort_20240515.log

[INFO] itas109::CSerialPort::CSerialPort - OS: Windows, Arch: x86_64, Compiler: vs2022(1932), Bit: 64, C++: 201402L, Version: https://github.com/itas109/CSerialPort - V4.3.1.240204
[INFO] CSerialPortWinBase::openPort - portName: COM4, baudRate: 9600, dataBit: 8, parity: 0, stopBit: 0, flowControl: 0, mode: async, readBufferSize:4096(4096), readIntervalTimeoutMS: 0, minByteReadNotify: 1
[INFO] CSerialPortWinBase::openPort - open COM4. code: 0, message: success
[INFO] CSerialPortWinBase::writeData - write. len: 5, hex(top100): 3132333435
[INFO] CSerialPortWinBase::writeData - write. len: 7, hex(top100): 69746173313039
[INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 1). len: 1, hex(top100): 31
[INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 1
 [INFO] CSerialPortWinBase::readData - read. len: 1, hex(top100): 31

[sistemicorp]

Console Result,

>python main.py
2024-05-15 11:49:00.235 [INFO] itas109::CSerialPort::CSerialPort - OS: Windows, Arch: x86_64, Compiler: vs2022(1937), Bit: 64, C++: 201402L, Version: https://github.com/itas109/CSerialPort - V4.3.1.240204
Version: https://github.com/itas109/CSerialPort - V4.3.1.240204

Available Friendly Ports:
1 - COM3 STMicroelectronics serial
2 - COM4 STMicroelectronics Virtual COM Port
Please Input The Index Of Port(1 - 2)
2
Port Name: COM4
2024-05-15 11:49:01.802 [INFO] CSerialPortWinBase::openPort - portName: COM4, baudRate: 115200, dataBit: 8, parity: 0, stopBit: 0, flowControl: 0, mode: async, readBufferSize:4096(4096), readIntervalTimeoutMS: 0, minByteReadNotify: 1
2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::openPort - open COM4. code: 0, message: success
Open COM4 Success
send: len 16 - 000e03a1616668636d645f70696e6700
2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::writeData - write. len: 17, hex(top100): 000E03C2A1616668636D645F70696E6700
waiting...
.2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 16). len: 16, hex(top100): 0004E4011005FEFFFFFF010101010000
2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 16
.
(venv312) C:\git\a43\stm\stm32h750xbh\tools>

At the end, the program just came back to the prompt... I didn't stop it.
Here is the log file,

2024-05-15 11:49:00.235 [INFO] itas109::CSerialPort::CSerialPort - OS: Windows, Arch: x86_64, Compiler: vs2022(1937), Bit: 64, C++: 201402L, Version: https://github.com/itas109/CSerialPort - V4.3.1.240204
2024-05-15 11:49:01.802 [INFO] CSerialPortWinBase::openPort - portName: COM4, baudRate: 115200, dataBit: 8, parity: 0, stopBit: 0, flowControl: 0, mode: async, readBufferSize:4096(4096), readIntervalTimeoutMS: 0, minByteReadNotify: 1
2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::openPort - open COM4. code: 0, message: success
2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::writeData - write. len: 17, hex(top100): 000E03C2A1616668636D645F70696E6700
2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 16). len: 16, hex(top100): 0004E4011005FEFFFFFF010101010000
2024-05-15 11:49:01.809 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 16

My Python program,

#***************************************************************************
# @file main.py
# @author itas109 (itas109@qq.com) \n\n
# Blog : https://blog.csdn.net/itas109 \n
# Github : https://github.com/itas109 \n
# Gitee : https://gitee.com/itas109 \n
# QQ Group : 129518033
# @brief Python CSerialPort Example  Python的CSerialPort示例程序
############################################################################
import cserialport
import time

def main():
    sp = cserialport.CSerialPort()
    print("Version: %s\n" %(sp.getVersion()))

    listener = MyListener(sp).__disown__()

    spInfoVec = cserialport.CSerialPortInfo.availablePortInfos()
    print("Available Friendly Ports:")
    for index, spInfo in enumerate(spInfoVec, start=1):
        print("%d - %s %s" %(index, spInfo.portName, spInfo.description))

    if len(spInfoVec) == 0:
        print("No Valid Port")
    else:
        while True:
            print("Please Input The Index Of Port(1 - %d)" %(len(spInfoVec)))
            myInput=int(input())
            if myInput >= 1 and myInput <= len(spInfoVec):
                break
        portName=spInfoVec[myInput-1].portName
        print("Port Name: %s" %(portName))

        sp.init(portName,               # windows:COM1 Linux:/dev/ttyS0
                115200,                   # baudrate
                cserialport.ParityNone, # parity
                cserialport.DataBits8,  # data bit
                cserialport.StopOne,    # stop bit
                cserialport.FlowNone,   # flow
                4096                    # read buffer size
                )
        sp.setReadIntervalTimeout(0); # read interval timeout
        sp.open()
        print("Open %s %s" %(portName, "Success" if sp.isOpen() else "Failed"))

        # connect for read
        sp.connectReadEvent(listener)

        # write hex data
        d = '\x00\x0e\x03\xa1afhcmd_ping\x00'
        print(f'send: len {len(d)} - {"".join(f"{ord(c):02x}" for c in d)}')
        dl = len(d) + 1
        hex = cserialport.malloc_void(dl)
        cserialport.memmove(hex, d)
        sp.writeData(hex, dl)
        cserialport.free_void(hex)

        print("waiting...")
        while True:
            print(".", end="", flush=True)
            time.sleep(0.1)

def byte2hexStr(data):
    ch = ['0x%02X' % ord(i) for i in data]
    return " ".join(ch)

class MyListener(cserialport.CSerialPortListener):
    def __init__(self, sp):
        MyListener.countRead=0
        MyListener.sp=sp
        cserialport.CSerialPortListener.__init__(self)

    def onReadEvent(self, portName, readBufferLen):
        data = cserialport.malloc_void(readBufferLen)
        recLen=MyListener.sp.readData(data,readBufferLen)
        str=cserialport.cdata(data, readBufferLen)
        MyListener.countRead+=1
        print("%s - Count: %d, Length: %d, Str: %s, Hex: %s" %(portName, MyListener.countRead, recLen, str, byte2hexStr(str)))
        MyListener.sp.writeData(data, readBufferLen)
        cserialport.free_void(data)

if __name__ == '__main__':
	main()

[itas109]

SENT: 000e03a1616668636d645f70696e6700
LOG:  30303065303361313631363636383633

send is same with log.
send format is ascii data.
log format is hex data.
0x30 ....

[sistemicorp]

I fixed formatting so things match in the above log.
The sent string in the python program, the target doesn't get it.
I ran the script again, and the new log has,

2024-05-15 11:55:00.227 [INFO] itas109::CSerialPort::CSerialPort - OS: Windows, Arch: x86_64, Compiler: vs2022(1937), Bit: 64, C++: 201402L, Version: https://github.com/itas109/CSerialPort - V4.3.1.240204
2024-05-15 11:55:02.951 [INFO] CSerialPortWinBase::openPort - portName: COM4, baudRate: 115200, dataBit: 8, parity: 0, stopBit: 0, flowControl: 0, mode: async, readBufferSize:4096(4096), readIntervalTimeoutMS: 0, minByteReadNotify: 1
2024-05-15 11:55:02.958 [INFO] CSerialPortWinBase::openPort - open COM4. code: 0, message: success
2024-05-15 11:55:02.959 [INFO] CSerialPortWinBase::writeData - write. len: 17, hex(top100): 000E03C2A1616668636D645F70696E6700
2024-05-15 11:55:02.959 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 432). len: 432, hex(top100): 000438011001000000041C01101D40282329302E302D3338392D67643466613864612D6373657269616C01000002F80210010103FE01000000048007100100000004280F100A696C6F61645F7365740101010101010106A0941A7D42000000000004C403
2024-05-15 11:55:02.959 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 432

The data received, the 432 bytes, is a fragment from my target, probably from its boot up log.

Again the python program just ends, there is no error message from Windows.

[sistemicorp]

Could it be a Python version problem? I am using Python 3.12.3.
Or a Swig version problem? I used Swig 4.2.1, whereas your instructions have 4.1.1.
I can try Swig 4.1.1. UPDATE: made no difference.

[itas109]

My python version 3.10.

Could you test the receive function with VSPD(Virtual Serial Port Driver)?

In this way, the program is not affected by the hardware.

[sistemicorp]

Do you mean with CDC serial driver? I have been using pyserial for my application and it works with my target.

However I am sending a lot of data and running into a Python GIL problem to read the serial data fast enough before Windows buffers overflow. I am running more tests to try and truly understand the nature of my problem using pyserial. With WireShark I am able to see the serial data packets and frames and they are good. However on the PC, I get packet decode errors and I see that some data is just missing, and I see large values of pyserial.inWaiting(), like >50k bytes, and I have delays in reading serial of up to 20ms.

Your CSerialPort interests me because you create your own thread and buffer to read serial, and that means your read loop will be outside of the Python GIL, and should read serial port no matter what Python is doing.

I don't know much about swig and how it creates the Python bindings, but my feeling is thats where the problem is.

[itas109]

Could you use the C++ program CSerialPortDemoNoGui to test the receive problem?

If CSerialPortDemoNoGui also crash, we could use Visual Studio debug it for more information.

cd CSerialPort
mkdir bin && cd bin
cmake ..
cmake --build .
cd bin
./CSerialPortDemoNoGui

[sistemicorp]

That seems to work, specifically,

 Directory of C:\git\CSerialPort\bin\bin\Release

2024-05-16  10:13 AM    <DIR>          .
2024-05-14  12:48 PM    <DIR>          ..
2024-05-14  12:48 PM            61,440 CSerialPortDemoNoGui.exe
2024-05-16  10:13 AM        11,012,467 CSerialPort_20240516.log
2024-05-14  12:49 PM            44,544 libcserialport.dll

Running it the console fills up very fast, here is a snippet from the log,

2024-05-16 10:13:27.916 [INFO] itas109::CSerialPort::CSerialPort - OS: Windows, Arch: x86_64, Compiler: vs2022(1937), Bit: 64, C++: 201402L, Version: https://github.com/itas109/CSerialPort - V4.3.1.240204
2024-05-16 10:13:33.199 [INFO] CSerialPortWinBase::openPort - portName: COM4, baudRate: 9600, dataBit: 8, parity: 0, stopBit: 0, flowControl: 0, mode: async, readBufferSize:4096(4096), readIntervalTimeoutMS: 0, minByteReadNotify: 1
2024-05-16 10:13:33.206 [INFO] CSerialPortWinBase::openPort - open COM4. code: 0, message: success
2024-05-16 10:13:33.206 [INFO] CSerialPortWinBase::writeData - write. len: 5, hex(top100): 3132333435
2024-05-16 10:13:33.207 [INFO] CSerialPortWinBase::writeData - write. len: 7, hex(top100): 69746173313039
2024-05-16 10:13:34.960 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 24). len: 24, hex(top100): 00010102100668656C6C6F04618415020101010100000000
2024-05-16 10:13:34.960 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 24
2024-05-16 10:13:34.960 [INFO] CSerialPortWinBase::readData - read. len: 24, hex(top100): 00010102100668656C6C6F04618415020101010100000000
2024-05-16 10:13:34.961 [INFO] CSerialPortWinBase::writeData - write. len: 24, hex(top100): 00010102100668656C6C6F04618415020101010100000000
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 28). len: 28, hex(top100): 0004E4011005FEFFFFFF0101010100000004B8011001010101010000
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 28
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::readData - read. len: 28, hex(top100): 0004E4011005FEFFFFFF0101010100000004B8011001010101010000
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::writeData - write. len: 28, hex(top100): 0004E4011005FEFFFFFF0101010100000004B8011001010101010000
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 12). len: 12, hex(top100): 0004B8011001010101010000
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 12
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::readData - read. len: 12, hex(top100): 0004B8011001010101010000
2024-05-16 10:13:34.962 [INFO] CSerialPortWinBase::writeData - write. len: 12, hex(top100): 0004B8011001010101010000
2024-05-16 10:13:34.963 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 24). len: 24, hex(top100): 0004B80110010101010100000004B8011001010101010000
2024-05-16 10:13:34.963 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 24
2024-05-16 10:13:34.963 [INFO] CSerialPortWinBase::readData - read. len: 24, hex(top100): 0004B80110010101010100000004B8011001010101010000
2024-05-16 10:13:34.963 [INFO] CSerialPortWinBase::writeData - write. len: 24, hex(top100): 0004B80110010101010100000004B8011001010101010000
2024-05-16 10:13:34.963 [INFO] CSerialPortWinBase::commThreadMonitor - write buffer(usedLen 12). len: 12, hex(top100): 0004B8011001010101010000
2024-05-16 10:13:34.963 [INFO] CSerialPortWinBase::commThreadMonitor - onReadEvent. portName: COM4, readLen: 12

I didn't decode anything to see if this my expected data...
CSerialPort did not creash.

I do have a Python 3.10 virtual environment, but when I try to run main.py, I get,

(venv) C:\git\a43-P1150\stm\stm32h750xbh\tools>python main.py
Traceback (most recent call last):
  File "C:\git\a43-P1150\stm\stm32h750xbh\tools\main.py", line 10, in <module>
    import cserialport
  File "C:\git\a43-P1150\stm\stm32h750xbh\tools\cserialport.py", line 12, in <module>
    import _cserialport
ImportError: DLL load failed while importing _cserialport: The specified module could not be found.

I tried putting the DLL local in this directory but that didn't help. I rebuilt CSerialPort from within the Python 3.10 venv.

The Python 3.12 venv the import does succeed, but port doesn't work as previously reported above.

[itas109]

I test different version python.

windows 10
cmake 3.22.1
vs2022
swig 4.1.1, 4.2.1
python 3.12.3 crash, 3.10.11 work fine

[sistemicorp]

Thank you. The Python 3.12 result at least matches my result.
Re Python 3.10, I don't know why DLL fails to import. The CSerialPort Python bindings files are identical when I build them with Python 3.10 or 3.12, or with Swig 4.1.1 or 4.2.1.

I am in less of a hurry now, as I resolved a problem with pyserial so I am unblocked on my product.

But I am interested still in CSerialPort because I have some low level processing that I would like to move out of Python and into C++ and your project might be ideal for that.

Thank you so much for trying.
I will watch this project for any update will retry at that time.

[itas109]

Sorry for the long wait, the crash problem has been fixed.