Fails on *+*

[mansayk]

Hello!

Streamparser always fails while trying to parse this output of apertium's tagger:
'^*+*/*+*$'

Original string is '*+*'.

Traceback (most recent call last):
File "apertium2fastmorph.py", line 137, in parse_apertium
pos = mainpos(r)
File "streamparser/streamparser.py", line 90, in mainpos
return reading[-1].tags[0] # type: ignore
IndexError: list index out of range

I parsed the entire corpus consisting of hundreds of millions tokens, and this is the only part where it fails. If I change *+* to +* or *+ or ** or *** it works.

It seems that it is a bug.

[unhammer]

Hm, mainpos should probably have a fallback argument.

A temporary workaround is to check that there's actually a reading there.

But that analysis looks weird; shouldn't the stars be escaped so it isn't ambiguous with unknowns?

[mansayk]

I already use this kind of checking. That part of the code:
for r in lexicalUnit.readings:
lemma = r[0].baseform
lemma = lemma.lower()
if r[0].tags:
pos = mainpos(r)

Stars should be escaped by me or by tagger?

[unhammer]

don't you also need to check that r is nonempty? (readings consist of subreadings, which contain tags and baseforms)

[mansayk]

I tried this checking and it also doesn't work:
for r in lexicalUnit.readings:
lemma = r[0].baseform
lemma = lemma.lower()
if r and r[0].tags:
pos = mainpos(r)
for s in r:
if tags == "":

And some debugging output:
LINE:^**/**$
R:[SReading(baseform='**', tags=[])]
LINE:^+*/+*$
R:[SReading(baseform='', tags=[]), SReading(baseform='*', tags=[])]
LINE:^*+/*+$
R:[SReading(baseform='', tags=['*'])]
LINE:^*+*/*+*$
R:[SReading(baseform='', tags=['*']), SReading(baseform='*', tags=[])]

Error occurs only in the last line.

Is it a bug or my fault?

[unhammer]

Well, ^*+*/*+*$ isn't valid apertium stream format, if the raw input form was *+* then the reading should be ^\*\+\*/\*\+\*$. So if lt-proc gave ^*+*/*+*$, then I think this bug should go in https://github.com/apertium/lttoolbox/issues :-/