search with attributes and child elements of entrys

[petermr]

flexible search

At present py4ami only uses a hardcoded set of fields in entry s. This allows us to adjust for:

• abbreviations
• synonyms
• language variants
• stemming

[petermr]

problem.

Currently only the @term of an entry is used, e.g:

<dictionary>
  <entry term="foo" name="bar">
  </entry>
 </dictionary>

will search with foo.

In the abb (abbreviation dictionaries) dictionaries from docanalysis (e.g. https://github.com/petermr/semanticClimate/blob/main/ipcc/ar6/wg3/Chapter03/dict/ip_3_3_longmitig_abb.xml )
we have:

<dictionary title="abb_chapter03">
  
  <metadata>
    <edit date="2022-12-04" editor="Sunag">Automatic creation from docanalysis</edit>
  </metadata>
  
  <entry name="IMPs" term="Illustrative Mitigation Pathways" wikidataID=""/>
  <entry name="SR1.5" term="Special Report on Global Warming of 1.5C" wikidataID=""/>
  <entry name="SDGs" term="Sustainable  Development  Goals" wikidataID=""/>
  <entry name="IPs" term="Illustrative Pathways" wikidataID=""/>
  <entry name="IAMs" term="Integrated  Assessment  Models" wikidataID=""/>
  <entry name="GWL" term="global  warming levels" wikidataID=""/>
  <entry name="BECCS" term="bio-energy-and-carbon-capture-and-storage" wikidataID=""/>

Where the abbreviation is in the name. (This should be rewritten, but it would be useful to use the @name for the search.

py4ami HTML --dict abb_chapter03.xml --annotate --inpath chapter03.html --outpath  chapter03_annot.html --color YELLOW

This will only use "Illustrative Mitigation Pathways", etc. We want "IMPs" (and maybe also "IMP" (stemming))

would like:

py4ami HTML 
--dict abb_chapter03.xml 
--annotate 
--fields '@name'
--stem
--inpath chapter03.html 
--outpath  chapter03_annot.html 
--color YELLOW
  

[petermr]

test-driven code:

current arguments

Note dictionary has format where @name contains the full term and @term contains the abbreviation. (This will change).

<dictionary title="abbrev_as">
  <entry term="APC" name="and Developing Pacific" count="2" wikidataID=""></entry>
  <entry term="ATB" name="Annual Technoloy Baseline" count="2" wikidataID=""></entry>
  <entry term="ATR" name="auto-thermal reforming" count="1" wikidataID=""></entry>
  <entry term="BECCS" name="be produced through biomass gasification with CCS" count="33" wikidataID=""></entry>
  <entry term="BEVs" name="battery electric vehicles" count="2" wikidataID=""></entry>

see py4ami test.test_html.HtmlTest.test_parse_commandline

    def test_parse_commandline(self):
        """
        Simulate running from commandline
        """
        infile = Path(Resources.IPCC_CHAP06, "fulltext.html")
        assert infile.exists()
        outpath = Path(Resources.IPCC_TEMP_CHAP06, 'fulltext.annot.html')
        dictfile = Path(Resources.IPCC_CHAP06, 'abbrev_as.xml')
        assert dictfile.exists()
        args = f"HTML --inpath {infile} --outpath {outpath} --annotate --dict {dictfile} --color YELLOW"
        PyAMI().run_command(args)

Manual inspection shows roughly satisfactory annotation. We have adjusted the dictionary to give 3 hits with Wikidata IDs

        # crude count of success
        xpath ="//*[@href]"
        href_elems = element.xpath(xpath)
        assert len(href_elems) == 3, f"expected 3 hrefs, found {len(href_elems)}"