feature-based .conllu format

[nschneid]

As proposed in #100, incorporate semantic annotations in MISC column as features. This way they can be incorporated into the official UD release.

[nschneid]

Note that the conversion script prints some warnings about simplification. These are generated in tagging.py and only refer to the expressions as reflected in the BIO tagging, which is omitted from the .conllu anyway. A synopsis of the 6 affected sentences:

Simplifying: removing weak group that interleaves with a strong gap:

reviews-037179-0002: this kebab shop is one of the best around the meat is good and fresh and the chilly sauce is the best , keep_ them$1 lovely kebabs _coming$1 and a happy_new_year to all the staff
reviews-159371-0001: ok but just becuse we where on_a_ tight$1 _budget$1 .
reviews-319816-0026: On Monday I called and again it was a_ big$1 _to_-_do$1 to find anyone who knew anything about it .
reviews-342807-0005: All you have_to do to make it authentic Jamaican food , is add a_ whole$1 _lot$1 of pepper .
reviews-374604-0002: If the animals are treated in the same way the customers are treated then this leaves_ a_lot$1 _to_be_desired$1 !


Simplifying: removing gappy group that is wholly contained within another gap:

reviews-359014-0005: I have_ nothing_but~ fantastic ~things _to_say .

[nschneid]

• Do we need a JSON converter/validator for the .conllu format? Can it be simpler than conllulex2json.py?
  • Validation should check the category values (MWECat, Supersense, ...) but there is less redundancy across .conllu features than there is across .conllulex columns, and thus less to check in terms of the format.
  • One new (?) thing to check for is that a word may be the start of at most one strong MWE and at most one weak MWE.
  • Is there any point in generating BIO tags from the .conllu? It's not clear that people would find them useful.

[nschneid]

Some of the necessary changes:

• Move deprecated data, docs, scripts to subdirectory
• Document new format in FORMAT.md
• syncud.py
• stats.sh
• release.sh
• pssid?
• json2conllu.py
• update imports to use conllu2json instead of conllulex2json
• tox
• Implement proper tests for scripts?