Optimize PMCFG generation

[adelon]

Hi @krangelov, this is the PMCFG generation optimization we discussed in Göteborg.
It would be awesome if you could take a look and see if this looks reasonable to you.

Summary

This PR optimizes GF.Compile.GeneratePMCFG, particularly for parameter-heavy grammars.

I looked into profiling and optimizing this after running into memory pressure situations while using the German and Finnish RGL grammars in some GF experiments.

Previously, getFIds enumerated FId alternatives as boxed lists. Each PMCFG rule then stored its full argument alternatives in a Production. At the end, getPMCFG grouped the production set by (result FId, FunId) and applied the Cartesian-product compression.

This patch changes the intermediate data structures:

• FIdAlts stores alternatives in an unboxed array. They use dense 0-based bounds, so their derived Ord matches the previous list ordering of alternatives.
• FIdFactors stores weighted parameter-choice factors.
• PMCFGEnv stores productions already grouped by (FId, FunId).
• ProdGroup keeps the distinct argument products, per-argument unions, and areaSum.

This avoids constructing a large boxed Set Production and then rebuilding the same grouping at the end. Compression behaviour is still preserved.

The following are probably the most correctness-sensitive to review:

• fIdAltsFromFactors, which does the enumeration into an unboxed array, and replaces an enumeration via reverse (solutions (variants schema) ())
• ProdGroup finalization, which replaces the old grouping in getPMCFG

Measurements

I benchmarked compile/link runs for 10 RGL grammars (using GHC 9.6.7 and -O1). Finnish improved the most with 24x lower peak memory and 6.3x faster compilation. Simpler grammars like English are essentially unchanged.

Grammar	Before	Optimized	Change
Finnish	27.2s, 75 GB allocated, 15 GB peak	4.3s, 25 GB allocated, 631 MB peak	6.3x faster, 24x lower peak
German	33.7s, 186 GB allocated, 1.8 GB peak	8.8s, 46 GB allocated, 1.3 GB peak	3.8x faster, 1.4x lower peak
Russian	5.4s, 24 GB allocated, 492 MB peak	3.5s, 16 GB allocated, 484 MB peak	1.5x faster
French	3.2s, 19 GB allocated, 106 MB peak	2.8s, 17 GB allocated, 98 MB peak	1.1x faster
Hindi	1.9s, 8.6 GB allocated, 154 MB peak	1.8s, 7.6 GB allocated, 153 MB peak	1.1x faster
Spanish	1.5s, 9.2 GB allocated, 85 MB peak	1.4s, 8.6 GB allocated, 92 MB peak	1.1x faster
Amharic	237 ms, 1.1 GB allocated, 32 MB peak	227 ms, 1.0 GB allocated, 32 MB peak	essentially unchanged
Thai	113 ms, 509 MB allocated, 18 MB peak	110 ms, 508 MB allocated, 19 MB peak	essentially unchanged
English	290 ms, 1.4 GB allocated, 30 MB peak	287 ms, 1.3 GB allocated, 31 MB peak	essentially unchanged
Chinese	247 ms, 994 MB allocated, 37 MB peak	248 ms, 969 MB allocated, 37 MB peak	essentially unchanged

Testing

I tried to keep the changes as local as possible and preserve as much of the existing behaviour as I could.

The code builds for me with cabal build all under GHC 9.6.7, as well as stack build for the Stack configs from GHC 8.4.4 through GHC 9.6.7. The test suite also passes.

I have a separate "scratchpad" copy of GF where I tested/compared the optimized and original PMCFG generation with some local test hooks. I also compiled 46 RGL grammars with both the 3.12 release and the optimized version, and the resulting PGF files were identical.

It looks like there might be some edge cases where the current Cartesian-product compression is a bit too aggressive. But this was handled the same in the previous version. I will look at this in more detail and maybe open a separate issue/PR for that later.