Don't autocorrect using capitalized suggestions

[eranl]

In shift state, suggestions are capitalized, but their scores are based on their non-capitalized versions. Using these capitalized suggestions for autocorrect is thus problematic.

In shift state, the typed word is also capitalized, and since the code I removed from InputLogic seems to treat the typed word as a "fallback autocorrect", that introduced a similar problem.

Fixes #2162.

[Helium314]

The issue was introduced in #1807.
While this PR seems to fix the problem, it's addressing the issue in a different place than what introduced the bug. I didn't investigate any further, but I fear this has a good chance of introducing unwanted changes in other places...
Do you see a good way of fixing this by adjusting the changes done in #1807?

Adjusted revert diff for #1807 to work with later changes:

show

diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyboardId.java b/app/src/main/java/helium314/keyboard/keyboard/KeyboardId.java
index 3c5de692..b55fb20c 100644
--- a/app/src/main/java/helium314/keyboard/keyboard/KeyboardId.java
+++ b/app/src/main/java/helium314/keyboard/keyboard/KeyboardId.java
@@ -313,8 +313,8 @@ public final class KeyboardId {
                 : EditorInfoCompatUtils.imeActionName(actionId);
     }
 
-    public int getKeyboardCapsMode() {
-        return switch (mElementId) {
+    public static int getKeyboardCapsMode(final int elementId) {
+        return switch (elementId) {
             case KeyboardId.ELEMENT_ALPHABET_SHIFT_LOCKED, KeyboardId.ELEMENT_ALPHABET_SHIFT_LOCK_SHIFTED ->
                 WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED;
             case KeyboardId.ELEMENT_ALPHABET_MANUAL_SHIFTED -> WordComposer.CAPS_MODE_MANUAL_SHIFTED;
diff --git a/app/src/main/java/helium314/keyboard/keyboard/KeyboardSwitcher.java b/app/src/main/java/helium314/keyboard/keyboard/KeyboardSwitcher.java
index eb29a0c6..940757a4 100644
--- a/app/src/main/java/helium314/keyboard/keyboard/KeyboardSwitcher.java
+++ b/app/src/main/java/helium314/keyboard/keyboard/KeyboardSwitcher.java
@@ -732,7 +732,7 @@ public final class KeyboardSwitcher implements KeyboardState.SwitchActions {
         if (keyboard == null) {
             return WordComposer.CAPS_MODE_OFF;
         }
-        return keyboard.mId.getKeyboardCapsMode();
+        return KeyboardId.getKeyboardCapsMode(keyboard.mId.mElementId);
     }
 
     public String getCurrentKeyboardScript() {
diff --git a/app/src/main/java/helium314/keyboard/latin/Suggest.kt b/app/src/main/java/helium314/keyboard/latin/Suggest.kt
index 98cb6174..fa455d39 100644
--- a/app/src/main/java/helium314/keyboard/latin/Suggest.kt
+++ b/app/src/main/java/helium314/keyboard/latin/Suggest.kt
@@ -24,7 +24,6 @@ import helium314.keyboard.latin.utils.AutoCorrectionUtils
 import helium314.keyboard.latin.utils.Log
 import helium314.keyboard.latin.utils.SuggestionResults
 import java.util.Locale
-import kotlin.math.min
 
 /**
  * This class loads a dictionary and provides a list of suggestions for a given sequence of
@@ -76,26 +75,23 @@ class Suggest(private val mDictionaryFacilitator: DictionaryFacilitator) {
                 settingsValuesForSuggestion, SESSION_ID_TYPING, inputStyleIfNotPrediction)
         val trailingSingleQuotesCount = StringUtils.getTrailingSingleQuotesCount(typedWordString)
         val suggestionsContainer = getTransformedSuggestedWordInfoList(wordComposer, suggestionResults,
-            trailingSingleQuotesCount, mDictionaryFacilitator.mainLocale, keyboard)
-        val keyboardShiftMode = keyboard.mId.keyboardCapsMode
-        val capitalizedTypedWord = capitalize(typedWordString, keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED,
-            keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFTED, mDictionaryFacilitator.mainLocale)
+            trailingSingleQuotesCount, mDictionaryFacilitator.mainLocale)
 
         // store the original SuggestedWordInfo for typed word, as it will be removed
         // we may want to re-add it in case auto-correction happens, so that the original word can at least be selected
-        val typedWordFirstOccurrenceWordInfo = suggestionsContainer.firstOrNull { it.mWord == capitalizedTypedWord }
-        val firstOccurrenceOfTypedWordInSuggestions = SuggestedWordInfo.removeDupsAndTypedWord(capitalizedTypedWord, suggestionsContainer)
+        val typedWordFirstOccurrenceWordInfo = suggestionsContainer.firstOrNull { it.mWord == typedWordString }
+        val firstOccurrenceOfTypedWordInSuggestions = SuggestedWordInfo.removeDupsAndTypedWord(typedWordString, suggestionsContainer)
         makeFirstTwoSuggestionsNonEmoji(suggestionsContainer)
 
         val (allowsToBeAutoCorrected, hasAutoCorrection) = shouldBeAutoCorrected(
             trailingSingleQuotesCount,
-            capitalizedTypedWord,
+            typedWordString,
             suggestionsContainer.firstOrNull(),
             {
                 val first = suggestionsContainer.firstOrNull() ?: suggestionResults.first()
                 val suggestions = getNextWordSuggestions(ngramContext, keyboard, inputStyleIfNotPrediction, settingsValuesForSuggestion)
                 val suggestionForFirstInContainer = suggestions.firstOrNull { it.mWord == first.word }
-                val suggestionForTypedWord = suggestions.firstOrNull { it.mWord == capitalizedTypedWord }
+                val suggestionForTypedWord = suggestions.firstOrNull { it.mWord == typedWordString }
                 suggestionForFirstInContainer to suggestionForTypedWord
             },
             isCorrectionEnabled,
@@ -104,14 +100,14 @@ class Suggest(private val mDictionaryFacilitator: DictionaryFacilitator) {
             firstOccurrenceOfTypedWordInSuggestions,
             typedWordFirstOccurrenceWordInfo
         )
-         val typedWordInfo = SuggestedWordInfo(capitalizedTypedWord, "", SuggestedWordInfo.MAX_SCORE,
+        val typedWordInfo = SuggestedWordInfo(typedWordString, "", SuggestedWordInfo.MAX_SCORE,
             SuggestedWordInfo.KIND_TYPED, typedWordFirstOccurrenceWordInfo?.mSourceDict ?: Dictionary.DICTIONARY_USER_TYPED,
             SuggestedWordInfo.NOT_AN_INDEX , SuggestedWordInfo.NOT_A_CONFIDENCE)
-        if (!TextUtils.isEmpty(capitalizedTypedWord)) {
+        if (!TextUtils.isEmpty(typedWordString)) {
             suggestionsContainer.add(0, typedWordInfo)
         }
         val suggestionsList = if (SuggestionStripView.DEBUG_SUGGESTIONS && suggestionsContainer.isNotEmpty())
-                getSuggestionsInfoListWithDebugInfo(capitalizedTypedWord, suggestionsContainer)
+                getSuggestionsInfoListWithDebugInfo(typedWordString, suggestionsContainer)
             else suggestionsContainer
 
         val inputStyle = if (resultsArePredictions) {
@@ -124,15 +120,14 @@ class Suggest(private val mDictionaryFacilitator: DictionaryFacilitator) {
         // If there is an incoming autocorrection, make sure typed word is shown, so user is able to override it.
         // Otherwise, if the relevant setting is enabled, show the typed word in the middle.
         val indexOfTypedWord = if (hasAutoCorrection) 2 else 1
-        if ((hasAutoCorrection || (Settings.getValues().mCenterSuggestionTextToEnter && !wordComposer.isResumed)
-                || capitalizedTypedWord != wordComposer.typedWord)
-            && suggestionsList.size >= indexOfTypedWord && !TextUtils.isEmpty(capitalizedTypedWord)) {
+        if ((hasAutoCorrection || (Settings.getValues().mCenterSuggestionTextToEnter && !wordComposer.isResumed))
+            && suggestionsList.size >= indexOfTypedWord && !TextUtils.isEmpty(typedWordString)) {
             if (typedWordFirstOccurrenceWordInfo != null) {
-                addDebugInfo(typedWordFirstOccurrenceWordInfo, capitalizedTypedWord)
+                addDebugInfo(typedWordFirstOccurrenceWordInfo, typedWordString)
                 suggestionsList.add(indexOfTypedWord, typedWordFirstOccurrenceWordInfo)
             } else {
                 suggestionsList.add(indexOfTypedWord,
-                    SuggestedWordInfo(capitalizedTypedWord, "", 0, SuggestedWordInfo.KIND_TYPED,
+                    SuggestedWordInfo(typedWordString, "", 0, SuggestedWordInfo.KIND_TYPED,
                         Dictionary.DICTIONARY_USER_TYPED, SuggestedWordInfo.NOT_AN_INDEX, SuggestedWordInfo.NOT_A_CONFIDENCE)
                 )
             }
@@ -275,18 +270,15 @@ class Suggest(private val mDictionaryFacilitator: DictionaryFacilitator) {
         val locale = mDictionaryFacilitator.mainLocale
         val suggestionsContainer = ArrayList(suggestionResults)
         val suggestionsCount = suggestionsContainer.size
-        val keyboardShiftMode = keyboard.mId.keyboardCapsMode
-        val shouldMakeSuggestionsOnlyFirstCharCapitalized = wordComposer.wasShiftedNoLock()
-            || keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFTED
-        val shouldMakeSuggestionsAllUpperCase = wordComposer.isAllUpperCase
-            || keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED
-        if (shouldMakeSuggestionsOnlyFirstCharCapitalized || shouldMakeSuggestionsAllUpperCase) {
+        val isFirstCharCapitalized = wordComposer.wasShiftedNoLock()
+        val isAllUpperCase = wordComposer.isAllUpperCase
+        if (isFirstCharCapitalized || isAllUpperCase) {
             for (i in 0 until suggestionsCount) {
                 val wordInfo = suggestionsContainer[i]
                 val wordLocale = wordInfo!!.mSourceDict.mLocale
                 val transformedWordInfo = getTransformedSuggestedWordInfo(
-                    wordInfo, wordLocale ?: locale, shouldMakeSuggestionsAllUpperCase,
-                    shouldMakeSuggestionsOnlyFirstCharCapitalized, 0
+                    wordInfo, wordLocale ?: locale, isAllUpperCase,
+                    isFirstCharCapitalized, 0
                 )
                 suggestionsContainer[i] = transformedWordInfo
             }
@@ -313,13 +305,4 @@ class Suggest(private val mDictionaryFacilitator: DictionaryFacilitator) {
             }
         }
 
-        val capitalizedTypedWord = capitalize(wordComposer.typedWord, keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED,
-            keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFTED, locale)
-        if (capitalizedTypedWord != wordComposer.typedWord && suggestionsContainer.drop(1).none { it.mWord == capitalizedTypedWord }) {
-            suggestionsContainer.add(min(1, suggestionsContainer.size),
-                SuggestedWordInfo(capitalizedTypedWord, "", 0, SuggestedWordInfo.KIND_TYPED,
-                    Dictionary.DICTIONARY_USER_TYPED, SuggestedWordInfo.NOT_AN_INDEX, SuggestedWordInfo.NOT_A_CONFIDENCE)
-            )
-        }
-
         useDefaultEmojiSkinTone(suggestionsContainer)
@@ -367,22 +350,19 @@ class Suggest(private val mDictionaryFacilitator: DictionaryFacilitator) {
 
         private fun getTransformedSuggestedWordInfoList(
             wordComposer: WordComposer, results: SuggestionResults,
-            trailingSingleQuotesCount: Int, defaultLocale: Locale, keyboard: Keyboard
+            trailingSingleQuotesCount: Int, defaultLocale: Locale
         ): ArrayList<SuggestedWordInfo> {
-            val keyboardShiftMode = keyboard.mId.keyboardCapsMode
             val shouldMakeSuggestionsAllUpperCase = wordComposer.isAllUpperCase && !wordComposer.isResumed
-                || keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFT_LOCKED
-            val shouldMakeSuggestionsOnlyFirstCharCapitalized = wordComposer.isOrWillBeOnlyFirstCharCapitalized
-                || keyboardShiftMode == WordComposer.CAPS_MODE_MANUAL_SHIFTED
+            val isOnlyFirstCharCapitalized = wordComposer.isOrWillBeOnlyFirstCharCapitalized
             val suggestionsContainer = ArrayList(results)
             val suggestionsCount = suggestionsContainer.size
-            if (shouldMakeSuggestionsOnlyFirstCharCapitalized || shouldMakeSuggestionsAllUpperCase || 0 != trailingSingleQuotesCount) {
+            if (isOnlyFirstCharCapitalized || shouldMakeSuggestionsAllUpperCase || 0 != trailingSingleQuotesCount) {
                 for (i in 0 until suggestionsCount) {
                     val wordInfo = suggestionsContainer[i]
                     val wordLocale = wordInfo.mSourceDict.mLocale
                     val transformedWordInfo = getTransformedSuggestedWordInfo(
                         wordInfo, wordLocale ?: defaultLocale,
-                        shouldMakeSuggestionsAllUpperCase, shouldMakeSuggestionsOnlyFirstCharCapitalized,
+                        shouldMakeSuggestionsAllUpperCase, isOnlyFirstCharCapitalized,
                         trailingSingleQuotesCount
                     )
                     suggestionsContainer[i] = transformedWordInfo
@@ -448,20 +428,27 @@ class Suggest(private val mDictionaryFacilitator: DictionaryFacilitator) {
 
         // public for testing
         fun getTransformedSuggestedWordInfo(
-            wordInfo: SuggestedWordInfo, locale: Locale, isAllUpperCase: Boolean,
+            wordInfo: SuggestedWordInfo?, locale: Locale?, isAllUpperCase: Boolean,
             isOnlyFirstCharCapitalized: Boolean, trailingSingleQuotesCount: Int
         ): SuggestedWordInfo {
-            var capitalizedWord = capitalize(wordInfo.mWord, isAllUpperCase, isOnlyFirstCharCapitalized, locale)
+            val sb = StringBuilder(wordInfo!!.mWord.length)
+            if (isAllUpperCase) {
+                sb.append(wordInfo.mWord.uppercase(locale!!))
+            } else if (isOnlyFirstCharCapitalized) {
+                sb.append(StringUtils.capitalizeFirstCodePoint(wordInfo.mWord, locale!!))
+            } else {
+                sb.append(wordInfo.mWord)
+            }
             // Appending quotes is here to help people quote words. However, it's not helpful
             // when they type words with quotes toward the end like "it's" or "didn't", where
             // it's more likely the user missed the last character (or didn't type it yet).
             val quotesToAppend = (trailingSingleQuotesCount
                     - if (-1 == wordInfo.mWord.indexOf(Constants.CODE_SINGLE_QUOTE.toChar())) 0 else 1)
             for (i in quotesToAppend - 1 downTo 0) {
-                capitalizedWord = "$capitalizedWord'"
+                sb.appendCodePoint(Constants.CODE_SINGLE_QUOTE)
             }
             return SuggestedWordInfo(
-                capitalizedWord, wordInfo.mPrevWordsContext,
+                sb.toString(), wordInfo.mPrevWordsContext,
                 wordInfo.mScore, wordInfo.mKindAndFlags,
                 wordInfo.mSourceDict, wordInfo.mIndexOfTouchPointOfSecondWord,
                 wordInfo.mAutoCommitFirstWordConfidence
diff --git a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java
index afa7162e..2f888f16 100644
--- a/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java
+++ b/app/src/main/java/helium314/keyboard/latin/inputlogic/InputLogic.java
@@ -683,7 +683,9 @@ public final class InputLogic {
                     break; // recapitalization and follow-up code should only trigger for alphabet shift, see #1256
                 performRecapitalization(inputTransaction.getSettingsValues());
                 inputTransaction.requireShiftUpdate(InputTransaction.SHIFT_UPDATE_NOW);
-                inputTransaction.setRequiresUpdateSuggestions();
+                if (mSuggestedWords.isPrediction()) {
+                    inputTransaction.setRequiresUpdateSuggestions();
+                }
                 if (mSpaceState == SpaceState.PHANTOM && inputTransaction.getSettingsValues().mShiftRemovesAutospace)
                     mSpaceState = SpaceState.NONE;
                 break;
@@ -813,13 +815,7 @@ public final class InputLogic {
                 // {@link KeyboardSwitcher#onEvent(Event)}, or {@link #onPressKey(int,int,boolean)} and {@link #onReleaseKey(int,boolean)}.
                 // We need to switch to the shortcut IME. This is handled by LatinIME since the
                 // input logic has no business with IME switching.
-            case KeyCode.EMOJI, KeyCode.TOGGLE_ONE_HANDED_MODE, KeyCode.SWITCH_ONE_HANDED_MODE:
-                break;
-            case KeyCode.CAPS_LOCK:
-                if (KeyboardSwitcher.getInstance().getKeyboard() == null
-                            || KeyboardSwitcher.getInstance().getKeyboard().mId.isAlphabetKeyboard()) {
-                    inputTransaction.setRequiresUpdateSuggestions();
-                }
+            case KeyCode.CAPS_LOCK, KeyCode.EMOJI, KeyCode.TOGGLE_ONE_HANDED_MODE, KeyCode.SWITCH_ONE_HANDED_MODE:
                 break;
             default:
                 if (KeyCode.INSTANCE.isModifier(keyCode))

[eranl]

For the suggestions part of this change , the other possible approach I can think of is modifying suggestion scores somehow when capitalizing them, but I don't know how.

For the typed word part, we could limit it to when the keyboard is shifted, but I don't understand the original reason why the typed word is considered a "fallback autocorrect", so I'm not sure about it.