Fix Python SDK detection and virtual environment management

[meanmail]

Overview

Python SDK detection has ongoing issues evidenced by multiple recent commits. SDK is not reliably found when reopening projects, virtual environment creation has platform-specific quirks, and there are multiple competing sources of truth for SDK state.

Related Commits

• e66f7a1c07 - Fix Python test detection and remove excessive logging
• 4ca9f740a2 - Improve Python SDK handling and dependency installation
• af2d80e0da - Add clickable link to install dependencies when tests fail to run

The presence of diagnostic logging throughout the code suggests complex debugging efforts.

Problem Areas

1. SDK Not Found on Project Reopen

When reopening a project, the SDK is sometimes not detected even though it exists in jdk.table.xml. The plugin needs to check multiple sources:

• Module SDK settings
• Project SDK settings
• jdk.table.xml (IntelliJ's SDK registry)

2. Virtual Environment Creation Issues

Platform-specific APIs with BACKCOMPAT workarounds. Code at:

• intellij-plugin/hs-Python/branches/252/src/com/jetbrains/edu/python/learning/compatibilityUtils.kt
• Different behavior between platform 252 and 253

3. Requirements Detection

requirements.txt detection sometimes fails even when file exists. Related to VFS refresh timing issues.

4. SDK Dropdown Filtering

PyLanguageSettings.kt SDK dropdown shows all Python versions instead of filtering by course requirements. Recent fix improved this but may have edge cases.

5. Multiple Sources of Truth

SDK state tracked in:

• PyEnvironmentChecker.kt:16-43
• PyEduUtils.kt
• PyLanguageSettings.kt
• Project/module settings

Key Files

• intellij-plugin/hs-Python/src/org/hyperskill/academy/python/learning/checker/PyEnvironmentChecker.kt
• intellij-plugin/hs-Python/src/org/hyperskill/academy/python/learning/PyEduUtils.kt
• intellij-plugin/hs-Python/src/org/hyperskill/academy/python/learning/newproject/PyLanguageSettings.kt
• intellij-plugin/hs-Python/src/org/hyperskill/academy/python/learning/newproject/PyCourseProjectGenerator.kt
• intellij-plugin/hs-Python/branches/252/src/com/jetbrains/edu/python/learning/compatibilityUtils.kt

What Makes This Hard

1. Python plugin integration: Must work with JetBrains Python plugin's internal APIs which are complex and underdocumented
2. Platform version differences: SDK APIs changed significantly between 252 and 253
3. State persistence: SDK must survive project close/reopen cycle reliably
4. Multiple entry points: SDK needed at project creation, reopening, and checking
5. Virtual environment lifecycle: Creation, validation, requirement installation all have failure modes
6. Error handling: Requirements installation can fail due to network, missing packages, version conflicts, etc.
7. User experience: Must provide clear error messages and recovery paths (like the clickable "Install Dependencies" link)

Requirements

• Redesign SDK state management to have a single source of truth
• Implement reliable SDK detection on project reopen:
  • Check jdk.table.xml properly
  • Check module SDK settings
  • Fall back to auto-detection if needed
• Fix virtual environment creation to work reliably on both platform versions 252 and 253
• Improve requirement installation error handling:
  • Network failures
  • Missing packages
  • Version conflicts
  • Permission issues
• Remove diagnostic logging once issues are resolved (or convert to proper debug logging)
• Add integration tests with real Python SDK (at least one version)
• Handle edge cases:
  • No Python installed on system
  • Corrupted virtual environment
  • Network failures during pip install
  • Multiple Python versions available
• Ensure "Install Dependencies" workaround action integrates properly

Success Criteria

• SDK reliably detected on project reopen (test by closing and reopening 10 times)
• Virtual environment creation succeeds on both platform versions
• Clear error messages for all failure modes
• No excessive logging in production code
• Integration tests pass with real Python SDK
• User can recover from any error state

Estimated Time

6-7 hours (includes investigation, state management redesign, platform compatibility, testing)