Add retry logic for App CR updates to handle optimistic concurrency conflicts

[mdzhigarov]

Summary

This PR implements retry logic for App CR updates in the PackageInstall reconciler to handle optimistic concurrency control conflicts that can occur when multiple operations attempt to update the same App CR simultaneously.

Changes

Core Implementation

• Added updateAppWithRetry() function in pkg/packageinstall/packageinstall.go
  • Implements retry logic with up to 5 attempts for App CR updates
  • Re-fetches App CR after each failed update to get the latest resourceVersion
  • Handles NotFound errors immediately without retries to avoid unnecessary delays
  • Uses flexible updateFunc signature that returns an App, allowing direct integration with NewApp()

Integration Points

• Updated two App update locations in PackageInstall reconciler:
  • Line 233: App creation/update during normal reconciliation (uses NewApp() for transformation)
  • Line 408: App update during deletion reconciliation (applies specific field changes)

Testing

• Added comprehensive unit tests in pkg/packageinstall/packageinstall_test.go:
  • Test_UpdateAppWithRetry_HandlesConflictErrors: Tests retry logic with simulated conflict errors
  • Test_UpdateAppWithRetry_HandlesNotFoundError: Verifies immediate return for NotFound errors
  • Test_UpdateAppWithRetry_HandlesUpdateFunctionError: Tests error propagation from update function

Problem Solved

Fixes the "object has been modified; please apply your changes to the latest version and try again" error that occurs when:

• Multiple PackageInstall reconcilers attempt to update the same App CR
• App CR is updated by other controllers or operations between fetch and update
• High-frequency reconciliation triggers concurrent updates

Testing

All existing tests pass, and new tests specifically validate:

• ✅ Retry logic correctly handles optimistic concurrency conflicts
• ✅ NotFound errors are handled specially (no retries)
• ✅ Update function errors are properly propagated
• ✅ Retry mechanism works as designed (5 attempts max, re-fetch between attempts)

go test ./pkg/packageinstall -v
# All tests pass including new retry logic tests

Backward Compatibility

This change is fully backward compatible:

• No API changes
• No breaking changes to existing behavior
• Only adds resilience to existing update operations

[praveenrewar]

LGTM!

Questions