Fix memory leak when invalidating NativeRendering fonts

When NativeRendering is used, we keep a reference to all the font
engines currently in use to make sure they stay alive, and then
dereference them when the scenegraph is invalidated, typically when
the window closes.

There was always a controlled leak here: If you loaded new font assets
continuously, then we would retain the old ones in memory. Due to
the bug in QTBUG-100697, we would keep them even if they were
application fonts that were removed.

However, when QTBUG-100697 was fixed, a side effect was that the memory
leak became more visible: It no longer only happens when loading new
fonts, but just loading and unloading the same two application fonts
and setting them on a text item in a loop would cause us to
continuously create new font engines, give them a font cache and then add
them to the m_fontEnginesToClean list.

The fix is to match the registerFontEngineToClean() with an unregister
function and then clean up the font engines that no longer have any
references during the synchronization step, similar to how we clean
up distance field caches that are no longer in use.

Note that this removes a bogus qDeleteAll() from the software backend:
This was added blindly by f1b188df132c42da62197055725e5f7eebcc4249.
Since the set will be empty, it doesn't cause a crash, but is not the
correct way to delete font engines, so to avoid future confusion or
cargo-culting, we just replace it with an assert that the set is
empty.

[ChangeLog][Text] Fixed a controlled memory leak with Text.NativeRendering
where loading and unloading the same application fonts could cause memory
usage to increase indefinitely and not be regained until the window was
closed.

Pick-to: 6.5 6.6
Fixes: QTBUG-113714
Change-Id: I34c60e647bf63a0d203f752066f1cbfaeb049bcf
Reviewed-by: Qt CI Bot <[email protected]>
Reviewed-by: Mårten Nordheim <[email protected]>
Reviewed-by: Andy Nichols <[email protected]>

1 files changed, 8 insertions, 0 deletions

diff --git a/src/quick/scenegraph/qsgdefaultglyphnode_p.cpp b/src/quick/scenegraph/qsgdefaultglyphnode_p.cpp
index e9e6c26d88..f10dbd4bc6 100644
--- a/src/quick/scenegraph/qsgdefaultglyphnode_p.cpp
+++ b/src/quick/scenegraph/qsgdefaultglyphnode_p.cpp
@@ -322,6 +322,8 @@ QSGTextMaskMaterial::QSGTextMaskMaterial(QSGRenderContext *rc, const QVector4D &
 
 QSGTextMaskMaterial::~QSGTextMaskMaterial()
 {
+    if (m_retainedFontEngine != nullptr)
+        m_rc->unregisterFontengineForCleanup(m_retainedFontEngine);
     delete m_texture;
 }
 
@@ -385,6 +387,12 @@ void QSGTextMaskMaterial::updateCache(QFontEngine::GlyphFormat glyphFormat)
         if (!m_glyphCache || int(m_glyphCache->glyphFormat()) != glyphFormat) {
             m_glyphCache = new QSGRhiTextureGlyphCache(m_rc, glyphFormat, glyphCacheTransform, color);
             fontEngine->setGlyphCache(cacheKey, m_glyphCache.data());
+            if (m_retainedFontEngine != nullptr)
+                m_rc->unregisterFontengineForCleanup(m_retainedFontEngine);
+
+            // Note: This is reference counted by the render context, so it will stay alive until
+            // we release that reference
+            m_retainedFontEngine = fontEngine;
             m_rc->registerFontengineForCleanup(fontEngine);
         }
     }