CrossModuleOptimization: fix a bug in cycle-detection

The check to avoid infinite recursion in case of call graph cycles didn't work correctly.
It didn't result in crashes, because the function also has an additional max-depth check, but it could lead to exponential complexity in some cases.

Unfortunately I don't have a test case for this fix.
This commit is contained in:
Erik Eckstein
2022-09-27 16:46:06 +02:00
parent fc2d621f6f
commit f0b0d3faaf

View File

@@ -185,6 +185,10 @@ bool CrossModuleOptimization::canSerializeFunction(
if (iter != canSerializeFlags.end())
return iter->second;
// Temporarily set the flag to false (to avoid infinite recursion) until we set
// it to true at the end of this function.
canSerializeFlags[function] = false;
if (DeclContext *funcCtxt = function->getDeclContext()) {
if (!canUseFromInline(funcCtxt))
return false;