The check that limited inference of actor isolation meant that we were
incorrectly computing actor isolation for (e.g.) overrides when
parsing from Swift interfaces. Only limit inference for cases where we
are guaranteed to synthesize an attribute by inference.