[sgen] Use non-synchronized gray stack when only one worker is running