Don't add IDREFs to set of missing IDs multiple times