I found the problem now… I didn’t set the glue records, because my registar’s interface doesn’t allow it (I thought I was setting glue records). I asked them to set the glue records manually, they did. Then I could set the nameservers. Very weird registar for not supporting glue records, but I am glad the issue is resolved!
Thank you @murgero!