Is there any way I can only allow RSS feed for known topic numbers at least? I would rather not disable it totally.

Hidden categories are not shown into RSS unless you are logged in with your username and password.

Try to check this with a browser where you are not logged in.

Ahh! Thank you!

I did just that (checked other browsers) and you are correct.
I noticed this on Chrome, I would log out and still see posts that I should not. That and I think also with RSS aggregation on other websites it would show those I think.

I will try all that again and mark this as solved for now then.