I'm not sure it *should* be secret. Likes aren't, why should the fact that you dislike be secret? If I block/mute someone, I don't care if they know.
In fact, you could decide that if 1 or more of those you follow blocks someone, they're blocked for you too. If downvoting happens, it could work same way. Not censored, but filtered.