The same is not true of something like free speech which is a natural right. My right to call the doctor an asshole doesn't interfere with his right to practice medicine freely. Consistency.