Yes exactly. If a big enough percentage of the population is immoral your "freedom" is going to be a walled compound with you working your ass off to grow your own food. Freedom baby!
You have to consider the costs of regulating this morality. Current government makes it easy to be in another persons business. Many regulations created (eg pronouns).