Yes exactly. If a big enough percentage of the population is immoral your "freedom" is going to be a walled compound with you working your ass off to grow your own food. Freedom baby!
If you think about all value/energy flows in society, the more you divert to regulating morality the less there is for moving society forward.