But if you are a malicious actor, and have a butt ton of money to spend without skipping a meal or losing your yacht, you can buy the power to vote everyone out. That's unfair.
You said yourself, a rich person could just buy up a server farm and attack. This actually happened to Bitcoin Gold which forked to an ASIC resistant PoW algo.
That'd be like if Jeff Bezos bought half of Congress and suddenly gained the ability to pass whatever legislation he wanted because he owns the majority voting power.
That's exactly why we need ASICS actually. If you want to affect the network you gotta buy them and contribute to the hashing power. ASIC resistance wrecks that incentive structure.