But what if that's the majority of miners and just breaking even miners? (You're probably right, but just want to be sure).
Difficulty, and with it profitability, of mining changes on every block. If some miners stop mining, difficulty and profitability will readjust for remaining miners. It self-adjusts.