That's why you need to lock your phone with a strong password and set a pin-code/fingerprint lock on your wallet. Most phones have built in encryption.
PSA: In the US it's legal to require fingerprints but illegal to make you give up you passcode.
Get a cheap #android phone which has the ability to remove the battery. Then put a piece of paper between the battery contacts & only remove it whenever you need the wallet.
That could work as an alternative hardware wallet for storing larger funds. However, for convenience I like to have a spending wallet on my primary phone for smaller purchases.