The 160 bit hash is just an extra layer to protect against a potential flaw in secp256k1. When you transact, you reveal the public key anyway (that's why we have change addresses).