Good point. I'm not soo cryptography-savvy, so I wasn't sure if just from somebody's public address (isn't it a HASH of the public key?) you could encrypt messages to them? Can you?
So every memo account has corresponding public key already exposed in the blockchain. You can fetch any transaction and extract public key from it. And than do ECDH with it.