So every memo account has corresponding public key already exposed in the blockchain. You can fetch any transaction and extract public key from it. And than do ECDH with it.