I think the problem with something like that is the danger that these keys will be taken away, and used by people other than their owners. You would need something to verify that the original owner still has possession of their key and is not under coercion at the time of the vote. (coercion not at the time of the vote could be dealt with by strong anonymous voting, which prevents the other person from verifying which way the key was voted).
But this is really going back to the proof of unique human debate.