Normally, the answer "yes" is possible for a yes-or-no question - either intended or not. "Is it Monday or Tuesday?" Can be answered with "yes", and indicates it is either of those, but not Wednesday (etc). "Are you American or not?" can likewise be answered "yes", which indicates that you are , indeed, either an American, or not (but not some potential third choice).
However, "is 4 less than 5, or more than 3?" can't properly be answered with "yes", since it is not one of those - it is both. Similarly, your answer indicates that you are talking either about yourself, or your Sim, but not something else entirely or both. I know the joke, and I was trying to build on it by doing the Thing a bit.