The difference is entirely a quantitative subjective judgement: How likely is this person to harm me?

There's no _hard, ethically relevant distinction_ between the situations. Law can and does distinguish between them, but that's because law is in the business of drawing arbitrary lines in the sand (as it must be).