Well, I sort of agree with your position. At least I agree with the idea that if A doesn’t think the property of B,C,etc is to be respected, A can’t object to his property being treated equally.
But I imagine that in practice people would make explicit arrangements as to how they want their property and persons protected…and ‘pacifist batman’ would not necessarily be a welcomed guest.
Why should the opinion of a bystander be relevant to a dispute, assuming the quarreling parties did not ask his opinion ?