Well but hold on…what counts as “defensive” force? Is it simply the force necessary to stop your attacks from harming me, if I can manage it? Is it the force necessary to stop you from trying to harm me, or to remove the danger of continued attack? Can I shoot you dead if you’re trying to pinch me? And what about property rights? Are you justified in using physical force to defend your property against attack? On what grounds? And what if someone else is being attacked? Can you use force against the attacker? Is there any single objective answer to these questions?
What if I take your property and put it in my house? Can you use force against me or my property in order to get it back? If so, can you use only that force which is strictly necessary, or do you have a blank check on how much damage you can do? What if I take something that’s yours and swallow it? Can you cut me open?
If I harm you or your property, and refuse to compensate you for it, what then? Can you take my money or property as you see fit? Can you take it if someone else declares that it’s okay? Can someone else take it for you? How much, if any, force would you or your agents be justified in using to extract payment from me? If you could take my stuff, how would that be an act of self-defense?
(Obviously, don’t respond to each of those questions if you don’t think it’s necessary.) I just don’t think that the defense criterion is a sufficient account of why some kinds of force are justified in certain situations and not justified in others. We need to spell out what sorts of actions are acceptable responses to infringements of rights, and what actions are not acceptable. The fact of a rights infringment does not justify every use of force; at some point, force becomes excessive or arbitrary. Perhaps you can define “self-defense” in a way that captures all of these nuances, but you do actually need to do that in order for your theory to be plausible.
The other point I’d make is that I don’t think self-defense is the only thing that justifies the use of force, as I argued in the article I linked. So you’d have to show why I’m wrong about that.