Okay, I honestly think you need to chill out a bit about your mom. I know this advice may be unwelcome, so if so, please just remember that I’m some idiot on the Internet, and ignore me. 
First of all, you do the libertarian movement no good by fighting with your mother. It just looks bad. Dude, respect your momma!
Second of all, as a libertarian you should believe in property rights first and foremost. Her home, her rules. You don’t like it, emancipate yourself. This is seriously important, and goes beyond just parental/child relations. You are living in her house. If you don’t want to live in a gun free zone, don’t live there, or at the very least, stick to respectful, peaceful persuasion to try to change the property owner’s mind. I have to say that obviously you believe the benefits of living there outweigh the cons. Of course, those benefits may be that you are young enough you don’t have any other choice and would face serious consequences if you left. But it’s the truth: right now, you’ve obviously judged that living in a gun free zone is your preference.
Finally, libertarian views on the subject of guns is not “pro gun,” per se. It’s pro gun rights. Really, it’s pro property rights. We believe that everyone should be permitted to make their own decisions about property, including what guns they will or will not own or use, provided they do not infringe on anyone else’s rights. That means if somebody wants their property to be a gun free zone, that’s their choice. They will probably be outcompeted by the market. But the market will make that decision; it’s not one that a central planner could make. (i.e., everybody should host guns on their property, which seems to be your position).
And really, there are a range of libertarian opinions about violence and self-defense. The core on which we all agree (or else we would not be libertarians), is the non-aggression principle: we do not initiate aggression against other people. Therefore, someone with a gun shouldn’t have it confiscated. (Responding to aggression in a violent way is of course not initiating aggression.) Beyond that, people have different opinions. There are some libertarians who are complete pacifists and would never defend themselves! Some would fight in wars, some would not. The pacifist libertarians would of course not deny anybody else their right to self-defense. But they just wouldn’t exercise it themselves for various reasons. Some believe that violent problems can never be solved with violence. Some, like myself, are under personal religious convictions on the subject. Some libertarians would use force in defense only under certain limited situations. Some might just believe that it’s a waste of resources; that the risk involved is not that high. You should know by know that value is subjective; risk is also appraised subjectively. Some people are always going to appraise risk low enough that they do not want to be armed. You’ll find some people who want to pass laws to arm everybody; as a libertarian, you should strenuously oppose these as a violation of liberty.
Your mother has the right not to be armed. She has a reason not to be armed; it may be good or it may be bad; chances are that it is subjective. You appraise the situation differently from you. That doesn’t mean she does not practice libertarianism! Not every libertarian is packing heat. Yes, it is fun to joke about believing we have the right to shoot someone for not respecting the “keep off the grass” sign on our own property. I do it, too, when I know people will realize I’m joking. (And as I said, I’m a pacifist.) But owning a gun is not the core of practicing libertarianism. Respecting the rights of other people is. And you need to start by respecting your mother’s right to dictate terms on her own property, even if those terms are not pleasing to you. Don’t be her central planner.
As for your question, there are existing libertarian theories of justice which you can check out. The one I’m most familiar with is found in Dr. Walter Block’s Radical Austrianism, Radical Libertarianism seminar, which you can listen to in mises.org’s media section. I believe the theory Block presented would allow for force to be used to apprehend someone guilty of a crime; I’m not sure how much force, though. Definitely check out what he has to say.
Rothbard would say (and did say in For a New Liberty, I believe), that it is always wrong to use force against someone not actually convicted of a crime. But he didn’t mean convicted by a state law court. I would tend to say that in the case of witnessing a thief, you have personally convicted them and could use force against them. But think it on through: if you’ve caught a thief with no proof that he was breaking and entering (i.e., no other witnesses), then after apprehending him you still couldn’t obtain justice in a real independent court. In such a case, I would say that you would be unable to publicly justify your use of force and would be taking a risk of the thief being able to obtain action against you! So possibly the maximum justice that can really be obtained is that you drive the thief off of your property, assuming he hasn’t stolen or broken anything. In that case, I wouldn’t think shooting him or apprehending him would be advisable.
Free market justice doesn’t just consider obtaining justice as an abstract goal; it considers the cost and quantity of justice to be obtained.