Precisely yes.
Liberty is about respect for property rights. It is wrong to take from someone, but it is not wrong to take from no one in particular.
It was not wrong for the spanish to overthrow an empire that literally fed on its slaves in religious rituals and replace it with its much milder form of serfdom.