Aggression means a lot of things to a lot of different people, even among libertarians. Pro-IP libertarians think of violating intellectual property as aggression. Some left-libertarians think of exclusive ownership of land as aggression. The debates over valid and invalid definitions of “aggression” go round and round, usually devolving into each side just dismissing the other as “false-libertarians”. It’s all byplay.
The heart of the matter is to figure out the most efficacious legal/moral order to establish with regard to effective rights for human purposes. Then if you want to define aggression as a violation of those most efficacious rights, then fine. I believe intellectual property is destructive to the market order, and thus to human welfare, as I discuss here. And abolishing land ownership would also be destructive to the market order, and thus to human welfare, as I discuss here.