That’s a good question because it gets to the heart of the scope of morally acceptable retaliatory action within both a Randian and Rothbardian ethical framework.
First though, I am not sure what you mean by Rand’s treating the nation as a homogenious lump here. We can’t just look at that section in a vacuum, but in its entirety as a chapter within a book here, it’s clear that she doesn’t treat groups of individuals as entities in themselves. That kind of was the entire point of that particular chapter on collectivized “rights.” Earlier in the chapter to the quoted portion, she states:
Since only an individual man can possess rights, the expression “individual rights” is a redundancy (which one has to use for purposes of clarification in today’s intellectual chaos). But the expression “collective rights” is a contradiction in terms. Any group or “collective,” large or small, is only a number of individuals. A group can have no rights other than the rights of its individual members.
In a free society, the “rights” of any group are derived from the rights of its members through their voluntary, individual choice and contractual agreement, and are merely the application of these individual rights to a specific undertaking. Every legitimate group undertaking is based on the participants’ right of free association and free trade.
Further she says:
The notion of “collective rights” (the notion that rights belong to groups, not to individuals) means that “rights” belong to some men, but not to others—that some men have the “right” to dispose of others in any manner they please—and that the criterion of such privileged position consists of numerical superiority. Nothing can ever justify or validate such a doctrine—and no one ever has.
Like the altruist morality from which it is derived, this doctrine rests on mysticism: either on the old-fashioned mysticism of faith in supernatural edicts, like “The Divine Right of Kings”—or on the social mystique of modern collectivists who see society as a super-organism, as some supernatural entity apart from and superior to the sum of its individual members.
The amorality of that collectivist mystique is particularly obvious today in the issue of national rights. A nation, like any other group, is only a number of individuals and can have no rights other than the rights of its individual citizens.
I don’t know what you mean by Rand not distinguishing between “free” and “not-free” because she states in the same section:
A free nation… recognizes, respects and protects the individual rights of its citizens… The government of such a nation is not the ruler, but the servant or agent of its citizens and has no rights other than the rights delegated to it by the citizens for a specific, delimited task (the task of protecting them from physical force, derived from their right of self-defense).
In other words, freer nations respect rights to a greater extent. Dictatorships however, are nations that don’t recognize this principle, and wield unlimited power over its citizens, she goes on to list four essential characteristics of a dictatorship.
As for your scenario, it would seem that in a Randian framework, if the socialist country refused to respect the property rights of the capitalist country’s citizens, that is, refused to listen to the capitalist country’s complaints and kept the expropriated oil fields and equipment, etc. then the capitalist country is justified in retaliatory force in order to attain justice and remove the threat of the socialist country to its citizens. In this framework, that which will keep Americans free should be the standard of the free nation’s politics, which in foreign policy requires only that the government use retaliatory force against external aggressors as the means to which maintaining the security and liberty of American citizens is the end. If having a socialist government take property makes some Americans less free, then retaliation is in order.
Rand argued that the West was gravely mistaken to abandon the property rights of Western oil companies in the Middle East, and that it should have asserted these rights and protected them against nationalizations by local governments. Peikoff made a similar argument in a NYT article “End States Who Sponsor Terrorism” saying:
Fifty years ago, Truman and Eisenhower surrendered the West’s property rights in oil, although that oil rightfully belonged to those in the West whose science, technology, and capital made its discovery and use possible. The first country to nationalize Western oil, in 1951, was Iran. The rest, observing our frightened silence, hurried to grab their piece of the newly available loot.
As for an anarcho-capitalist defense agency, the problem is similar. Because the agency is funded voluntarily and staffed by voluntarily employed soldiers, it doesn’t make whatever the actions this agency takes automatically right. It still needs to be defined when retaliation can be proper for this agency, as well as the Randian type of government, where that force is equally voluntarily funded and staffed, and those who want no part of it can effectively secede by ignoring it.
There is still the question of non-combatants in the socialist nation’s territory. In the Rothbardian framework, it is illegitimate to cause any collateral damage whatsoever, so your question about one’s moral code allowing him to get pushed around is legitimate. Without appeal to a philosophical basis in ethics, we have no answer to the question of whether or not this amounts to self-sacrifice. Suppose one is being aggressed against and the only way he can defend himself would happen to hurt a non-combatants in the enemy’s territory, is it legitimate to do so? Answering this is necessary for practical guidance to an individual’s right of self-defense to both anarcho-capitalist and Objectivist cases. It is clear in the Randian framework, if the actions undertaken by a government in the name of self-defense are not legitimate for an individual to undertake in self-defense, then those actions must be condemned as morally improper. But at the same time, if it is morally licit for an individual to take an action in defense of himself, then it is morally licit for the collective use of force by a group of them, since they have only the rights of their individual members.