It’s only irrational if the black person is nonviolent. But how is the walker supposed to know that? With limited information to go on and a belief, whether valid or not, that many blacks engage in criminal activity, it is perfectly rational (in the way you mean it) to exert a bit more energy to avoid the risk of being mugged. The DC or NY cab driver that refuses to pick up a black person is also not being irrational. Yes, he is forgoing cab fare by avoiding a certain part of the populace, but he does so because he calculates the risk to be greater than the potential benefits. Perhaps he is miscalculating and, in truth, he would come out with a net gain, but he certainly can’t know that ex ante. He is most definitely stereotyping, but stereotypes provide a fantastic service: they facilitate action in situations in which the actor has little information and no reasonable opportunity to gather the knowledge necessary to make a more informed decision.
Two points I want to bring up, which may or may not be understood already, but I feel like writing it. First, you said, “I think human actions reflect racism.” Like Sieben, I think this is a largely pointless statement. That some people are racist means little in a political discussion. If a bunch of people are racists, then the society is going to be racist, regardless of the state; and as for the origin and potential cure of racist sentiments, political economy cannot help us. It can help us determine under which institutions racism can survive or thrive. It is clear that in the free market, in which private property is respected and actors must bear losses along with their profits, there exists an incentive to regard others based upon their productivity and not their race or sex. It is clear that in the state, in which the legislator is largely immune to losses, there is no such incentive (though, of course, there may be other incentives, such as the legislator’s personal disgust toward racism). Indeed, in democratic governments, there exists an incentive to enact racist legislation, so as to gain the favor of the racist vote. With a state that claims the authority to regulate commerce, levy special taxes and fines, enact zoning regulations, etc., racists free-ride off the state. With a benign or nonexistent state, racists must bear the full economic costs of their racism.
Second point: Though an individual libertarian may be racist, the libertarian argument against political prohibition of nonviolent racial discrimination is not meant to imply that such discrimination is acceptable. This is why I think Tim Wise and many others (including libertarians) mischaracterize libertarian doctrine when they say that libertarians believe private racism ought to be “allowed.” The libertarian, acting as a libertarian, only says that such discrimination is not aggression, and as only aggression ought to be punishable, the law ought to have nothing to say about such discrimination. There is no moral or legal right to use violence against a person unwilling to hire or sell to or buy from a black person; this does not mean the libertarian necessarily applauds such action. The racist, acting in this capacity, is not committing aggression against anyone, and so ought not to be subject to legal repercussions.
To the unbiased reader, it should be obvious from this statement that libertarians may still believe racism in any form is wrong and that it is virtuous to combat racism wherever it exists. But this combat must be nonviolent. Boycotts, sit-ins, persuasion, publicizing the names of racists, etc. The kind of activism Americans tend to associate with MLK and the civil rights movement, and which was quite effective against the state’s Jim Crow laws.