WARNING: Important dl;dr response below:
Freedom of Speech is not merely, or even primarily, for the government to protect -- although that promise of protection is part of nearly all deomocratic societies' charters.
No matter how much we dislike our opponents or despise their messages, it is our duty, and one that is very dangerous to ignore, that we campaign for freedom of speech when it's threatened (and our duty depends not at all on whose speech is in danger).
The rape apologists, the pedos who have the temerity to discuss their grotesque anomaly openly and in a complimentary fashion, the TERFs, the Neo-Nazis, the deranged, the abhorrent, the most depraved and disgusting things to crawl out of their holes in Guam just long enough to write thinly veiled porn tracts -- whereas they must be punished, and some quite severely, for any actual crimes they are found guilty of having committed; it is immeasurably important that they have the right to speak their minds and to publish their deranged raving in any forum (online, on TV, in radio, in news papers, etc.) that will permit it.
Some of the reasons this unpleasant duty must be carried out -- i.e., to defend the right of each creep to speak his or her mind -- are as follows:
a) Public opinion is a chimera, and sometimes its malleable nature can be manipulated -- who, in 2014/15, for example, saw Trump coming? -- in such a way that our own opinions could at some point be considered a threat to public order, In such cases as these, the government breaks its contract with the People, in service to some rotten ideology. NO! FUCK NO! Government exists for one reason only, and that is to serve all the People in its jurisdiction (often by attempting to balance the will of a majority with the human and civil rights of minorities while maintaining a common infrastructure using money from the treasury - from the common account).
b) We cannot expect anyone to do for us what we ourselves will not do for others.
c) In places where the citizens also refuse to defend whatever speech the government fails to protect -- I'm looking at you, Uganda! -- then such abrogation by government is often followed by violence, pogroms, purges. And on an individual level, it means death threats, and sometimes actual murder.
Free Speech:
It is the foundation on which every other freedom ever devised must necessarily rest: No one can organise protests without it. No one can openly worship -- or not -- according to the light of personal conscience without it. No one can question official narratives without it.
No large media outfit could freely operate; they'd be handed a brief of what allegedly happened and the official account would be the only acceptable one.
The OP demonstrates the necessity of defending freedom by answering this question - "....what other freedoms racists should lose in addition to not being allowed in any restaurants..." - with THIS: "Their liberty....[and] should perverts be permitted to air their views?"
Denounce bad speech.
Refute bad speech.
Protect bad speech. That is the only way to protect good speech.