The paradox disappears the moment, you handle tolerance differently.

Handle it like a social contract that everyone is automatically part of. All are tolerated until they stop tolerating people who are also part of the contract.

Whoever stops tolerating people who are part of the contract loses their “being tolerated privileges”.