I get that you’re trying to do a reductio ad absurdum but your post really falls flat on its face.
The Roman Empire no longer exists in any shape or form.
The United States of America is a surviving institution that has continuity to its darker history as it pertains to slavery.
The problem is not simply that slavery is causing some sort of moral taint; it’s that there are still institutions today that continue to have advantages that stem from the unjust practice of slavery in the US.
I just refuse to order off Uber Eats or Doordash. Does that count as not punishing the drivers?