As with Rome, the limitation is often communication and transportation.

You’d have a hard time even keeping Mars part of unified empire with Earth given our current technology level. We simply can’t move things back and forth easily enough until we figure out fusion reactors (or some other power source) to a much higher level than we currently have.

Any sort of empire spanning more than a single solar system would require faster than light travel and communications.