I did not get through the entire opinion, but don’t think it says what you think it says.

The question presented is whether the President, acting through the Secretary of State, has authority to revoke a passport on the ground that the holder’s activities in foreign countries are causing or are likely to cause serious damage to the national security or foreign policy of the United States.

The court was not asked to consider weather passports could be revoked on other grounds.