We do not currently know of any antonyms for come to the party.
The verb come to the party is defined as:
See full definition of the word come to the party.