We do not currently know of any synonyms for I won't go with you.
For similar words to I won't go with you, try: