We do not currently know of any antonyms for have a joke with.
For similar words to have a joke with, try: