We do not currently know of any antonyms for having a rapport with one.
For similar words to having a rapport with one, try: