We do not currently know of any antonyms for swearing like a pagan.
For similar words to swearing like a pagan, try: