We do not currently know of any antonyms for have the time of your life.
For similar words to have the time of your life, try: