We do not currently know of any antonyms for have a whale of a time.
The verb have a whale of a time is defined as: