We do not currently know of any antonyms for with flying colours.
The prepositional phrase with flying colours is defined as: