We do not currently know of any antonyms for as a matter of fact.
The prepositional phrase as a matter of fact is defined as: