We do not currently know of any antonyms for man of influence.
For similar words to man of influence, try: