The past tense of rejoice with is rejoiced with.
The third-person singular simple present indicative form of rejoice with is rejoices with.
The present participle of rejoice with is rejoicing with.
The past participle of rejoice with is rejoiced with.