The 66-year-old, who is from the Democratic Republic of Congo and also holds Swiss citizenship, had been appointed in March 2021.
His tenure was not without criticism, particularly for remaining in his role beyond the organisation's mandatory retirement age of 63.
He was also the subject of an internal investigation following staff complaints about a "toxic" work environment, though he was ultimately cleared of any wrongdoing.
Mosengo-Omba resigns
Mosengo-Omba's departure comes as the organisation grapples with the controversial aftermath of the 2025 Africa Cup of Nations (AFCON) final and the last-minute postponement of the women's tournament.
In a statement released on Sunday, Mosengo-Omba announced he was retiring. However, his exit is widely seen as being linked to a growing crisis of confidence within African football's governing body.
Mosengo-Omba, who served as deputy to CAF president Patrice Motsepe, addressed the controversies he faced during his time in office.
"Now that I have been able to dispel the suspicions that some people have gone to great lengths to cast on me, I can retire with peace of mind and without constraint, leaving the CAF more prosperous than ever," he wrote.
According to reports from the Reuters news agency, CAF's competitions director, Samson Adamu, is expected to step in as the acting general secretary.
The organisation is currently awaiting a crucial ruling from the Court of Arbitration for Sport (CAS) regarding Senegal's appeal against being stripped of the AFCON 2025 title.
Senegal is contesting a decision by CAF's appeals body, which overturned their 1-0 victory against hosts Morocco in the final.