Carrying a commanding three-goal advantage into the second leg, Forest began positively, with Lorenzo Lucca’s header after 15 minutes offering some early encouragement.
However, the visitors soon displayed their ability midway through the first half when an incisive breakaway saw Sidiki Cherif’s pass evade significant contact from the retreating Neco Williams, allowing Kerem Akturkoglu to tuck home the game’s opener.
The hosts looked for a riposte through Elliot Anderson’s close-range effort, but it was Fenerbahce who were the more threatening as Aktürkoğlu swept a first-time shot wide shortly before flicking a neat pass to Cherif, who fired straight at Stefan Ortega.
Boos greeted the home side as they headed down the tunnel at the break, and Vitor Pereira was evidently unimpressed as he made four alterations ahead of the second period. However, just three minutes after the restart, Akturkoglu was celebrating his second of the evening by tucking home a penalty after being upended by Jair Cunha.
Fenerbahce had never progressed from a European tie after losing the first leg, yet with plenty of time left on the clock, momentum was with the Turkish side in their 27th such attempt.
Forest had been a shadow of the side that put Fener to the sword in Turkey, and they needed a moment of quality to re-establish some breathing room in the tie.
That class came from Callum Hudson-Odoi, who collected Ola Aina’s cross in the box before working some space to calmly slot his sixth goal of the season into the bottom corner.
With 22 minutes to play, the Premier League side were back in charge of the contest, going close again through a bending Hudson-Odoi strike and an Igor Jesus one-on-one opportunity, but ultimately doing enough to see out the tie.
The result marks just the second time Fenerbahçe have ever won a European away match against an English side (D2, L7), but that will be of little consolation as a disappointing first-leg defeat was their undoing.
As for Forest, a missed opportunity to claim a third successive win in a UEFA competition for the first time since 1983/84 is disappointing, especially given the lacklustre display.
Nonetheless, progression was the ultimate goal, and they will now face either Real Betis Balompié or FC Midtjylland in the last 16 as they look to claim a first European trophy since 1979/80.
Flashscore Man of the Match: Kerem Akturkoglu (Fenerbahce)