This case, in which one has ushered in Motzaei Shabbat but still wants to recognize Shabbat in Birkat HaMazon, tests the hierarchy of two competing halachic rules: 1) Once a person becomes obligated in one of the additions to Birkat HaMazon for special occasions, he recites it even after the occasion has passed. 2) When a point in time can relate to either of two consecutive periods, we do not allow an internal contradiction (tartei d’satrei) by relating the time to both. Let us survey sources and applications of each rule before coming to an answer to your question.
The first rule is actually not unanimously held. The Rosh[1] says that if one starts seuda shlishit before sunset but bentches[2] after the day is over, he does not recite R’tzei. In the opposite situation, if he ate a meal before Shabbat but recited Birkat HaMazon on Shabbat, the Rosh posits that he should recite R’tzei. The Tur[3] writes that according to the Rosh (his father), one who did not bentch for his Purim seuda until nightfall after Purim would not recite Al HaNisim. In other words, the time at which one is ready to make the pronouncement determines whether an addition to Birkat HaMazon is included.
However, we do not adopt the Rosh’s opinion as standard halacha. Rather, if the timing of one’s meal caused an obligation of a special addition, such as R’tzei or Al HaNisim, one should include it even when the occasion ostensibly is over.[4] (An exception to this is Sheva Berachot that finish after the week has been completed, as one cannot make independent berachot after their time period has passed.[5])
The next rule, that we do not allow tartei d’satrei situations, has far too many applications to address in this context, so we will stick close to our topic. One application concerns a person who started a seudat Purim on Erev Shabbat, continued it on Shabbat, and made Kiddush in the middle. When he eventually recites Birkat HaMazon, there is reason to say both Al HaNisim, due to the food that he ate during the day, and R’tzei, due to the food that he ate after Kiddush. The Chayei Adam[6] maintains that you cannot say both Al HaNisim and R’tzei in the same Birkat HaMazon when they relate to two different days. Thus, in the Purim case, one should say R’tzei, which is more important.
Another case of tartei d’satrei regarding R’tzei arises when one starts eating on Shabbat and continues eating bread after nightfall and Rosh Chodesh begins. There are grounds to say both R’tzei and Ya’aleh V’Yavoh in this case, but again, they belong to two different days. The Magen Avraham[7] asserts that one cannot say both, and since it is clear that there is more of an obligation of Ya’aleh V’Yavo, which applies at that moment, than there is an obligation of R’tzei, one says only Ya’aleh V’yavo. On the other hand, the Taz[8] is of the opinion that one may say both despite the apparent contradiction, whereas the Mishna Berura does not come to a clear conclusion.
There is some logic to say that tartei d’satrei comes into play only when one has to say two contradictory things in the same context, such as within Birkat HaMazon. If so, in your case, since there is nothing in Birkat HaMazon that indicates that it is already Motzaei Shabbat, saying R’tzei would not be a problem. However, the consensus of poskim is that davening Ma’ariv is such a strong indication of ending Shabbat that it is not possible to say R’tzei in bentching afterward.[9] Some major poskim[10] express uncertainty regarding whether reciting HaMavdil[11] without davening Ma’ariv precludes saying R’tzei.
In any case, the omission of R’tzei is not a critical matter because of the following ruling. Since not all agree that one has to eat bread at seuda shlishit, if one forgets R’tzei at sedua shlishit, he does not repeat Birkat HaMazon.[12]