The mishna[2] states that one can bentch only until the food is digested. The gemara[3] offers two opinions for the definition of this cut-off point. R. Yochanan says that it is when one becomes hungry again, whereas Reish Lakish says that it does not occur as long as one is still thirsty from having eaten or for 72 minutes, depending on how much he ate. We rule according to R. Yochanan and assume that the mishna refers to when one begins to become hungry as the food is being digested.[4] However, most poskim assume that R. Yochanan accepts a 72-minute minimum time limit; he simply maintains that if one has not yet begun to become hungry, he can bentch even after 72 minutes have passed.[5]
The problem is that a feeling of hunger is hard to quantify or determine with certainty.[6] Therefore, one should be careful to bentch no later than 72 minutes after having finished eating. You did not succeed in doing so in the case that you describe. Unless you were certain that you were either still full (should bentch) or already hungry (too late), you entered the realm of safek (doubt).
Ostensibly, your desire to continue eating and then bentch was the halachic preference. In order to make sure this would have worked, the continuation of your meal would have had to include at least a k’zayit[7] of bread.[8] If it did, then Birkat HaMazon would be of value either way. If you were still full enough from the earlier eating before continuing, then Birkat HaMazon would have covered both parts of the meal. If you were not full enough, then while you would not have addressed the first part of the meal, at least Birkat HaMazon would not have been l’vatala,[9] as it was called for by the post-Ma’ariv eating.[10]
However, the idea of eating more raises another question. If a break is actually long enough for feelings of hunger to begin, making it too late for a beracha acharona,[11] is it necessary to make a new beracha rishona[12] before eating again? The Magen Avraham[13] assumes that once the time for a beracha acharona has passed, a new beracha is necessary because the previous eating is a matter of the past. However, the Even HaOzer[14] argues that there is no source to indicate that digestion breaks the validity of a beracha rishona. On the contrary, the Rambam[15] asserts that a beracha that one makes when he begins eating can cover other foods “even if he breaks all day long,” provided that he has not decided to stop eating.
Although there are attempts to deflect the Even HaOzer’s proofs,[16] and some poskim agree with the Magen Avraham, the Even HaOzer’s opinion is the more accepted one.[17] Therefore, you would not have needed to make a beracha in order to have continued eating. Furthermore, in a case like yours, where there is doubt whether digestion occurred, even the Magen Avraham[18] allows eating more without a new beracha, and indeed suggests it in order to avoid the doubt regarding Birkat HaMazon. Apparently, then, it is preferable to be in a situation in which one might need to say HaMotzi but refrain out of doubt than to forego a Birkat HaMazon that he might still be able to recite.[19]
Therefore, eating more bread (without a beracha) would have been the best way for you to have salvaged bentching.[20] In the event that there was no more bread available, if you already had a satiating meal and you thought it was likely that you were still considered full, you could have bentched even though there was some doubt about it.[21]