Clearly, the most important beracha of Havdala is the final one, HaMavdil, which speaks about contrasts, especially between Shabbat and the rest of the week. The berachot on besamim and on fire are not crucial obligations,[1] and while they are preferably attached to Havdala, they can be said separately as well.[2] Thus, if one did not hear those berachot, this does not affect his having fulfilled his Havdala obligation,[3] although he should make the berachot when he is able to over the course of the night.
In contrast, the cup of wine that one uses for Havdala is part and parcel of the mitzva on a Rabbinic level. Usually, by the time one recites Havdala on a cup of wine, he has already fulfilled the Torah-level mitzva of Havdala by reciting a form of Havdala during Ma’ariv.[4] The reason we recite Havdala again is specifically to recite it over a cup of wine. In your case, you heard the beracha of Havdala recited by someone who made it over a cup of wine, as he was supposed to. On the other hand, you yourself did not hear the beracha of Borei Pri HaGafen on that wine. Have you therefore not fulfilled the Rabbinic-level mitzva of reciting Havdala over wine?
The Magen Avraham[5] discusses the case of one who heard a complete Havdala but had in mind to include himself only in the beracha of HaMavdil but not in that of Borei Pri HaGafen. He rules that such a person did fulfill the mitzva of Havdala; he simply may not drink the Havdala wine without making a new beracha. The Mishna Berura[6] explains that HaMavdil, recited over wine, is the essential beracha of Havdala, whereas Borei Pri HaGafen is necessary only to enable one to drink the wine.
Several Acharonim[7] demonstrate the extent of this distinction’s cogency by citing a halacha with regard to the parallel mitzva of Kiddush, which is also a declaration about Shabbat’s special status and which is similarly recited over a cup of wine. The Shulchan Aruch[8] posits that if one was in the midst of a meal in which he had already made a beracha on wine when Shabbat began, he should make Kiddush without reciting Borei Pri HaGafen. We see from this halacha that reciting the blessing of Borei Pri HaGafen is not an integral part of the Kiddush. Similarly, if one recites Havdala during seuda shlishit when one had already made a beracha on wine during the meal, there are two opinions in the Shulchan Aruch[9] regarding whether he should recite Borei Pri HaGafen in Havdala, and the accepted opinion is that he does not need to.[10] Furthermore, the reason for another Borei Pri HaGafen during Havdala, according to the opinion that requires it, is that Havdala cannot be considered part of the meal, and the beracha made during the meal is therefore no longer effective at that point.[11] There is no concern raised that one may possibly not fulfill the obligation of Havdala made over wine if he fails to recite the beracha of Borei Pri HaGafen during Havdala. Your case, in which Borei Pri HaGafen was recited but you did not hear it, is no worse.
There is even a question regarding whether reciting Borei Pri HaGafen is crucial for the one who makes Havdala when not in the midst of the meal, who clearly should be saying it. Based on the rule mentioned above that the beracha is important only to allow one to drink the wine, Rav Moshe Feinstein posits[12] that if one mistakenly recited Shehakol on Havdala wine, he fulfills the obligation of Havdala, as he is permitted to drink the wine based on the beracha of Shehakol. He further proposes that even if one forgot to make any beracha at all on the Havdala wine and he has already drunk it,[13] he has fulfilled the mitzva of Havdala. Certainly, then, if the person who needed the beracha for his own drinking did indeed recite it, the fact that one of the listeners missed it is not a problem
Shemirat Shabbat K’Hilchata[14] cites Rav S.Z Auerbach as ruling that since someone must drink the wine and he needs the beracha of Borei Pri HaGafen to be able to do so, the beracha is indeed a part of Havdala that listeners must take seriously. However, if one missed hearing that beracha, Shemirat Shabbat K’Hilchata[15] rules that b’di’eved, there is no need to hear Havdala again, and he cites several poskim who agree with this ruling. This applies both to one who came in after Borei Pri HaGafen was said, as well as to one who did not hear it or concentrate on it. In your case, while you would have needed a beracha before drinking the wine,[16] there was, of course, no need for you drink.