The sources lead us to many ideas, but none that we can prove.
The Shulchan Aruch[2] deals with ways to fulfill the requirement that a m’lo lugmav (enough to fill the cheeks – approximately 2 fl.oz) of the Kiddush wine must be drunk. Some say that one person has to drink the whole amount; others say we can add up the amounts that different people drink. The Shulchan Aruch points out that no matter how the requirement is met, it is best for everyone to drink some of the wine. This recommendation[3] that everyone should drink is based on the Rosh,[4] who does not offer a source or an explanation. The Rosh also does not mention if this is a special mitzva regarding Kiddush, although that is the context of the gemara upon which he is commenting.
The Shulchan Aruch is understood as maintaining that it is sufficient for each person to drink a small amount,[5] but if this results in not having enough wine for at least one person to drink a m’lo lugmav or for the next day’s Kiddush, the idea of everyone drinking is waived.[6] On the other hand, giving out some wine to everyone is important enough that the mekadesh[7] may delay between the beracha and the drinking in order to pour for them.[8]
The Rambam[9] writes that after drinking a m’lo lugmav, one “gives to all the members of the group to drink.” The Mirkevet HaMishneh[10] searches for a Talmudic source for the Rambam (who rarely includes a halacha that lacks one). He points to the gemara in Berachot,[11] which lists things that one is supposed to do to enhance a kos shel beracha (cup of wine used in a mitzva context). R. Avahu mentions ten things and then adds that some say to “send it to the members of one’s household.” R.Yochanan says that only four of the practices need to be observed. The Mirekevet HaMishneh reasons that R. Yochanan argued about the number (four rather than ten) but did not take issue with the importance of “sending” to one’s household. The logic is that drinking the wine bestows importance to the mitzva cup.
If this is the Rambam’s source, then the requirement should apply to all cups of beracha, not just Kiddush. Indeed, the Shulchan Aruch[12] writes that it is true regarding wine for Birkat HaMazon, and it should ostensibly apply to Havdala, although the Shulchan Aruch does not discuss it.[13]
The Rambam himself,[14] while not emphasizing the matter, does use the plural in referring to those drinking the wine used for Birkat HaMazon. Within the halachot of Havdala, the Rambam[15] does not mention drinking at all, even though it is clear that someone must drink. This could indicate that the drinking of Havdala wine follows the same rule as for Kiddush.
The Shibolei HaLeket[16] is an early source that says that our practice is to not give Havdala wine to others to drink, and this view is accepted by the Magen Avraham.[17] The Mishna Berura[18] presents a technical explanation for this practice: Since Havdala is not made in the framework of a meal, we want the mavdil to drink at least a revi’it (which is more than m’lo lugmav) so that the requirement for a beracha acharona on the wine will be a certainty. While it is not clear whether there was an early consensus on this matter, our minhag indeed is that not all drink that wine.
There are some positive reasons for drinking specifically at Kiddush, which might also explain the minhag. Some claim that the obligation to make Kiddush over wine and/or to drink a certain amount of it has a stronger basis than for other cups of beracha.[19] Also, Kiddush is connected to the meal in which all are partaking[20] (there are different explanations for the connection). Since it is recommended to drink wine during the meal,[21] and when one drinks wine at Kiddush he is exempt from making a beracha during the meal,[22] it makes sense to start drinking at Kiddush.
In any case, while Halacha does not obligate everyone to drink Kiddush wine and does not forbid anyone from drinking Havdala wine, your observation has both supporting sources and a variety of possible explanations.