The gemara[1] asks what the earliest time to recite Tefillat HaDerech is, and it answers, “from the time he seizes the road.”[2] The Magen Avraham[3] explains that this means after the traveler leaves the city and comes to a place that is beyond any string of peripheral houses 70 amot from each other.[4]
In contrast, the Taz[5] writes that there is no classical source for waiting until after leaving the city; the gemara about “seizing the road” means that one must simply be certain he will be embarking on the journey. In fact, the Tur and Shulchan Aruch[6] cite the practice of the Maharam MeRotenburg to connect Tefillat HaDerech to Birchot HaShachar.[7] The Taz understands that this is done during the Shacharit before the trip, before leaving the city. However, others, including the Eliya Rabba,[8] offer a different understanding of the Maharam, arguing that he connected Tefillat HaDerech to Birchot HaShachar only when he davened during his trip.
The Shulchan Aruch[9] connects Tefillat HaDerech to the beracha said upon leaving the city one was visiting. The Pri Megadim[10] views this as an indication that according to the Shulchan Aruch, Tefillat HaDerech should be recited right after leaving the city, even before passing the peripheral houses.
The Mishna Berura[11] prefers the Magen Avraham’s approach of waiting until leaving the city, and you are therefore likely correct that people in your tour groups sometimes recite Tefillat HaDerech inadvisably early. However, nearly all Acharonim agree that one fulfills his obligation b’di’eved if he recites the beracha at least after preparing to leave.[12] Therefore, the beracha is not l’vatala, and you may and should answer amen.[13]
We will add that it is logical to argue that in our day, Tefillat HaDerech should l’chatchila be recited earlier than it once was. After all, in the past, the main safety concerns were being attacked by bandits and animals, which were more prevalent outside the city, whereas nowadays, our primary concern is car accidents, which can happen anywhere. On the other hand, it is difficult to change long-established halachic rules based on such a factor.[14]
One could argue that it would be best for you to answer amen but have in mind not to be yotzei, and then to later say Tefillat HaDerech yourself at the preferable time. However, besides possibly leading to awkwardness, you might lose out by doing so. One advantage of saying the beracha relatively early is that Rashi[15] presents a view that one must say Tefillat HaDerech within the first parsa (approximately 4 km) after leaving the city, and this view is partially accepted by the Rama.[16] Thus, if you do not act quickly and say Tefillat HaDerech quite early during the trip, it is possible you will miss the opportunity to recite it according to this opinion. Thus, simply being yotzei with everyone else is actually worthwhile.
There is no reason for concern if the person reciting Tefillat HaDerech omitted the phrase “v’tachzireinu l’shalom.” This phrase (different siddurim have variations) is mentioned for all trips by some Rishonim[17] and is not mentioned at all by others.[18] Although it is indeed quite accepted in contemporary practice to add this phrase specifically if one plans to return home the same day, this distinction first appears only in certain Acharonim. Therefore, one need not be concerned that the validity of the beracha is affected by omitting it.[19]