We will answer this question on several levels, starting with the most basic.
The Shulchan Aruch[2] rules that one should check daily that his tzitzit strings are kosher (and did not rip) before making the beracha to avoid the possibility of uttering a beracha l’vatala.[3] There are some indications[4] that this is more of an act of piety than a real halachic obligation, and it might only apply if one has to make a beracha on this particular tallit, as avoiding a beracha l’vatala is the most acute concern in that case.[5] The reason the check is not absolutely required is that there is a chazaka (a presumption based on status quo and/or experience) that the tzitzit remain kosher. On the other hand, there is a principle that we prefer not to rely on a chazaka when one can easily verify the situation directly.[6] In any case, it is agreed that if one checked the tzitzit before putting them away, the likelihood of change is too remote to warrant checking.[7]
Nowadays, few people check their tzitzit daily (presumably, this is why you were taken aback). Rav Ovadia Yosef explains that this phenomenon is based on classical halachic principles.[8] Additionally, our tzitzit are likely stronger and there may be fewer things that are liable to rip them than in the past. If the minhag to be lenient is based on practical considerations, then it makes a lot of sense to check a shul tallit, as such tallitot are often not checked for a long time and ripped strings are therefore more common. Since a guest has no way of knowing the condition of a particular tallit, one can argue that it is more problematic to rely on a chazaka and that he is actually required to check it before putting it on with a beracha. (The guidelines for what disqualifies tzizit[9] are beyond our present scope.)
There is no halachic prohibition to check the tzitzit of a tallit on Shabbat. The Bi’ur Halacha[10] discusses what the gain is in checking them, considering that fixing them on Shabbat is not permissible. He concludes that at the very least, the check will alert the person not to make a beracha on the tzitzit and, when possible, to avoid wearing the tallit. Based on the above, the guest followed the preferred approach of the classical halachic sources, and you should therefore view him as having acted honorably.
There is a flipside, however, which should be noted by those who are in the guest’s predicament, as opposed to the observers’ position. The Rabbis were often concerned[11] lest one act differently from those around him in a manner that could be insulting, especially in the context of religious observance. Since checking the tzitzit of a tallit is rarely done, a guest who checks the tzitzit of the shul tallit may give the impression of disrespect to the host shul, as the congregants might assume that he specifically has doubts about them, rather than that he simply is being halachically careful as a matter of course.
The question is whether a person who chooses to be strict about checking tzitit (at least of a shul tallit) has alternatives. Besides the obvious options of relying on the lenient opinions or checking inconspicuously, we will present another one. One fulfills the mitzva of tzitizit by wearing a tallit katan;[12] wearing a tallit is not necessary for that purpose. Despite this, married men have the practice of wearing a tallit when davening Shacharit.[13] Putting on a tallit with a beracha as an additional fulfillment of the mitzva of tzitzit, as is the norm, requires owning the tallit. It is true that one can make the beracha on a shul-owned tallit,[14] either because a public tallit may be jointly owned [15] or because he has tacit permission to acquire it for the time he wears it in order to fulfill the mitzva.[16] However, many men have the minhag not to make a beracha,[17] and they thus do not intend to acquire the tallit. The matter of joint ownership is also very questionable regarding a guest in shul. Consequently, those who do not make the beracha are neither obligated in nor fulfill the mitzva of tzitzit when using the shul tallit and thus have no need to check its tzitzit. In our opinion, from the perspective of a guest, this approach is better than making a spectacle that people in shul (rightly or wrongly) find offensive.