Most of the halachic discussion regarding tziruf (joining together as a halachic group) for a minyan relates to people in adjoining rooms or courtyards or cases in which some people are inside and some outside. In those situations, the main question is whether they are considered to be in the same domain,[1] although other issues play a role as well.[2]
In the case of tziruf for zimun in Birkat HaMazon, visual contact between the people connects them, even if they are in different domains.[3] A major question is whether visual contact also connects people for the apparently stricter matter of a minyan for tefilla. The Rashba[4] was asked why a chazan standing on a bima that is a halachically separate domain combines with the others. He suggests two possible answers that can solve the problem: 1) A bima is made to serve as an integral part of the shul; 2) If some people in one domain see people in the other one, they all constitute one unit. There is a major machloket among Acharonim regarding whether we accept this second reasoning and can rely upon visual contact for tefilla.[5] The Mishna Berura[6] cites both opinions and seems to prefer the lenient one, while the Kaf HaChayim[7] accepts the stringent opinion.
What should be the determining factor when all the people are outside? Mishneh Sachir[8] and Minchat Yitzchak[9] maintain that being able to see and hear each other creates tziruf. These conditions were fulfilled in your case. However, you may have failed another condition. The Shulchan Aruch[10] cites an opinion that a public path that separates parts of the group prevents tziruf for zimun, even if they see each other and are served by a common waiter.[11] The Taz[12] writes that even a minor private path prevents tziruf.[13] Other than the insignificantly low wall, you do not describe what was in between your group and the kohen. However, it is likely that within 50 meters in a cemetery there are pathways. The Mishneh Sachir and Minchat Yitzchak seem to indicate that this precludes tziruf with regard to Kaddish as well.
There may be logic to suggest that if the cemetery is fenced in, minor internal paths do not separate people. There is also logic to accept lenient opinions in our context. One reason is based on need, as it is emotionally important for people to be able to say Kaddish for their loved ones. Moreover, a possibly unauthorized Kaddish is not nearly as severe as a possibly unauthorized beracha (i.e., beracha l’vatala). A beracha l’vatala entails saying HaShem’s name unnecessarily in the context of the beracha, which is either a Torah-level or at least a serious Rabbinic violation.[14] In contrast, we find poskim who hold that one can be lenient in questionable cases of Kaddish. (See, for example, Maharshag[15] regarding the leniency to count a child as the tenth person for a minyan for Kaddish.)
The fact that the kohen was forbidden to come to where the majority of you were standing is not an issue. It is true that if three people eat together in a manner that each may not, for halachic reasons, eat from the other’s food, there is no tziruf for zimun.[16] However, we are unaware of a source that a problem exists in your situation. In the context of zimun, the problem is that a joint eating is required. In your case, in contrast, the content of the Kaddish could unite you; there was simply a question of whether you were close enough to be unified. If you were, there was no halachic need to be closer than that. Furthermore, in the case of zimun, it is enough that Reuven can eat from Shimon’s food, even if Shimon cannot eat from Reuven’s. In your case, you and the others could have gone to where the kohen was. In fact, on practical grounds, you probably should have done that, so that the kohen could have said the Kaddish in a more natural way.