Food that is cooked on Shabbat is one of many examples of ma’aseh Shabbat (something produced as a result of chillul Shabbat), and as such is forbidden to be eaten.[1] Your question is a good one: Does such food make utensils treif?
The answer seems dependent on whether ma’aseh Shabbat regarding food is simply a prohibition against benefiting from it (in the case of food, usually by eating) or whether the food is considered ma’achalot assurot (what we call non-kosher food). If the former is true, there is no need for concern about the oven, as any residue in the oven will not bring you real benefit. If the latter reasoning is true, however, the food is like any other forbidden food that “treifs up” an oven. (In this context, we will not discuss how an oven becomes treif, how it affects foods that are cooked in it, or how it is kashered.)
Logically, one reason that we should not consider food that is cooked on Shabbat to be ma’achalot assurot is that it is prohibited for an external reason – due to its connection to an improper situation – and not because of an intrinsic problem with the food. The Ktav Sofer[2] compares ma’aseh Shabbat food to bishul akum,[3] another case of food that is not intrinsically problematic but is “artificially tainted” by an unwanted situation. There is a machloket Rishonim regarding whether bishul akum causes the utensils that were used to become not kosher.[4] The Shulchan Aruch[5] cites both positions, but prefers the stringent one (although he is slightly lenient regarding how to kasher such utensils).
Indeed, the Magen Avraham[6] cites the Rashba as saying that ma’aseh Shabbat food treifs the utensil in which it was cooked, and he and the Mishna Berura[7] accept this position. Rav S.Z. Auerbach[8] considers the Magen Avraham’s ruling to be proof that ma’aseh Shabbat food is indeed viewed as ma’achalot assurot, which treifs up utensils.
Many authorities disagree, however. Aside from significant opinions that are lenient regarding a pot used for bishul akum, ma’aseh Shabbat has additional reasons for leniency. The Mateh Yehuda[9] says that the Rashba implies that a utensil would become treif due to ma’aseh Shabbat only according to the opinion of R. Yochanan HaSandler, who views ma’aseh Shabbat as an intrinsic Torah law.[10] According to the opinion of R. Yehuda, which the Shulchan Aruch[11] accepts, the food is forbidden forever only for the person who was mechallel Shabbat by cooking. It is difficult to make such a distinction if ma’aseh Shabbat food is actually ma’achalot assurot, which are generally objective prohibitions that apply equally to all Jews.[12] Finally, some authorities[13] note that the Gra[14] rules like R. Meir, who is even more lenient than R. Yehuda and maintains that the food becomes permitted after Shabbat even for the one who violated Shabbat. According to that opinion, there is certainly no problem in your case.
We would add that since ma’aseh Shabbat applies to many non-food-related melachot, in which case the category of ma’achalot assurot does not apply, the concept of ma’aseh Shabbat is better understood if all the items share the categorization of prohibitions of benefit.
Nevertheless, it is difficult for an Ashkenazi posek to argue with the opinions of the Magen Avraham and the Mishna Berura, at least without other grounds for leniency.[15] Rav Ovadia Yosef,[16] on the other hand, concludes that the basic halacha is to be lenient regarding utensils used for ma’aseh Shabbat, although he views those who do kasher such utensils as acting laudably.
Despite these considerations, in your case there is little room for concern. We forbid ma’aseh Shabbat after Shabbat only when the chillul Shabbat was intentional, and even then only for the one who was mechallel Shabbat. According to most poskim, the ma’aseh Shabbat is not even forbidden for a person for whose benefit the Shabbat desecrator did the work,[17] and it is certainly permitted for others.[18] Therefore, since you had nothing to do with the chillul Shabbat, the food and certainly its residue in the oven are permitted for you. (You did not ask, and we will therefore not discuss, the topic of trusting the kashrut standards of one who does not observe Shabbat.)