This response does not relate to use of an oven on Shabbat, which has potential pitfalls and solutions beyond our present scope. We are also assuming that the food is fully cooked.
Hatmana (insulating hot food) is rabbinically forbidden in two basic circumstances: 1) when it takes place on Shabbat; 2) if the hatmana is done in a medium where heat is being added, even before Shabbat.[1] You refer to hatmana on Shabbat, so we will have to look for situations where the prohibition of hatmana does not apply.
In order to be considered hatmana, the food or its utensil must not only be covered but must be surrounded relatively tightly by the insulating material[2] on all sides (at least for Ashkenazim[3]). Only then is it similar to hatmana in remetz (a mixture of sand and coals), the prototype of the prohibition. In some cases, an insulating container gives a relatively snug fit; in others, the food’s container and the insulation do not come in such close contact. In the latter case, there is no problem. Let us continue to look for solutions for cases that meet the general description of hatmana or are borderline in this regard.
The gemara[4] cites Rashbag, who says that if one has moved food from the utensil in which it was heated into another utensil, he can do hatmana (that does not add heat) to the second utensil even on Shabbat. The gemara explains that, normally, hatmana is prohibited because there is a concern that it might lead a person to violate Shabbat in his desire to make sure it is hot. Here, however, since the person cooled down the food by moving it into another utensil, we do not have to be concerned that he will now reheat it. Thus, the following system should solve all problems. Before putting the hot food into the insulation, first transfer the food into another pan or container, using Rashbag’s leniency, which is accepted as halacha.[5]
Is our case, though, a legitimate application of Rashbag’s leniency? An important machloket exists between the Rambam and Rashi whether Rashbag’s logic applies in a case where the food was moved to a second utensil without intention to cool it off. The Rambam[6] writes that the prohibition exists only in the “kli rishon shenitbashel bo” (the utensil that the food was cooked in) without further distinction, as Rashbag’s statement implies. Rashi[7] implies that there must be intention to cool off the food for the leniency’s logic to apply. The major poskim accept the Rambam’s view.[8] A possibly more stringent application is heated water that is poured into a thermos, where the transfer was done specifically to maintain the heat for as long as possible. Still, most poskim permit the matter based on the Rambam, as the hatmana occurs in a kli sheini (a utensil that was not heated up). Additional factors are raised that might allow even Rashi to be lenient regarding a thermos.[9]
We must consider whether our case is more problematic than that of a thermos. Liquids that are poured into a new utensil cool off significantly and are said to be in a kli sheini, where several halachic leniencies exist. However, many rule that a solid (davar gush) is not very affected by a kli sheini’s cold walls, maintains its heat, and retains the status of kli rishon.[10] Thus, one could claim that Rashbag’s leniency does not apply to solids, as in our case. However, the Rambam’s wording[11] implies and the Pri Megadim[12] states clearly that hatmana is forbidden only in the actual utensil where the food was heated. It does not apply in another utensil, even when the food has a halachic status of being in a kli rishon.[13]
In summary, if one wants to put food heated in an oven pan into a tight-fitting insulating container, it is necessary and sufficient to transfer it into another utensil before insulating.