Chazal consider a number of factors in giving one mitzva precedence over another, such as tadir (the one that is more regular)[1] and mekudash (the one that has greater sanctity).[2] Although these two factors are certainly relevant to our discussion of Musaf and Hallel,[3] they require complex and complicated discussions, which in any event will remain inconclusive. Therefore, our discussion will concentrate on other elements.
There are a few reasons to give preference to Musaf over Hallel in your case. One is the difference between the levels of importance of Musaf and Hallel for women. There is a machloket regarding whether women are obligated in Musaf. The reason many opinions obligate women in general tefilla, despite the fact that it is a time-bound mitzva, is that it is critical, since it is a request for mercy.[4] However, since Musaf is a special tefilla added on holidays in connection to the sacrificial service in the Beit HaMikdash, and it is not focused on requesting mercy, the Tzelach[5] posits that women are not obligated in Musaf. On the other hand, there are a few reasons to claim that women are obligated in Musaf,[6] with perhaps the strongest one being that we do not easily distinguish between one tefilla and another.[7] The Mishna Berura[8] cites both opinions without expressing a preference.
In contrast, women are clearly exempt from reciting Hallel, since it is a time-bound mitzva, and, at least on Rosh Chodesh, there are no special reasons to obligate them.[9] While some women (perhaps including you) have accepted upon themselves an obligation to recite Hallel,[10] one would still give preference to a part of davening regarding which there is a possibility that women are actually Rabbinically obligated (Musaf). To this we add the fact that reciting Hallel on Rosh Chodesh is only a minhag even for men.[11]
Another factor favoring Musaf relates to the firmness of the deadline. Your question is based on the assumption that Musaf and Hallel (and presumably Mincha[12]) must be recited before sunset, as they are mitzvot of the day,[13] and this view indeed makes sense for non-Chassidic Ashkenazim.[14] However, there may nevertheless be room to recite tefillot of the day later.
Many authorities maintain that b’di’eved, Mincha may be recited for several minutes after sunset,[15] based on one or more of the following reasons: 1) According to Rabbeinu Tam and others, the day does not end until well after what we call sunset.[16] 2) Even after the end of daytime, it does not immediately become definite night; rather, the period that begins afterwards is bein hashemashot, which is treated as a doubt regarding whether it is day or night.[17] 3) If a certain moment during bein hashemashot is, in fact, too late for Mincha, then it is already time for Ma’ariv, and one can therefore daven by making a condition that the tefilla should count for whichever of these two tefillot is appropriate.[18]
Reasons #1 and #2 apply to both Musaf and Hallel, and there would therefore seem to be some logic in allowing their recital after sunset. However, reason #3 does not apply to either of them, as there is no parallel mitzva after nightfall on which to “hang” a condition. Therefore, davening Musaf after sunset is problematic, as it includes berachot that might be l’vatala,[19] a possibility we must take great measures to avoid. In contrast, one can fulfill the recitation of Hallel even if one does not say its berachot, as Sephardic practice is always on Rosh Chodesh.[20] Thus, if you recited Musaf before sunset and then directly followed it after sunset with Hallel without a beracha, you would have a reasonable chance of fulfilling the mitzva of Hallel, and there would be no possibility of doing something forbidden when reciting it. Although one may not recite Hallel freely,[21] doing so on a one-time basis when it is possibly still the time for it should be acceptable.
For one or both of the above reasons, we would have recommended reciting Musaf first, followed by Hallel without a beracha, if you did not expect to finish by sunset.[22]