As you seem to be aware, the very long-standing practice has been to roll the mezuza klaf and put it into some sort of canister. The practice is documented already by the gemara’s[2] statement that the mezuza should be rolled from the left side of the parchment to the right, so that its beginning (in right-to-left Hebrew) is opened up first. These instructions are codified as halacha by the Rambam[3] and the Shulchan Aruch.[4]
However, your assumption that the rolled method is at least preferred is not necessarily evident from these classical sources. It is possible that the rolled-up method is simply technically preferable or assumed, possibly because it takes up less space and/or is better protected. Perhaps being able to read the words is even an advantage.[5] Rav Ovadia Yosef[6] actually raises this as a real possibility. Of course, it is difficult for many of us (including this respondent) to diverge from tradition and ignore possible mystical implications[7] about which we know little. Note also that there are also apparent advantages to being able to read the contents of tefillin scrolls (which are closely related to mezuzot), yet these are certainly rolled up and unseen.
Let us return to your question and consider whether an unrolled mezuza could be halachically prohibited. The gemara[8] asks why we do not take the pasuk literally and write a mezuza’s content directly on our doorposts. In this event, there would not be anything to roll up, of course, which seems to prove that there is no intrinsic need for rolling. However, the gemara’s conclusion, which provides a source for writing the mezuza specifically on a sefer (i.e., a normal writing surface), may impact our question. The gemara says that we learn from another appearance in the Torah of the word “writing,” which we know is done on a normal writing surface, that the same is true for a mezuza. The Rishonim cite three possible identifications of the other “writing”: a get, the scroll of a sota, and a sefer Torah. There is a very strict, lone opinion[9] that assumes that the gemara compares a mezuza to a sefer Torah and then reasons that since a sefer Torah must be rolled, a mezuza that is not rolled cannot be used to fulfill the mitzva of mezuza. This opinion is convincingly disposed of by Rav Ovadia and others.[10]
The more serious remaining question is whether changing from standard tradition is objectionable, in and of itself. In Rav Ovadia Yosef’s aforementioned responsum, he cites an opinion that, on these grounds, an open mezuza should be removed and reaffixed (without a beracha). His own approach is that while he does not recommend it, one could allow such a mezuza display to remain.
Although the question of changes in tradition is a major topic that cannot be treated properly in a sentence or two, it is pertinent to mention an important factor – intention. Affixing a mezuza in an open manner usually would not be motivated by subversive intentions. Rather, it would be intended to glorify the mitzva, to make it more apparently significant to some, and/or to be unique. Therefore, although we too would not encourage the practice, we do not choose to criticize it either. We would hope that those affixing such mezuzot do not mean to cast aspersions on the classical custom of affixing a mezuza, which would certainly be unfounded and improper.
We also refer to our responsum in Living the Halachic Process, vol. I.[11] We ruled there that if the scroll was not yet used as a halachic mezuza, it can be displayed in a non-halachic setting in a room in which it will not be disgraced. Therefore, one who is inclined toward displaying a mezuza scroll is likely better off using the readable one on his wall and putting a standard mezuza case on his doorpost.