Few classical sources discuss this type of scenario, which is common these days. This can be attributed to changes in economics. Our point of departure, as you correctly assume, is that the sale of a home obligates one in ma’aser on the net gain, as Rav Moshe Feinstein[2] assumes. However, some of the technicalities in arriving at the real net gain make it almost impossible to arrive at an exact figure.
Rav Feinstein discusses Halacha’s outlook on inflation’s erosion of a currency’s value. For complicated reasons, despite inflation, one who borrows $1,000 should return $1,000, and if he gives more, he violates the prohibition of ribbit.[3] Regarding ma’aser kesafim, though, Rav Feinstein’s instruction is to adjust the sale price for inflation to determine the real net gain. However, he feels that the government’s publicized Consumer Price Index (known as madad in Israel) is not correct for our context. Rather, we should consider the inflation on only basic items, not expensive items that one does not buy regularly. Nevertheless, in practice, the CPI may still be the most realistic tool people have access to.
One could claim that considering the interest payments on the mortgage, the real cost of buying the home was greater than the price actually paid, thus lowering the net gain. However, a major component of the cost of a mortgage is due to the inflation component of the mortgage. Thus, if one already reduced the net profits due to twenty-five years of inflation, as above, he cannot also take off all of the interest-related parts of the payments on the mortgage. Providing an accounting system to deal with this is not practical in this forum.
Certain home improvements and repairs that are needed to maintain and/or raise the resale value of the house should also be deductible. However, a major element in fixing things and making improvements in a home over the years should be viewed as being motivated by improving the quality of life of its residents, as much as – or perhaps more than – a means of raising the eventual resale value, and these costs should therefore not be deducted. This certainly is impossible to calculate with any precision.
One could claim that the practice of ma’aser is not applicable to the purchase and eventual sale of residential real estate, assuming one gave ma’aser on his earnings before buying it. When Chazal extended (either by their understanding of p’sukim, Rabbinic decree, or spiritual advice[4]) the concept of ma’aser from agricultural produce to include other earnings, they primarily addressed business dealings.[5] Buying $1,000 of merchandise to sell as soon as possible for $1,500 is a commercial activity designed to create earnings, and that falls under the obligation of ma’aser kesafim. Using one’s earnings (especially after giving ma’aser from them) to buy a home in which to live is a matter of consumption, not creating profits. According to this view, one would not need to give further ma’aser on these holdings, even if turns out that the home’s price increased in real terms.
The question is whether selling a home creates a new obligation. One could distinguish between one who buys real estate to sell at a profit, which is commerce like any other, and one who sells because he needs to change homes for some reason. The case to exempt is strongest when one needs all the proceeds of the sale to buy a new home. If two people swap homes, intuition dictates that neither would have to pay for the appreciation in the home he is transferring. It is not clear that selling a home to buy another one of similar value is fundamentally different in our context.
The strongest position is that calculating ma’aser kesafim is only a proper minhag, not an outright obligation.[6] This justifies leniency regarding calculations and halachic disagreements, especially if that was one’s stated intention when starting the practice. On the other hand, ma’aser kesafim is not an independent simple custom. Rather, it is the recommended average amount of tzedaka to be given in fulfilling that mitzva.[7] One should always want to give tzedaka generously. Cash flow issues often make it difficult, and the sale of a home may give one the opportunity to do so. It may also be an appropriate time for one to internalize that his home purchase decades before was blessed by HaShem, which, in turn, provides an incentive to give significant contributions to those in need. Accordingly, the question of whether there is a formal obligation of ma’aser, which is anyway hard to calculate, can be made almost moot.