The main issue involved in this question is the definition of tochein (grinding). Although the classic case is grinding wheat into grain flour, the gemara[1] states that it is forbidden even to cut up vegetables into fine pieces. In a response regarding the topic of making guacamole on Shabbat,[2] we cited different opinions as to whether the Rashba’s leniency[3] that one may cut vegetables finely soon before eating applies only when he leaves the vegetables in “slightly large pieces.” We also cited Rav Moshe Feinstein’s opinion[4] that mashing bananas is not grinding, as one does not create new fine particles, but rather mashes the food into a softer form. On the other hand, the Chazon Ish[5] rejects this distinction.
Skipping other possible leniencies mentioned in that context that can be applied to the potatoes you discussed, we will raise a leniency that applies to your potatoes that does not apply to guacamole. Poskim[6] infer from the Rambam[7] that one may take a food that was already made soft by cooking and soften it further by means of mashing. The Chazon Ish[8] explains that the softness created by the cooking causes the additional mashing to be too trivial to be forbidden by the melacha of tochein. Contemporary poskim[9] discuss your case explicitly and rule that it is permitted to mash soft, well-boiled potatoes on Shabbat for this reason.
Unfortunately, however, you seem to have violated a rule that is an important part of this leniency. Specifically, when one grates a food that is not subject to the prohibition of tochein, it is nevertheless forbidden to use a utensil that is intended especially for mincing or crushing food items.[10] The reason for this prohibition is that using such a utensil is uvdin d’chol (a weekday-like activity).[11] A hand-masher seems to fit the bill of a utensil intended for crushing; you should have used a fork. This stringency is relevant even to a food to which tochein does not apply, such as cheese,[12] and even if one does the mincing immediately before the use.[13]
On the other hand, as far as what to do with the potatoes after the fact, you were more stringent than Halacha mandates. There is a broad machloket among the Tannaim as to the extent to which the end result of a violation of a melacha is forbidden, and there are too many opinions and permutations to address here.[14] We will deal with your case, where the violation was done b’shogeg (by mistake, i.e., not knowing the halacha). In such a case, the Shulchan Aruch[15] rules that it is forbidden to benefit from the melacha done on Shabbat, but it is permitted right after Shabbat. (If necessary, there are opinions that can be relied upon to be lenient regarding benefitting even on Shabbat itself.[16]) This indeed seems to fit with your decision not to use the potatoes. However, the Gra[17] posits that if an action done by mistake was prohibited only Rabbinically, there is no prohibition to make use of the result, even on Shabbat. (An important exception to this leniency is if one left food on the fire on Shabbat without proper precautions to avoid mistakes.[18]) Not only does the Gra’s leniency apply fully to this case, but it is further strengthened by the fact that uvdin d’chol is a weaker Rabbinical prohibition than the average one. Therefore, despite your mistaken use of the hand masher, you could have eaten the mashed potatoes even on Shabbat.