Should the fiancée's inappropriate behavior toward the mother after performing Istikhara prayer be considered a sign to break off the engagement, or should the engagement continue despite the family's concerns about its negative impact on family relations?

If the girl is pleasing in religion and character, and after seeking guidance from Allah the Almighty, then proceed with her proposal and do not hesitate; for the ease of the matter is a sign of istikharah (seeking guidance), and if it becomes difficult, that is an indication that there is no good in it. Her incorrect behavior with your mother might be a slip or an oversight, and it is not a sufficient reason to leave her unless it blemishes her religion or character. And you must remove any animosity between your family and your fiancée.