What is the ruling on a wife boycotting her husband's mother, and is the wife justified in boycotting her due to her refusal to return the dowry and her not visiting her?

An engagement is merely a promise of marriage, and either party has the right to dissolve it. The jewelry given (al-shabakah), in Egyptian custom, is considered part of the dowry and must be returned to the suitor upon the dissolution of the engagement. If your mother refuses to return it, she is committing an injustice, and she should be advised accordingly.

As for your wife shunning your mother, if she speaks to her and greets her, this is not considered a legitimate shunning (hajr) in Islamic law. However, if their relationship is completely severed, and if this is due to personal indignation, then shunning is not permissible for more than three days, based on the Prophet's (peace be upon him) saying: "It is not lawful for a Muslim to shun his brother for more than three days." Whoever shuns for more than three days without a legitimate Islamic reason is exposed to the warning mentioned in the hadith about deeds being presented on Mondays and Thursdays.

However, if the shunning is for a legitimate Islamic reason, such as disobedience or injustice, then it is permissible. The matter of shunning or maintaining ties depends on what serves the best interest (maslahah). Sometimes maintaining ties is better, and sometimes shunning is better, as explained by Sheikh al-Islam Ibn Taymiyyah.