Is boycotting a sister who has caused psychological and familial problems, and has habitually engaged in bad behavior and committed unlawful acts, considered a severing of kinship ties? And how can these kinship ties be maintained despite the hatred for her actions?

Your disapproval of your sister's relationship with her fiancé before the marriage contract is correct, as what transpires between them are مقدمات للزنا (preliminaries to adultery/fornication). Your zeal for the sacred limits of Allah and your honor is commendable. Try to convince your father of the necessity of being firm with your sister, reminding him of his religious responsibility as a guardian of his household, and of the hadith (definition): "All of you are shepherds and all of you are responsible for your flock."

Boycotting a disobedient Muslim is originally forbidden if it exceeds three days, unless there is a benefit in it, such as the disobedient person abandoning their sin or deterring others. Boycotting a sister is permissible if it leads to a change of the evil, otherwise not.

You must supplicate sincerely for guidance for your sister and your parents, and rely upon Allah, for He guides whom He wills.