Is it permissible for the questioner to abandon the responsibility of caring for his deceased brother's family after he has spent on them from his private funds and from funds belonging to them, and must he return all their money that was in his possession?

The orphan's money is a grave matter in the sight of Allah, and the guardian must manage it in a way that benefits the orphan, not their own personal interests. It is not permissible for the guardian to benefit from the orphan's money in exchange for their assistance. If your eagerness to keep the money with you is merely for the purpose of safeguarding it, and you deem that to be in their best interest, then there is no blame upon you. However, if you intend to benefit from the profits in exchange for what you disburse, then you must cease that and share the profits with the orphans.