What is the ruling on my prayer and fasting based on the discharge of brown secretions and blood at intermittent times after the end of my usual menstrual period, and are these secretions and blood considered menstruation?

What you saw of brownish discharge after seeing the clear discharge of purity is not menstruation, because it is not connected to blood and is not within the time of your regular period. The blood you saw after that is irregular bleeding (istihadah), and it cannot be considered a new menstruation; because there was not the minimum period of purity between the two menstruations separating it from the previous blood, and it cannot be combined with what preceded it as their total, along with the period of purity, would exceed fifteen days.

You should have fasted and prayed with the irregular bleeding (istihadah), and you must make up the day you broke your fast due to your interpretation, and make up the prayers you missed thinking it was menstruation. As for the blood that returned at its usual time, that is a new menstruation.