Does the prayer become invalid if the worshiper intended to salute the guardian angels without intending to conclude the prayer, then doubted and repeated the salutation and prostrated for forgetfulness?

What you did, intending to greet only the guardian angels, is considered sufficient by some scholars, such as Imam Ahmad, who said: "If he intends by his salutation to respond to the guardian angels, it is sufficient for him." Based on this, your prostration of forgetfulness was out of place, but it does not invalidate the prayer.