Is it permissible to read the Holy Quran and religious books with a slight, difficult-to-remove trace of dirt on the hand? Does that affect memorization?

If you have diligently washed and cleaned your hands and you are in a state of purity (tahara), there is no harm in holding the Mushaf (Qur'an) with them. The same applies to books of knowledge. This is because the dirt on your hands that you mentioned, if it was difficult for you to remove, then it is excused (ma'fu 'anhu). Honoring and revering the Mushaf is required, and this includes perfuming it, placing it on a stand, and not putting anything on top of it. This dirt is excused in purification (tahara) if its removal is difficult. Sheikh al-Islam Ibn Taymiyyah holds the view that the excuse (al-afw) covers it even if washing it was not difficult.