I suggest you add the acid to the water, not the mash. If you add it to the mash, it will only have any effect on the immedaite area around the mash. Being a weak acid, the phosphoric acid tends to get buffered out by the malt immediately aroung the point of addition, and so has little effect elsewhere.
If you add it to the water, then it will affect the mash consistently. Simply find out what will give you the target mash pH, and then repeat , tweaking alittle if necessary. You can generally add quite a lot of phophoric with little cahnge in pH because it is a weak acid, so don't worry if any extra addition / reduction seems to be a large proportion of the original.
As an aside - 5.5 at 20 C seems high - most people aim for closer to 5.3 for better enzyme action