How high was the nitrogen content of the fertilizer you used? In my area, we use 10-10-10 or 15-15-15 fertilizer in the spring - not low nitrogen, just balanced.
I don't know of any way to counteract nitrogen once you've used it, unfortunately. It does get used up, though.
