Are these harming my pea plants?

Weeds take the food away from your peas. If they're large weeds, you might not want to pull them up, because you might pull up the peas also. What we do is, take a pair of cutters and just snip off the weeds close to the ground.