Can plucking facial hair damage a hair follicle to the point that it won't grow back?

I'm a 23 year old guy, with what used to be a nice goatee. But I have a nervous habit (like when I'm driving or whatever). I'll continuously just pull at my goatee hairs... and eventually, I pull at one long enough and it comes out. Eventually I'd get a little bare spot and shave it off. It used to grow back fine, but now I'm noticing that no matter if I keep it shaved for months, when it does grow back, it looks a lot thinner in the middle (where I do the most nervous plucking). Soooo... will it eventually fill back out, or did I do to much damage to the follicle? Also, would various hair regrowth treatments / solutions do any good? Thanks!