Kent, I have tried every configuration in the boolean modifier possible (combining difference, intersect, and union with both carve and b-mesh), applied the scale to the rivet, tried flipping its normals and tried each configuration again, the boolean object's origin is correct, I changed the stack order of the modifiers putting the boolean at the top and bottom both, and nothing is working. The best I get is in the jpg. What else can I do here? 
pointoflife14 It looks like you're trying to boolean and use a subsurface modifier, which I don't recommend. Like the subsurf is trying to smooth the bad topology that booleans generate. Is that what's happening?
Let me try that real quick. That's a solution I've not heard yet, so that just might be it (fingers crossed).
I deleted the sub-surf mod and then tried it again and the result got worse. I attached a pic of the topology of the thigh guard (it's basically just like yours).
Hi David. Did you figure this issue out? I ran into the same thing but it does away when I enable auto-smooth under the data tab. But i'm not sure if that is good or bad for texture baking.
I did finally find out that enabling auto-smooth makes the problem go away, but then marking seams and unwrapping is next to impossible because of all the extra geometry added by auto-smooth. Thanks for the help Prashan.