When I use a boolean I get artifacts around the area of intersection (where the two meet) on the main part. "Auto smooth" seem to fix it. Are you using auto smooth? Also at this point did you apply your SubSurfs at level 2 to all parts?
I was getting a head of my self. Bool tool and auto-smooth. got it!