Hi,
This is my work so far. But these black noises still exist on my gem. I tried to change the order of my layer group but it didn't help.

Hey there! Would you be able to share what the base colour map looks like with your UVs overlayed? As well, could you take a screenshot of what the Bake Dialog settings are, specifically what the settings for Pixel Padding/Dilation are?
I have a feeling it's related to the baking process, but taking a look at these should help narrow it down!