I slightly beveled the furthest arch in the picture and my UV map exploded. I have no idea what I have done wrong.
I had to clean my mesh by merging by distance..
I flipped my normals while I was there at the same time and came back and it was all fine.
Thanks a lot, for anyone who has the following issue :
You baked you curved meshes and end up with weird UVs.
Select all the mesh in Edit Mode > Merge By Distance (basically you want to remove the bevel you added when doing the vertex painting) and bake again, it fixed the issue for me :)