EDIT: fixed by recalculating the normals (shift + N)
When I add the bevel modifier to the edges, these weird shapes show up on one side. On the other side of the model it looks normal. What am I doing wrong?