Caste systems arise IRL just fine without any fantasy genetics, so I think you're good there.
The Sidhe genetics ramifications are pretty interesting. Depending on whether genealogy can be easily traced, you either get something like the Numenorian bloodlines from Middle-Earth, or you get magic-talented people popping up at random depending on whether they are born with the appropriate genes or not. Or even both. It could get even more interesting if the magic genes are not related to physical appearance at all, or if they come with a steep penalty like a susceptibility to iron.