I think the answer could be really simple: Because you expect it. You expect that there are people on this planet that you don't like and you think that it is possible to interact with them and there they are, right in front of your face. It's also possible that they show you any vibrational patterns you have going on. For example, my mother always gets upset by macho men on a bike (at least she thinks they are). And what does she get? Universe delivers a ton of macho men on bikes to be upset about. So there may be any quality in them you don't like and you have focused on before.

How this relationships play out in the long run is in your hand of cause. You can choose to not longer think about the things that annoy you, you can focus on the things you like about them, you can decide to love them unconditionally.