I know what you mean about really clear and disturbing samples. That sort of thing definitely isn't ambient. (Why do I think you are referring to Woob - Strange Air?
Actually, I like that, BUT not when I want ambient. I like it when I want "experimental.")
There are times when I believe that voices -- even disturbing voices such as in a fight or a riot -- are ambient. This is when they are mixed well into the music and don't stand out. I don't believe that ambient is always the same as relaxing: for me it means music that blends with an environment (usually the relaxing kind) OR music that creates an impression of an environment -- and that may include crowd sounds or even a disturbance of some sort -- as long as it's happening in the distance and not right next to me, as it were. In those cases, I can drift and pretend there's just a crowd somewhere around, but I don't have to pay attention if I don't want. Otherwise, I can "eavesdrop" on various members of the crowd. It depends on my mood.
If the voices demand attention even if I don't want to give that attention, then it's not ambient, it's experimental.
Those are my opinions.