This one is pretty self explanatory, and it pretty much wrote itself after the season 5 premiere. If they keep on skipping in time then at some point they’ll go back in time far enough and die there. So if they ever go to the caves then say good bye to those characters.
I guess the next question is, where did they get the black and white stones? I can guess what they represent but I don’t know where they’d get them from.
(I can’t think of any way to expand this theory.)