It's definitely a workable premise! You can always downplay the actual "mechanics" of the time travel jutsu, and focus more on the emotional and physical toll it takes on Naruto as his memories merge. It's bound to cause odd behaviors, and all sorts of peculiar situations.