Physical time travel would violate conservation by removing matter and energy from the departure time and adding matter and energy to the destination time.

My favorite time travel story is The End of Eternity by Isaac Asimov.