Shouldn't means "ne dolzhny", don't have to means "ne neobhodimo" ->
Ne speshi. Nam ne neobhodimo byt' vo vremya. Do you understand how the whole meaning changes if you say shouldn't/"ne dolzhny"?
...