Why was it that a long time ago people didn't live for very long?

I'm talking about cavemen, or the neanderthals. was it because of no medicine or that they just worked to hard? I just need more reasons.