The analysis of data from more than a million people is part of a study of physical activity published in the Lancet to coincide with the Olympics.
Watching TV was found to be worse than sitting at a desk, probably because of associated habits like snacking.
Current NHS guidelines recommend 150 minutes of moderate exercise a week.
Being inactive is known to increase the risk of conditions such as heart disease, diabetes and some cancers.
It has been linked to 5.3 million deaths globally a year - compared with 5.1 million linked to smoking.
The Lancet research says the global cost, for healthcare and lost productivity, is estimated at $67.5bn per year.
To look at the the impact of activity and inactivity, researchers went back to the authors of 13 existing papers and asked all of them to reanalyse their data.
People were classed depending on how active they were - from the least active who did less than five minutes a day, up to 60-75 minutes a day for the most active.
Researchers then looked at how many people died during the follow-up period - between two and 14 years.
Those who sat for eight hours a day, but were physically active, had a much lower risk of premature death compared with people who sat for fewer hours a day, but were not active.
Sitting for a long time as well as being inactive carried the greatest risk.
Prof Ulf Ekelund, of the Norwegian School of Sports Sciences and the University of Cambridge, led the study.
He said: "For many people who commute to work and have office-based jobs, there is no way to escape sitting for prolonged periods of time.
"For these people in particular, we cannot stress enough the importance of getting exercise, whether it's getting out for a walk at lunchtime, going for a run in the morning or cycling to work.
"An hour of physical activity per day is the ideal, but if this is unmanageable, then at least doing some exercise each day can help reduce the risk."
But he admitted: "One hour's moderate activity is substantially higher than current recommendations."
Watching TV for more than three hours was associated with an increased risk of premature death for all but the most active.
The researchers suggest this is likely to be because people might snack while they watch, or because they are more likely to watch TV after eating their evening meal which might affect their metabolism.
It could, they say, also be a sign of a more unhealthy lifestyle in general.
Dr Pedro Hallal of Brazil's Federal University of Pelotas looked at the effect of the Olympics on the general public's activity levels.
He said that, despite a blip around the Games where people temporarily take up a sport, there is no long-term legacy.
"There's been no health legacy of the Olympics reported ever, but it's the perfect time to talk about human movement."
The scientists said governments should ensure their policies encouraged physical activity - citing the example of a bus scheme where stops are placed further apart to encourage walking - and employers should make it easier for staff to be active during their working day - such as flexible lunch breaks and the provision of showering facilities.
Lisa Young, a physical activity specialist at the British Heart Foundation said: "Although we recognise the link between sedentary behaviour and poor health, we do advocate further research in this area to establish categorical statistics in relation to cardiovascular morbidity and mortality."
Dr Mike Loosemore, from the English Institute of Sport, said: "An hour of brisk walking is hard work this is essentially moderate exercise, I suspect not many people would be able to manage that amount of moderate activity a day.
"So if you change the guidelines then it puts them even further out of reach of the people who would benefit most from increasing their physical activity, which are those that do very little.
"For the vast majority of people while the best way to stay healthy would be to do an hour of moderate activity a day, realistically the best place to start is reducing your sedentary behaviour at work by sitting less and try to increase whatever physical activity you are doing."