"The PM and president acknowledged that there had been significant differences between their countries in the past, and agreed that we should seek to progressively improve our bilateral relationship," Cameron's office said in a statement.
The UK severed direct diplomatic relations with Iran after activists stormed its embassy in Tehran in late 2011. The 2013 election in Iran of Rouhani, a moderate pragmatist who replaced Mahmoud Ahmadinejad, paved the way for a thaw in ties.
Britain announced in June that it would reopen its embassy in Iran in the coming months.
The statement said Cameron and Rouhani agreed that nuclear negotiations between Iran and Britain, China, France, Germany, Russia and the UN were at a "critical moment and that it was vital to seize the opportunity of securing a comprehensive [nuclear] agreement".
Iran's official news agency IRNA quoted Rouhani as telling Cameron: "I hope we can reach a comprehensive deal with the strong will of all parties involved because there is no other way to resolve this issue but reaching a common understanding."
The talks are expected to continue until Friday. Diplomats close to the negotiations say a breakthrough is unlikely, even though a November 24 deadline for a deal is only two months away.
Britain’s statement said: "The prime minister and president noted the threat posed to the whole region by ISIL , and agreed that all states in the region must do more to cut off support for all terrorist groups, including financial support."