Brazil manager Luiz Felipe Scolari says he was sad to turn down the England manager's job in 2006 while he was boss of Portugal.