Senior doctors at a flagship NHS hospital have accused the health secretary of avoiding their over-stretched departments during a visit.