Don’t get flustered. Review the recent conversation instead. You may have been so vague that colleagues thought you were saying yes when you really meant no.
