I never saw anyone whisper into the ear of the Messenger of Allah ﷺ and have him pull his head away, rather, it was always the other person who would pull his own head away first. And I never saw anyone take his hand and have him withdraw his hand, rather, it was always the other person who would let go of his hand first.
ما رأيت رجلا التقم أذن رسول الله ﷺ فينحي رأسه، حتى يكون الرجل هو الذي ينحي رأسه، وما رأيت رجلا أخذ بيده فترك يده، حتى يكون الرجل هو الذي يدع يده.
Narrated by Abu Dawud (4794), Tirmidhi (2490), Ibn Majah (3716), and Ibn Hibban (6435), all of them through various routes from Anas bin Malik, who mentioned it. None of these routes is free from some criticism, but they strengthen one another, since none of the narrators involved is accused of fabrication, so the hadith becomes hasan.