Abdullah bin Abbas reported,

When the Messenger of Allah ﷺ arrived, he refused to enter the House while the idols were still inside, so he ordered that they be removed. They brought out an image of Ibrahim and Ismail with divining arrows in their hands.The Messenger of Allah ﷺ said, "May Allah destroy them! By Allah, they knew that these two never sought decisions through divining arrows."Then he entered the House and pronounced the takbir in its corners, but did not pray inside it.

Azami 5310

Sahih

عن ابن عباس، قال

إن رسول الله ﷺ لما قدم، أبي أن يدخل البيت، وفيه الآلهة، فأمر بها فأخرجت، فأخرجوا صورة إبراهيم وإسماعيل في أيديهما الأزلام فقال رسول الله ﷺ: قاتلهم الله! أما والله قد علموا أنهما لم يستقسما بها قط. فدخل البيت، فكبر في نواحيه، ولم يصل فيه.

الأعظمي 5310صحيح
Contents
Reference
Commentary
Ayat
Collection
Jami al-Kamil
Book
Hajj
Chapter
Description of the Hajj of the Prophet ﷺ as narrated by Jabir and others, and the rulings mentioned therein
Section
Those who said the Prophet ﷺ did not pray in the Kaaba
Numbering
Jami al-Kamil 5310 (2nd edition)
Azami 5310