The Messenger of Allah ﷺ said, "Were it not for the recent transition of your people from disbelief, I would have demolished the House and rebuilt it upon the foundation of Ibrahim , and I would have made for it a back door. For when Quraysh built the House, they fell short of its original dimensions."
أخبرنا إسحق بن إبراهيم قال أنبأنا عبدة وأبو معاوية قالا حدثنا هشام بن عروة عن أبيه
قال رسول الله ﷺ لولا حداثة عهد قومك بالكفر لنقضت البيت فبنيته على أساس إبراهيم عليه السلام وجعلت له خلفا فإن قريشا لما بنت البيت استقصرت
Bukhari, Hajj, chapter on the merit of Makkah and its construction, etc., no. 1585, as a suspended narration, and Muslim, Hajj, chapter on the demolition and rebuilding of the Kaaba, no. 1333, from the hadith of Abu Muawiya through this route. It is also in Al-Kubra, no. 3885.