He traced it back to the Prophet ﷺ, who said, "Children of Abd Manaf, do not prevent anyone from circumambulating this House or praying at any hour of the night or day."
قال حدثنا سفيان حدثنا أبو الزبير عن عبد الله بن باباه
يبلغ به النبي ﷺ قال يا بني عبد مناف لا تمنعن أحدا طاف بهذا البيت أو صلى أي ساعة من ليل أو نهار
Its isnad is sahih, and Abdullah bin Babah al-Makki is trustworthy, the imams have praised him. The hadith was narrated by Abu Dawud, 2/180, no. 1894, in the chapter on rites, tawaf after Asr, and Tirmidhi, 3/211, no. 668, in the chapter on Hajj, where he said, hasan sahih, and Nasai, 1/84, no. 582, and Ibn Majah, 1/398, no. 1254, and Darimi, 2/96, no. 1926.