Abu Dharr used to say regarding someone who performs Hajj and then converts it into Umra that this was permitted only for the riders who were with the Messenger of Allah ﷺ.
حدثنا هناد يعني ابن السري عن ابن أبي زائدة أخبرنا محمد بن إسحق عن عبد الرحمن بن الأسود
أن أبا ذر كان يقول فيمن حج ثم فسخها بعمرة لم يكن ذلك إلا للركب الذين كانوا مع رسول الله ﷺ
No references are available.
[Its chain of transmission is daif] Bayhaqi narrated it, 5/22, from the hadith of Abu Dawud through this route, and its chain is daif because of Ibn Ishaq's use of "an" (indicating a possible break in transmission). The hadith has corroborating narrations in Muslim, no. 1224, and Humaydi, nos. 133 and 134, and others.