"Or maybe they were coming home from climbing"
You can actually get a permit from the Park Rangers for rock climbing on the crags. I was inexplicably chuffed to possess one a few years back when a friend decided to have a climbing session as part of his stag celebrations.
(Climbing and BBQ in the park followed by an evening in Diane's Pool Hall and then err... some pubs... Blue Blazer... mebbe... all gets a bit hazy after Diane's tbh)
If you really want to know what sort of rocks are present on Arthurs Seat, or indeed anywhere else you may happen to be in the UK, this app provides endless nerdish fun.