Ask 4 cyclists this question, get 73 different replies.
I'm more a fan of 'you get what you pay for'. There can be a law of diminishing returns, but there can also be some bargains that won't necessarily look like it initially.
Rapha cause many many debates, but the fit for me is astonishingly good (and I struggle with 'normal' cycle clothing), and if bought in one of their regular sales it might seem expensive (or even worse if somewhere like Evans is seen as 'extortionate'!), but... it lasts and lasts and lasts and lasts.
Just bought some Swrv 3/4 troos as well, and incredibly impressed by them so far, but can't comment on the durability yet.
To be honest, clothes that work for you, and work for your budget, are an entirely personal choice (I've a Vulpine jacket which isn't bad, but the fit makes it feel uncomfortable for any ride longer than half an hour, but others absolutely love it, and rightly so if it works for them).
All of that said, I might pop into Aldi...