I commute via this route, though I am only on a train which stops in Kirkcaldy once a week: the rest of the time I'm on the express which goes straight to Dundee.
I would certainly recommend a folding bike. Most of the time there is no problem with space for bikes on Scotrail services, at least on the way out. Because you are travelling the 'wring' way ie. most people are commuting into Edinburgh. it should be okay. However the odds of the spaces being taken are certainly higher at Haymarket: this is where a folder can be handy as you can fold and it can go in a luggage rack if need be. On the way back you may find the trains busier so again its good to have the folding option.
Generally there are 4 bike spaces on the 3 carriage Turbostar trains, but technically one of these is for wheelchairs too so if there's a wheelchair user (quite rare) or a buggy (rare in peak services) that might reduce the spaces to only 2 bikes. Sometimes they use rolling stock from England, or the old Sprinter trains, on the route and that means only 2 spaces.
On a Sunday if you need to get there before 9:00 the CrossCountry service may be your only option, in which case you'll need a folder, or be prepared to walk/bus. You can reserve your bike on such longer distance trains, but it's a bit of a hassle and depending on the rolling stock the bike rack may be in the carriage or in an old fashioned guard's van. This latter is not a problem at Waverley, but can be very stressful at intermediate stations (eg. Haymarket/Kirkcaldy) because you often don't know until the last minute which end of the train to rush to to load your bike.
Again I can't emphasise enough how useful a folder is. Most of the time you don't need to fold it, but there's always the odd occasion when you do. As the weather gets better, the number of cyclists on the trains increases...