I use the swirl method also. Depending on the size of the starter, the age of the yeast, and the temperature, you can see varying times. I will allow my 1L-2.5L starters to go 3 days and crash 2, my larger 4L starters go 5 or 6 days and crash for 2. My larger starters are usually using some old harvested yeast so they need a little more time to finish up since I’m pitching 10-20B cells as opposed to 100B cells from a smack pack.
A good indicator, like 560 says, is when the yeast begin to flocc out and the top clears. Another is when you swirl you get no more gas escaping. This means they’re done eating.
FYI the yeast do not need to fully attenuate the starter. You want them in a nice sugary, oxygen rich environment to reproduce but you could crash the starter before they’ve actually finished fermenting.