Step-up Starter, Decant or Not?

If I’m stepping up a starter, I’ve always chilled and decanted the old wort off of the starter prior to pitching the fresh wort. Any reason why I can’t just pitch the new wort right on the top of the old wort? Will it result in off flavors or any other issues?
Thanks

Flavor impact would depend on the relative volumes of the starter wort and fresh batch, the temperature of fermentation, and whether or not the starter was made on a stirplate.

Assume a typical first stage starter of 1 liter, on a stir plate, at 68-70 degrees ambient. Second stage stepped up with 2 liters of wort, on a stir plate, 68-70 degrees. I guess I was assuming that there may not be any impact to the beer as long as I decanted the completed starter and pitched only slurry. My main question had to do with chilling and decanting the first wort prior to adding the second. I always chill my starters when they are finished and just pitch slurry into my beer.

Missed the part about decanting between steps - I wouldn’t simply to make sure I get all the yeast.

I only decant between steps when space is needed. But, I always decant before pitching.

Never thought about losing yeast by decanting between. I’ll have to re-think my procedure. It would also help to speed the process a bit. Thanks for your help guys, appreciate it.