Halep had sailed through the first two rounds on the Spanish clay without dropping a set but she ended up second best in a match which yielded 15 breaks of serve.
Browsing: Elise Mertens
The match ended in dramatic fashion when both Mertens and Sabalenka were drawn into the net and Barty sent a lob over both their heads.
World No 1 Simona Halep has ended her collaboration with the Belgian coach Thierry van Cleemput following her loss in the final of the Qatar Open.
Defending champion, Sloane Stephens, eased into the quarterfinals of the US Open with a 6-3, 6-3 victory over Belgian Elise Mertens on Sunday.
Aryna Sabalenka moved within one win of her first WTA victory after defeating fifth seed Julia Goerges 6-4 7-6(3) on Friday to reach the final of the Connecticut Open.
Top seed Simona Halep returned to sew up victory over Ajla Tomljanovic on Friday after their second round match at the Cincinnati Open had been suspended the previous day due to rain.
Romanian Halep, twice a runner-up at Roland Garros, started slowly but once she found her groove, there was nothing the 16th seed could do to avoid a one-sided defeat.
The 37-year-old American has had a stressful build-up to the tournament after police last week blamed her driving for a fatal motoring accident in Florida.