If you‘ve ever got brake fluid on your greasy hands then you know how well it removes oil. Brake fluid is not something you want to have in the oil. Even if the amount is very small, I would better change the oil. As said before, better save than sorry.
On the other hand failing clutch slave...