It's important to note that array_diff() is NOT a fast or memory-efficient function on larger arrays.
In my experience, when I find myself running array_diff() on larger arrays (50+ k/v/pairs) I almost always realize that I'm working the problem from the wrong angle.
Typically, when reworking the problem to not require array_diff(), especially on bigger datasets, I find significant performance improvements and optimizations.