I found out that in order to really have the session data cleared after calling session_destroy(), you need to refresh the page. For example by calling
header('Location: '.$_SERVER['PHP_SELF']."?page=profile");
if you don't then there are still data in $_SESSION which was there before. I don't know if this is intended behavior but it's kind of confusing IMO...