It so happens that sometimes the client proxy is created and destroyed
immediately, almost instantly, which causes this code to crash in case
the proxy pointer is gone when our idle callback tries to destroy it.

This change makes the whole operation safe.