It so happens that sometimes the client proxy is created and destroyed
immediately, almost instantly, which causes this code to crash in case
the proxy pointer is gone when our idle callback tries to destroy it.

This change makes the whole operation safe.

Currently this will just grant full access to all clients.
A future version will refine that to grant access to specific
objects only.