endpoint: fix clearing the weak ref

g_weak_ref_clear destroys the weak ref, so it will crash
when _finalize() tries to destroy it again