Clean up some iothreads

Improve the iothread behavior by enabling an iothread to stick around for
a while waiting for work. This reduces the amount of iothread churn, which
is useful on platforms where threads are expensive.

Also do other modernization like clean up the locking discipline and use
FLOG.

src/iothread.h

@@ -27,7 +27,8 @@ int iothread_port(void);
 void iothread_service_completion(void);
 
 /// Waits for all iothreads to terminate.
-void iothread_drain_all(void);
+/// \return the number of threads that were running.
+int iothread_drain_all(void);
 
 // Internal implementation
 int iothread_perform_impl(std::function<void(void)> &&func, std::function<void(void)> &&completion);