Windows: Fix deadlock in backend when submitting transfers.

Without this change the Windows backend needed to call usbi_fd_notification()
from within the backend's submit_transfer. This can cause deadlock when
attempting to lock the event lock if another thread was processing events on
the just-submitted transfer.

The deadlock comes about as the thread calling libusb_submit_transfer acquires
the transfer mutex before trying to acquire the event lock; this is the other
order of lock acquisition from an event thread handling activity on the just
submitted transfer. This could lead to one of two deadlocks:

1) If the transfer completes while usbi_fd_notification() is waiting for
the event lock and the callback attempts to resubmit the transfer.

2) If the transfer timeout is hit while usbi_fd_notification() is waiting
for the event lock then the attempt to cancel the transfer will deadlock.

This patch fixes both of these deadlocks by having libusb_submit_transfer()
only call usbi_fd_notification() after having released the transfer mutex.

1 files changed, 4 insertions, 0 deletions

diff --git a/libusb/io.c b/libusb/io.c
index d7fae7e..ab32e70 100644
--- a/libusb/io.c
+++ b/libusb/io.c
@@ -1292,6 +1292,7 @@ int API_EXPORTED libusb_submit_transfer(struct libusb_transfer *transfer)
 		LIBUSB_TRANSFER_TO_USBI_TRANSFER(transfer);
 	int r;
 	int first;
+	int updated_fds;
 
 	usbi_mutex_lock(&itransfer->lock);
 	itransfer->transferred = 0;
@@ -1325,7 +1326,10 @@ int API_EXPORTED libusb_submit_transfer(struct libusb_transfer *transfer)
 #endif
 
 out:
+	updated_fds = (itransfer->flags & USBI_TRANSFER_UPDATED_FDS);
 	usbi_mutex_unlock(&itransfer->lock);
+	if (updated_fds)
+		usbi_fd_notification(ctx);
 	return r;
 }