Based on kernel version 3.9. Page generated on 2013-05-02 23:08 EST.
1 INFINIBAND MIDLAYER LOCKING 2 3 This guide is an attempt to make explicit the locking assumptions 4 made by the InfiniBand midlayer. It describes the requirements on 5 both low-level drivers that sit below the midlayer and upper level 6 protocols that use the midlayer. 7 8 Sleeping and interrupt context 9 10 With the following exceptions, a low-level driver implementation of 11 all of the methods in struct ib_device may sleep. The exceptions 12 are any methods from the list: 13 14 create_ah 15 modify_ah 16 query_ah 17 destroy_ah 18 bind_mw 19 post_send 20 post_recv 21 poll_cq 22 req_notify_cq 23 map_phys_fmr 24 25 which may not sleep and must be callable from any context. 26 27 The corresponding functions exported to upper level protocol 28 consumers: 29 30 ib_create_ah 31 ib_modify_ah 32 ib_query_ah 33 ib_destroy_ah 34 ib_bind_mw 35 ib_post_send 36 ib_post_recv 37 ib_req_notify_cq 38 ib_map_phys_fmr 39 40 are therefore safe to call from any context. 41 42 In addition, the function 43 44 ib_dispatch_event 45 46 used by low-level drivers to dispatch asynchronous events through 47 the midlayer is also safe to call from any context. 48 49 Reentrancy 50 51 All of the methods in struct ib_device exported by a low-level 52 driver must be fully reentrant. The low-level driver is required to 53 perform all synchronization necessary to maintain consistency, even 54 if multiple function calls using the same object are run 55 simultaneously. 56 57 The IB midlayer does not perform any serialization of function calls. 58 59 Because low-level drivers are reentrant, upper level protocol 60 consumers are not required to perform any serialization. However, 61 some serialization may be required to get sensible results. For 62 example, a consumer may safely call ib_poll_cq() on multiple CPUs 63 simultaneously. However, the ordering of the work completion 64 information between different calls of ib_poll_cq() is not defined. 65 66 Callbacks 67 68 A low-level driver must not perform a callback directly from the 69 same callchain as an ib_device method call. For example, it is not 70 allowed for a low-level driver to call a consumer's completion event 71 handler directly from its post_send method. Instead, the low-level 72 driver should defer this callback by, for example, scheduling a 73 tasklet to perform the callback. 74 75 The low-level driver is responsible for ensuring that multiple 76 completion event handlers for the same CQ are not called 77 simultaneously. The driver must guarantee that only one CQ event 78 handler for a given CQ is running at a time. In other words, the 79 following situation is not allowed: 80 81 CPU1 CPU2 82 83 low-level driver -> 84 consumer CQ event callback: 85 /* ... */ 86 ib_req_notify_cq(cq, ...); 87 low-level driver -> 88 /* ... */ consumer CQ event callback: 89 /* ... */ 90 return from CQ event handler 91 92 The context in which completion event and asynchronous event 93 callbacks run is not defined. Depending on the low-level driver, it 94 may be process context, softirq context, or interrupt context. 95 Upper level protocol consumers may not sleep in a callback. 96 97 Hot-plug 98 99 A low-level driver announces that a device is ready for use by 100 consumers when it calls ib_register_device(), all initialization 101 must be complete before this call. The device must remain usable 102 until the driver's call to ib_unregister_device() has returned. 103 104 A low-level driver must call ib_register_device() and 105 ib_unregister_device() from process context. It must not hold any 106 semaphores that could cause deadlock if a consumer calls back into 107 the driver across these calls. 108 109 An upper level protocol consumer may begin using an IB device as 110 soon as the add method of its struct ib_client is called for that 111 device. A consumer must finish all cleanup and free all resources 112 relating to a device before returning from the remove method. 113 114 A consumer is permitted to sleep in its add and remove methods.