Based on kernel version 4.9. Page generated on 2016-12-21 14:34 EST.
1 ORANGEFS 2 ======== 3 4 OrangeFS is an LGPL userspace scale-out parallel storage system. It is ideal 5 for large storage problems faced by HPC, BigData, Streaming Video, 6 Genomics, Bioinformatics. 7 8 Orangefs, originally called PVFS, was first developed in 1993 by 9 Walt Ligon and Eric Blumer as a parallel file system for Parallel 10 Virtual Machine (PVM) as part of a NASA grant to study the I/O patterns 11 of parallel programs. 12 13 Orangefs features include: 14 15 * Distributes file data among multiple file servers 16 * Supports simultaneous access by multiple clients 17 * Stores file data and metadata on servers using local file system 18 and access methods 19 * Userspace implementation is easy to install and maintain 20 * Direct MPI support 21 * Stateless 22 23 24 MAILING LIST 25 ============ 26 27 http://beowulf-underground.org/mailman/listinfo/pvfs2-users 28 29 30 DOCUMENTATION 31 ============= 32 33 http://www.orangefs.org/documentation/ 34 35 36 USERSPACE FILESYSTEM SOURCE 37 =========================== 38 39 http://www.orangefs.org/download 40 41 Orangefs versions prior to 2.9.3 would not be compatible with the 42 upstream version of the kernel client. 43 44 45 BUILDING THE USERSPACE FILESYSTEM ON A SINGLE SERVER 46 ==================================================== 47 48 When Orangefs is upstream, "--with-kernel" shouldn't be needed, but 49 until then the path to where the kernel with the Orangefs kernel client 50 patch was built is needed to ensure that pvfs2-client-core (the bridge 51 between kernel space and user space) will build properly. You can omit 52 --prefix if you don't care that things are sprinkled around in 53 /usr/local. 54 55 ./configure --prefix=/opt/ofs --with-kernel=/path/to/orangefs/kernel 56 57 make 58 59 make install 60 61 Create an orangefs config file: 62 /opt/ofs/bin/pvfs2-genconfig /etc/pvfs2.conf 63 64 for "Enter hostnames", use the hostname, don't let it default to 65 localhost. 66 67 create a pvfs2tab file in /etc: 68 cat /etc/pvfs2tab 69 tcp://myhostname:3334/orangefs /mymountpoint pvfs2 defaults,noauto 0 0 70 71 create the mount point you specified in the tab file if needed: 72 mkdir /mymountpoint 73 74 bootstrap the server: 75 /opt/ofs/sbin/pvfs2-server /etc/pvfs2.conf -f 76 77 start the server: 78 /opt/osf/sbin/pvfs2-server /etc/pvfs2.conf 79 80 Now the server is running. At this point you might like to 81 prove things are working with: 82 83 /opt/osf/bin/pvfs2-ls /mymountpoint 84 85 You might not want to enforce selinux, it doesn't seem to matter by 86 linux 3.11... 87 88 If stuff seems to be working, turn on the client core: 89 /opt/osf/sbin/pvfs2-client -p /opt/osf/sbin/pvfs2-client-core 90 91 Mount your filesystem. 92 mount -t pvfs2 tcp://myhostname:3334/orangefs /mymountpoint 93 94 95 OPTIONS 96 ======= 97 98 The following mount options are accepted: 99 100 acl 101 Allow the use of Access Control Lists on files and directories. 102 103 intr 104 Some operations between the kernel client and the user space 105 filesystem can be interruptible, such as changes in debug levels 106 and the setting of tunable parameters. 107 108 local_lock 109 Enable posix locking from the perspective of "this" kernel. The 110 default file_operations lock action is to return ENOSYS. Posix 111 locking kicks in if the filesystem is mounted with -o local_lock. 112 Distributed locking is being worked on for the future. 113 114 115 DEBUGGING 116 ========= 117 118 If you want the debug (GOSSIP) statements in a particular 119 source file (inode.c for example) go to syslog: 120 121 echo inode > /sys/kernel/debug/orangefs/kernel-debug 122 123 No debugging (the default): 124 125 echo none > /sys/kernel/debug/orangefs/kernel-debug 126 127 Debugging from several source files: 128 129 echo inode,dir > /sys/kernel/debug/orangefs/kernel-debug 130 131 All debugging: 132 133 echo all > /sys/kernel/debug/orangefs/kernel-debug 134 135 Get a list of all debugging keywords: 136 137 cat /sys/kernel/debug/orangefs/debug-help 138 139 140 PROTOCOL BETWEEN KERNEL MODULE AND USERSPACE 141 ============================================ 142 143 Orangefs is a user space filesystem and an associated kernel module. 144 We'll just refer to the user space part of Orangefs as "userspace" 145 from here on out. Orangefs descends from PVFS, and userspace code 146 still uses PVFS for function and variable names. Userspace typedefs 147 many of the important structures. Function and variable names in 148 the kernel module have been transitioned to "orangefs", and The Linux 149 Coding Style avoids typedefs, so kernel module structures that 150 correspond to userspace structures are not typedefed. 151 152 The kernel module implements a pseudo device that userspace 153 can read from and write to. Userspace can also manipulate the 154 kernel module through the pseudo device with ioctl. 155 156 THE BUFMAP: 157 158 At startup userspace allocates two page-size-aligned (posix_memalign) 159 mlocked memory buffers, one is used for IO and one is used for readdir 160 operations. The IO buffer is 41943040 bytes and the readdir buffer is 161 4194304 bytes. Each buffer contains logical chunks, or partitions, and 162 a pointer to each buffer is added to its own PVFS_dev_map_desc structure 163 which also describes its total size, as well as the size and number of 164 the partitions. 165 166 A pointer to the IO buffer's PVFS_dev_map_desc structure is sent to a 167 mapping routine in the kernel module with an ioctl. The structure is 168 copied from user space to kernel space with copy_from_user and is used 169 to initialize the kernel module's "bufmap" (struct orangefs_bufmap), which 170 then contains: 171 172 * refcnt - a reference counter 173 * desc_size - PVFS2_BUFMAP_DEFAULT_DESC_SIZE (4194304) - the IO buffer's 174 partition size, which represents the filesystem's block size and 175 is used for s_blocksize in super blocks. 176 * desc_count - PVFS2_BUFMAP_DEFAULT_DESC_COUNT (10) - the number of 177 partitions in the IO buffer. 178 * desc_shift - log2(desc_size), used for s_blocksize_bits in super blocks. 179 * total_size - the total size of the IO buffer. 180 * page_count - the number of 4096 byte pages in the IO buffer. 181 * page_array - a pointer to page_count * (sizeof(struct page*)) bytes 182 of kcalloced memory. This memory is used as an array of pointers 183 to each of the pages in the IO buffer through a call to get_user_pages. 184 * desc_array - a pointer to desc_count * (sizeof(struct orangefs_bufmap_desc)) 185 bytes of kcalloced memory. This memory is further intialized: 186 187 user_desc is the kernel's copy of the IO buffer's ORANGEFS_dev_map_desc 188 structure. user_desc->ptr points to the IO buffer. 189 190 pages_per_desc = bufmap->desc_size / PAGE_SIZE 191 offset = 0 192 193 bufmap->desc_array.page_array = &bufmap->page_array[offset] 194 bufmap->desc_array.array_count = pages_per_desc = 1024 195 bufmap->desc_array.uaddr = (user_desc->ptr) + (0 * 1024 * 4096) 196 offset += 1024 197 . 198 . 199 . 200 bufmap->desc_array.page_array = &bufmap->page_array[offset] 201 bufmap->desc_array.array_count = pages_per_desc = 1024 202 bufmap->desc_array.uaddr = (user_desc->ptr) + 203 (9 * 1024 * 4096) 204 offset += 1024 205 206 * buffer_index_array - a desc_count sized array of ints, used to 207 indicate which of the IO buffer's partitions are available to use. 208 * buffer_index_lock - a spinlock to protect buffer_index_array during update. 209 * readdir_index_array - a five (ORANGEFS_READDIR_DEFAULT_DESC_COUNT) element 210 int array used to indicate which of the readdir buffer's partitions are 211 available to use. 212 * readdir_index_lock - a spinlock to protect readdir_index_array during 213 update. 214 215 OPERATIONS: 216 217 The kernel module builds an "op" (struct orangefs_kernel_op_s) when it 218 needs to communicate with userspace. Part of the op contains the "upcall" 219 which expresses the request to userspace. Part of the op eventually 220 contains the "downcall" which expresses the results of the request. 221 222 The slab allocator is used to keep a cache of op structures handy. 223 224 At init time the kernel module defines and initializes a request list 225 and an in_progress hash table to keep track of all the ops that are 226 in flight at any given time. 227 228 Ops are stateful: 229 230 * unknown - op was just initialized 231 * waiting - op is on request_list (upward bound) 232 * inprogr - op is in progress (waiting for downcall) 233 * serviced - op has matching downcall; ok 234 * purged - op has to start a timer since client-core 235 exited uncleanly before servicing op 236 * given up - submitter has given up waiting for it 237 238 When some arbitrary userspace program needs to perform a 239 filesystem operation on Orangefs (readdir, I/O, create, whatever) 240 an op structure is initialized and tagged with a distinguishing ID 241 number. The upcall part of the op is filled out, and the op is 242 passed to the "service_operation" function. 243 244 Service_operation changes the op's state to "waiting", puts 245 it on the request list, and signals the Orangefs file_operations.poll 246 function through a wait queue. Userspace is polling the pseudo-device 247 and thus becomes aware of the upcall request that needs to be read. 248 249 When the Orangefs file_operations.read function is triggered, the 250 request list is searched for an op that seems ready-to-process. 251 The op is removed from the request list. The tag from the op and 252 the filled-out upcall struct are copy_to_user'ed back to userspace. 253 254 If any of these (and some additional protocol) copy_to_users fail, 255 the op's state is set to "waiting" and the op is added back to 256 the request list. Otherwise, the op's state is changed to "in progress", 257 and the op is hashed on its tag and put onto the end of a list in the 258 in_progress hash table at the index the tag hashed to. 259 260 When userspace has assembled the response to the upcall, it 261 writes the response, which includes the distinguishing tag, back to 262 the pseudo device in a series of io_vecs. This triggers the Orangefs 263 file_operations.write_iter function to find the op with the associated 264 tag and remove it from the in_progress hash table. As long as the op's 265 state is not "canceled" or "given up", its state is set to "serviced". 266 The file_operations.write_iter function returns to the waiting vfs, 267 and back to service_operation through wait_for_matching_downcall. 268 269 Service operation returns to its caller with the op's downcall 270 part (the response to the upcall) filled out. 271 272 The "client-core" is the bridge between the kernel module and 273 userspace. The client-core is a daemon. The client-core has an 274 associated watchdog daemon. If the client-core is ever signaled 275 to die, the watchdog daemon restarts the client-core. Even though 276 the client-core is restarted "right away", there is a period of 277 time during such an event that the client-core is dead. A dead client-core 278 can't be triggered by the Orangefs file_operations.poll function. 279 Ops that pass through service_operation during a "dead spell" can timeout 280 on the wait queue and one attempt is made to recycle them. Obviously, 281 if the client-core stays dead too long, the arbitrary userspace processes 282 trying to use Orangefs will be negatively affected. Waiting ops 283 that can't be serviced will be removed from the request list and 284 have their states set to "given up". In-progress ops that can't 285 be serviced will be removed from the in_progress hash table and 286 have their states set to "given up". 287 288 Readdir and I/O ops are atypical with respect to their payloads. 289 290 - readdir ops use the smaller of the two pre-allocated pre-partitioned 291 memory buffers. The readdir buffer is only available to userspace. 292 The kernel module obtains an index to a free partition before launching 293 a readdir op. Userspace deposits the results into the indexed partition 294 and then writes them to back to the pvfs device. 295 296 - io (read and write) ops use the larger of the two pre-allocated 297 pre-partitioned memory buffers. The IO buffer is accessible from 298 both userspace and the kernel module. The kernel module obtains an 299 index to a free partition before launching an io op. The kernel module 300 deposits write data into the indexed partition, to be consumed 301 directly by userspace. Userspace deposits the results of read 302 requests into the indexed partition, to be consumed directly 303 by the kernel module. 304 305 Responses to kernel requests are all packaged in pvfs2_downcall_t 306 structs. Besides a few other members, pvfs2_downcall_t contains a 307 union of structs, each of which is associated with a particular 308 response type. 309 310 The several members outside of the union are: 311 - int32_t type - type of operation. 312 - int32_t status - return code for the operation. 313 - int64_t trailer_size - 0 unless readdir operation. 314 - char *trailer_buf - initialized to NULL, used during readdir operations. 315 316 The appropriate member inside the union is filled out for any 317 particular response. 318 319 PVFS2_VFS_OP_FILE_IO 320 fill a pvfs2_io_response_t 321 322 PVFS2_VFS_OP_LOOKUP 323 fill a PVFS_object_kref 324 325 PVFS2_VFS_OP_CREATE 326 fill a PVFS_object_kref 327 328 PVFS2_VFS_OP_SYMLINK 329 fill a PVFS_object_kref 330 331 PVFS2_VFS_OP_GETATTR 332 fill in a PVFS_sys_attr_s (tons of stuff the kernel doesn't need) 333 fill in a string with the link target when the object is a symlink. 334 335 PVFS2_VFS_OP_MKDIR 336 fill a PVFS_object_kref 337 338 PVFS2_VFS_OP_STATFS 339 fill a pvfs2_statfs_response_t with useless info <g>. It is hard for 340 us to know, in a timely fashion, these statistics about our 341 distributed network filesystem. 342 343 PVFS2_VFS_OP_FS_MOUNT 344 fill a pvfs2_fs_mount_response_t which is just like a PVFS_object_kref 345 except its members are in a different order and "__pad1" is replaced 346 with "id". 347 348 PVFS2_VFS_OP_GETXATTR 349 fill a pvfs2_getxattr_response_t 350 351 PVFS2_VFS_OP_LISTXATTR 352 fill a pvfs2_listxattr_response_t 353 354 PVFS2_VFS_OP_PARAM 355 fill a pvfs2_param_response_t 356 357 PVFS2_VFS_OP_PERF_COUNT 358 fill a pvfs2_perf_count_response_t 359 360 PVFS2_VFS_OP_FSKEY 361 file a pvfs2_fs_key_response_t 362 363 PVFS2_VFS_OP_READDIR 364 jamb everything needed to represent a pvfs2_readdir_response_t into 365 the readdir buffer descriptor specified in the upcall. 366 367 Userspace uses writev() on /dev/pvfs2-req to pass responses to the requests 368 made by the kernel side. 369 370 A buffer_list containing: 371 - a pointer to the prepared response to the request from the 372 kernel (struct pvfs2_downcall_t). 373 - and also, in the case of a readdir request, a pointer to a 374 buffer containing descriptors for the objects in the target 375 directory. 376 ... is sent to the function (PINT_dev_write_list) which performs 377 the writev. 378 379 PINT_dev_write_list has a local iovec array: struct iovec io_array; 380 381 The first four elements of io_array are initialized like this for all 382 responses: 383 384 io_array.iov_base = address of local variable "proto_ver" (int32_t) 385 io_array.iov_len = sizeof(int32_t) 386 387 io_array.iov_base = address of global variable "pdev_magic" (int32_t) 388 io_array.iov_len = sizeof(int32_t) 389 390 io_array.iov_base = address of parameter "tag" (PVFS_id_gen_t) 391 io_array.iov_len = sizeof(int64_t) 392 393 io_array.iov_base = address of out_downcall member (pvfs2_downcall_t) 394 of global variable vfs_request (vfs_request_t) 395 io_array.iov_len = sizeof(pvfs2_downcall_t) 396 397 Readdir responses initialize the fifth element io_array like this: 398 399 io_array.iov_base = contents of member trailer_buf (char *) 400 from out_downcall member of global variable 401 vfs_request 402 io_array.iov_len = contents of member trailer_size (PVFS_size) 403 from out_downcall member of global variable 404 vfs_request 405 406 Orangefs exploits the dcache in order to avoid sending redundant 407 requests to userspace. We keep object inode attributes up-to-date with 408 orangefs_inode_getattr. Orangefs_inode_getattr uses two arguments to 409 help it decide whether or not to update an inode: "new" and "bypass". 410 Orangefs keeps private data in an object's inode that includes a short 411 timeout value, getattr_time, which allows any iteration of 412 orangefs_inode_getattr to know how long it has been since the inode was 413 updated. When the object is not new (new == 0) and the bypass flag is not 414 set (bypass == 0) orangefs_inode_getattr returns without updating the inode 415 if getattr_time has not timed out. Getattr_time is updated each time the 416 inode is updated. 417 418 Creation of a new object (file, dir, sym-link) includes the evaluation of 419 its pathname, resulting in a negative directory entry for the object. 420 A new inode is allocated and associated with the dentry, turning it from 421 a negative dentry into a "productive full member of society". Orangefs 422 obtains the new inode from Linux with new_inode() and associates 423 the inode with the dentry by sending the pair back to Linux with 424 d_instantiate(). 425 426 The evaluation of a pathname for an object resolves to its corresponding 427 dentry. If there is no corresponding dentry, one is created for it in 428 the dcache. Whenever a dentry is modified or verified Orangefs stores a 429 short timeout value in the dentry's d_time, and the dentry will be trusted 430 for that amount of time. Orangefs is a network filesystem, and objects 431 can potentially change out-of-band with any particular Orangefs kernel module 432 instance, so trusting a dentry is risky. The alternative to trusting 433 dentries is to always obtain the needed information from userspace - at 434 least a trip to the client-core, maybe to the servers. Obtaining information 435 from a dentry is cheap, obtaining it from userspace is relatively expensive, 436 hence the motivation to use the dentry when possible. 437 438 The timeout values d_time and getattr_time are jiffy based, and the 439 code is designed to avoid the jiffy-wrap problem: 440 441 "In general, if the clock may have wrapped around more than once, there 442 is no way to tell how much time has elapsed. However, if the times t1 443 and t2 are known to be fairly close, we can reliably compute the 444 difference in a way that takes into account the possibility that the 445 clock may have wrapped between times." 446 447 from course notes by instructor Andy Wang