Based on kernel version 3.13. Page generated on 2014-01-20 22:00 EST.
1 Introduction 2 ============ 3 4 dm-cache is a device mapper target written by Joe Thornber, Heinz 5 Mauelshagen, and Mike Snitzer. 6 7 It aims to improve performance of a block device (eg, a spindle) by 8 dynamically migrating some of its data to a faster, smaller device 9 (eg, an SSD). 10 11 This device-mapper solution allows us to insert this caching at 12 different levels of the dm stack, for instance above the data device for 13 a thin-provisioning pool. Caching solutions that are integrated more 14 closely with the virtual memory system should give better performance. 15 16 The target reuses the metadata library used in the thin-provisioning 17 library. 18 19 The decision as to what data to migrate and when is left to a plug-in 20 policy module. Several of these have been written as we experiment, 21 and we hope other people will contribute others for specific io 22 scenarios (eg. a vm image server). 23 24 Glossary 25 ======== 26 27 Migration - Movement of the primary copy of a logical block from one 28 device to the other. 29 Promotion - Migration from slow device to fast device. 30 Demotion - Migration from fast device to slow device. 31 32 The origin device always contains a copy of the logical block, which 33 may be out of date or kept in sync with the copy on the cache device 34 (depending on policy). 35 36 Design 37 ====== 38 39 Sub-devices 40 ----------- 41 42 The target is constructed by passing three devices to it (along with 43 other parameters detailed later): 44 45 1. An origin device - the big, slow one. 46 47 2. A cache device - the small, fast one. 48 49 3. A small metadata device - records which blocks are in the cache, 50 which are dirty, and extra hints for use by the policy object. 51 This information could be put on the cache device, but having it 52 separate allows the volume manager to configure it differently, 53 e.g. as a mirror for extra robustness. This metadata device may only 54 be used by a single cache device. 55 56 Fixed block size 57 ---------------- 58 59 The origin is divided up into blocks of a fixed size. This block size 60 is configurable when you first create the cache. Typically we've been 61 using block sizes of 256KB - 1024KB. The block size must be between 64 62 (32KB) and 2097152 (1GB) and a multiple of 64 (32KB). 63 64 Having a fixed block size simplifies the target a lot. But it is 65 something of a compromise. For instance, a small part of a block may be 66 getting hit a lot, yet the whole block will be promoted to the cache. 67 So large block sizes are bad because they waste cache space. And small 68 block sizes are bad because they increase the amount of metadata (both 69 in core and on disk). 70 71 Cache operating modes 72 --------------------- 73 74 The cache has three operating modes: writeback, writethrough and 75 passthrough. 76 77 If writeback, the default, is selected then a write to a block that is 78 cached will go only to the cache and the block will be marked dirty in 79 the metadata. 80 81 If writethrough is selected then a write to a cached block will not 82 complete until it has hit both the origin and cache devices. Clean 83 blocks should remain clean. 84 85 If passthrough is selected, useful when the cache contents are not known 86 to be coherent with the origin device, then all reads are served from 87 the origin device (all reads miss the cache) and all writes are 88 forwarded to the origin device; additionally, write hits cause cache 89 block invalidates. To enable passthrough mode the cache must be clean. 90 Passthrough mode allows a cache device to be activated without having to 91 worry about coherency. Coherency that exists is maintained, although 92 the cache will gradually cool as writes take place. If the coherency of 93 the cache can later be verified, or established through use of the 94 "invalidate_cblocks" message, the cache device can be transitioned to 95 writethrough or writeback mode while still warm. Otherwise, the cache 96 contents can be discarded prior to transitioning to the desired 97 operating mode. 98 99 A simple cleaner policy is provided, which will clean (write back) all 100 dirty blocks in a cache. Useful for decommissioning a cache or when 101 shrinking a cache. Shrinking the cache's fast device requires all cache 102 blocks, in the area of the cache being removed, to be clean. If the 103 area being removed from the cache still contains dirty blocks the resize 104 will fail. Care must be taken to never reduce the volume used for the 105 cache's fast device until the cache is clean. This is of particular 106 importance if writeback mode is used. Writethrough and passthrough 107 modes already maintain a clean cache. Future support to partially clean 108 the cache, above a specified threshold, will allow for keeping the cache 109 warm and in writeback mode during resize. 110 111 Migration throttling 112 -------------------- 113 114 Migrating data between the origin and cache device uses bandwidth. 115 The user can set a throttle to prevent more than a certain amount of 116 migration occurring at any one time. Currently we're not taking any 117 account of normal io traffic going to the devices. More work needs 118 doing here to avoid migrating during those peak io moments. 119 120 For the time being, a message "migration_threshold <#sectors>" 121 can be used to set the maximum number of sectors being migrated, 122 the default being 204800 sectors (or 100MB). 123 124 Updating on-disk metadata 125 ------------------------- 126 127 On-disk metadata is committed every time a REQ_SYNC or REQ_FUA bio is 128 written. If no such requests are made then commits will occur every 129 second. This means the cache behaves like a physical disk that has a 130 write cache (the same is true of the thin-provisioning target). If 131 power is lost you may lose some recent writes. The metadata should 132 always be consistent in spite of any crash. 133 134 The 'dirty' state for a cache block changes far too frequently for us 135 to keep updating it on the fly. So we treat it as a hint. In normal 136 operation it will be written when the dm device is suspended. If the 137 system crashes all cache blocks will be assumed dirty when restarted. 138 139 Per-block policy hints 140 ---------------------- 141 142 Policy plug-ins can store a chunk of data per cache block. It's up to 143 the policy how big this chunk is, but it should be kept small. Like the 144 dirty flags this data is lost if there's a crash so a safe fallback 145 value should always be possible. 146 147 For instance, the 'mq' policy, which is currently the default policy, 148 uses this facility to store the hit count of the cache blocks. If 149 there's a crash this information will be lost, which means the cache 150 may be less efficient until those hit counts are regenerated. 151 152 Policy hints affect performance, not correctness. 153 154 Policy messaging 155 ---------------- 156 157 Policies will have different tunables, specific to each one, so we 158 need a generic way of getting and setting these. Device-mapper 159 messages are used. Refer to cache-policies.txt. 160 161 Discard bitset resolution 162 ------------------------- 163 164 We can avoid copying data during migration if we know the block has 165 been discarded. A prime example of this is when mkfs discards the 166 whole block device. We store a bitset tracking the discard state of 167 blocks. However, we allow this bitset to have a different block size 168 from the cache blocks. This is because we need to track the discard 169 state for all of the origin device (compare with the dirty bitset 170 which is just for the smaller cache device). 171 172 Target interface 173 ================ 174 175 Constructor 176 ----------- 177 178 cache <metadata dev> <cache dev> <origin dev> <block size> 179 <#feature args> [<feature arg>]* 180 <policy> <#policy args> [policy args]* 181 182 metadata dev : fast device holding the persistent metadata 183 cache dev : fast device holding cached data blocks 184 origin dev : slow device holding original data blocks 185 block size : cache unit size in sectors 186 187 #feature args : number of feature arguments passed 188 feature args : writethrough or passthrough (The default is writeback.) 189 190 policy : the replacement policy to use 191 #policy args : an even number of arguments corresponding to 192 key/value pairs passed to the policy 193 policy args : key/value pairs passed to the policy 194 E.g. 'sequential_threshold 1024' 195 See cache-policies.txt for details. 196 197 Optional feature arguments are: 198 writethrough : write through caching that prohibits cache block 199 content from being different from origin block content. 200 Without this argument, the default behaviour is to write 201 back cache block contents later for performance reasons, 202 so they may differ from the corresponding origin blocks. 203 204 passthrough : a degraded mode useful for various cache coherency 205 situations (e.g., rolling back snapshots of 206 underlying storage). Reads and writes always go to 207 the origin. If a write goes to a cached origin 208 block, then the cache block is invalidated. 209 To enable passthrough mode the cache must be clean. 210 211 A policy called 'default' is always registered. This is an alias for 212 the policy we currently think is giving best all round performance. 213 214 As the default policy could vary between kernels, if you are relying on 215 the characteristics of a specific policy, always request it by name. 216 217 Status 218 ------ 219 220 <#used metadata blocks>/<#total metadata blocks> <#read hits> <#read misses> 221 <#write hits> <#write misses> <#demotions> <#promotions> <#blocks in cache> 222 <#dirty> <#features> <features>* <#core args> <core args>* <#policy args> 223 <policy args>* 224 225 #used metadata blocks : Number of metadata blocks used 226 #total metadata blocks : Total number of metadata blocks 227 #read hits : Number of times a READ bio has been mapped 228 to the cache 229 #read misses : Number of times a READ bio has been mapped 230 to the origin 231 #write hits : Number of times a WRITE bio has been mapped 232 to the cache 233 #write misses : Number of times a WRITE bio has been 234 mapped to the origin 235 #demotions : Number of times a block has been removed 236 from the cache 237 #promotions : Number of times a block has been moved to 238 the cache 239 #blocks in cache : Number of blocks resident in the cache 240 #dirty : Number of blocks in the cache that differ 241 from the origin 242 #feature args : Number of feature args to follow 243 feature args : 'writethrough' (optional) 244 #core args : Number of core arguments (must be even) 245 core args : Key/value pairs for tuning the core 246 e.g. migration_threshold 247 #policy args : Number of policy arguments to follow (must be even) 248 policy args : Key/value pairs 249 e.g. 'sequential_threshold 1024 250 251 Messages 252 -------- 253 254 Policies will have different tunables, specific to each one, so we 255 need a generic way of getting and setting these. Device-mapper 256 messages are used. (A sysfs interface would also be possible.) 257 258 The message format is: 259 260 <key> <value> 261 262 E.g. 263 dmsetup message my_cache 0 sequential_threshold 1024 264 265 266 Invalidation is removing an entry from the cache without writing it 267 back. Cache blocks can be invalidated via the invalidate_cblocks 268 message, which takes an arbitrary number of cblock ranges. Each cblock 269 range's end value is "one past the end", meaning 5-10 expresses a range 270 of values from 5 to 9. Each cblock must be expressed as a decimal 271 value, in the future a variant message that takes cblock ranges 272 expressed in hexidecimal may be needed to better support efficient 273 invalidation of larger caches. The cache must be in passthrough mode 274 when invalidate_cblocks is used. 275 276 invalidate_cblocks [<cblock>|<cblock begin>-<cblock end>]* 277 278 E.g. 279 dmsetup message my_cache 0 invalidate_cblocks 2345 3456-4567 5678-6789 280 281 Examples 282 ======== 283 284 The test suite can be found here: 285 286 https://github.com/jthornber/device-mapper-test-suite 287 288 dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ 289 /dev/mapper/ssd /dev/mapper/origin 512 1 writeback default 0' 290 dmsetup create my_cache --table '0 41943040 cache /dev/mapper/metadata \ 291 /dev/mapper/ssd /dev/mapper/origin 1024 1 writeback \ 292 mq 4 sequential_threshold 1024 random_threshold 8'