About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / vrf.txt




Custom Search

Based on kernel version 4.9. Page generated on 2016-12-21 14:36 EST.

1	Virtual Routing and Forwarding (VRF)
2	====================================
3	The VRF device combined with ip rules provides the ability to create virtual
4	routing and forwarding domains (aka VRFs, VRF-lite to be specific) in the
5	Linux network stack. One use case is the multi-tenancy problem where each
6	tenant has their own unique routing tables and in the very least need
7	different default gateways.
8	
9	Processes can be "VRF aware" by binding a socket to the VRF device. Packets
10	through the socket then use the routing table associated with the VRF
11	device. An important feature of the VRF device implementation is that it
12	impacts only Layer 3 and above so L2 tools (e.g., LLDP) are not affected
13	(ie., they do not need to be run in each VRF). The design also allows
14	the use of higher priority ip rules (Policy Based Routing, PBR) to take
15	precedence over the VRF device rules directing specific traffic as desired.
16	
17	In addition, VRF devices allow VRFs to be nested within namespaces. For
18	example network namespaces provide separation of network interfaces at the
19	device layer, VLANs on the interfaces within a namespace provide L2 separation
20	and then VRF devices provide L3 separation.
21	
22	Design
23	------
24	A VRF device is created with an associated route table. Network interfaces
25	are then enslaved to a VRF device:
26	
27	         +-----------------------------+
28	         |           vrf-blue          |  ===> route table 10
29	         +-----------------------------+
30	            |        |            |
31	         +------+ +------+     +-------------+
32	         | eth1 | | eth2 | ... |    bond1    |
33	         +------+ +------+     +-------------+
34	                                  |       |
35	                              +------+ +------+
36	                              | eth8 | | eth9 |
37	                              +------+ +------+
38	
39	Packets received on an enslaved device and are switched to the VRF device
40	in the IPv4 and IPv6 processing stacks giving the impression that packets
41	flow through the VRF device. Similarly on egress routing rules are used to
42	send packets to the VRF device driver before getting sent out the actual
43	interface. This allows tcpdump on a VRF device to capture all packets into
44	and out of the VRF as a whole.[1] Similarly, netfilter[2] and tc rules can be
45	applied using the VRF device to specify rules that apply to the VRF domain
46	as a whole.
47	
48	[1] Packets in the forwarded state do not flow through the device, so those
49	    packets are not seen by tcpdump. Will revisit this limitation in a
50	    future release.
51	
52	[2] Iptables on ingress supports PREROUTING with skb->dev set to the real
53	    ingress device and both INPUT and PREROUTING rules with skb->dev set to
54	    the VRF device. For egress POSTROUTING and OUTPUT rules can be written
55	    using either the VRF device or real egress device.
56	
57	Setup
58	-----
59	1. VRF device is created with an association to a FIB table.
60	   e.g, ip link add vrf-blue type vrf table 10
61	        ip link set dev vrf-blue up
62	
63	2. An l3mdev FIB rule directs lookups to the table associated with the device.
64	   A single l3mdev rule is sufficient for all VRFs. The VRF device adds the
65	   l3mdev rule for IPv4 and IPv6 when the first device is created with a
66	   default preference of 1000. Users may delete the rule if desired and add
67	   with a different priority or install per-VRF rules.
68	
69	   Prior to the v4.8 kernel iif and oif rules are needed for each VRF device:
70	       ip ru add oif vrf-blue table 10
71	       ip ru add iif vrf-blue table 10
72	
73	3. Set the default route for the table (and hence default route for the VRF).
74	       ip route add table 10 unreachable default
75	
76	4. Enslave L3 interfaces to a VRF device.
77	       ip link set dev eth1 master vrf-blue
78	
79	   Local and connected routes for enslaved devices are automatically moved to
80	   the table associated with VRF device. Any additional routes depending on
81	   the enslaved device are dropped and will need to be reinserted to the VRF
82	   FIB table following the enslavement.
83	
84	   The IPv6 sysctl option keep_addr_on_down can be enabled to keep IPv6 global
85	   addresses as VRF enslavement changes.
86	       sysctl -w net.ipv6.conf.all.keep_addr_on_down=1
87	
88	5. Additional VRF routes are added to associated table.
89	       ip route add table 10 ...
90	
91	
92	Applications
93	------------
94	Applications that are to work within a VRF need to bind their socket to the
95	VRF device:
96	
97	    setsockopt(sd, SOL_SOCKET, SO_BINDTODEVICE, dev, strlen(dev)+1);
98	
99	or to specify the output device using cmsg and IP_PKTINFO.
100	
101	TCP services running in the default VRF context (ie., not bound to any VRF
102	device) can work across all VRF domains by enabling the tcp_l3mdev_accept
103	sysctl option:
104	    sysctl -w net.ipv4.tcp_l3mdev_accept=1
105	
106	netfilter rules on the VRF device can be used to limit access to services
107	running in the default VRF context as well.
108	
109	The default VRF does not have limited scope with respect to port bindings.
110	That is, if a process does a wildcard bind to a port in the default VRF it
111	owns the port across all VRF domains within the network namespace.
112	
113	################################################################################
114	
115	Using iproute2 for VRFs
116	=======================
117	iproute2 supports the vrf keyword as of v4.7. For backwards compatibility this
118	section lists both commands where appropriate -- with the vrf keyword and the
119	older form without it.
120	
121	1. Create a VRF
122	
123	   To instantiate a VRF device and associate it with a table:
124	       $ ip link add dev NAME type vrf table ID
125	
126	   As of v4.8 the kernel supports the l3mdev FIB rule where a single rule
127	   covers all VRFs. The l3mdev rule is created for IPv4 and IPv6 on first
128	   device create.
129	
130	2. List VRFs
131	
132	   To list VRFs that have been created:
133	       $ ip [-d] link show type vrf
134	         NOTE: The -d option is needed to show the table id
135	
136	   For example:
137	   $ ip -d link show type vrf
138	   11: mgmt: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
139	       link/ether 72:b3:ba:91:e2:24 brd ff:ff:ff:ff:ff:ff promiscuity 0
140	       vrf table 1 addrgenmode eui64
141	   12: red: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
142	       link/ether b6:6f:6e:f6:da:73 brd ff:ff:ff:ff:ff:ff promiscuity 0
143	       vrf table 10 addrgenmode eui64
144	   13: blue: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
145	       link/ether 36:62:e8:7d:bb:8c brd ff:ff:ff:ff:ff:ff promiscuity 0
146	       vrf table 66 addrgenmode eui64
147	   14: green: <NOARP,MASTER,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP mode DEFAULT group default qlen 1000
148	       link/ether e6:28:b8:63:70:bb brd ff:ff:ff:ff:ff:ff promiscuity 0
149	       vrf table 81 addrgenmode eui64
150	
151	
152	   Or in brief output:
153	
154	   $ ip -br link show type vrf
155	   mgmt         UP             72:b3:ba:91:e2:24 <NOARP,MASTER,UP,LOWER_UP>
156	   red          UP             b6:6f:6e:f6:da:73 <NOARP,MASTER,UP,LOWER_UP>
157	   blue         UP             36:62:e8:7d:bb:8c <NOARP,MASTER,UP,LOWER_UP>
158	   green        UP             e6:28:b8:63:70:bb <NOARP,MASTER,UP,LOWER_UP>
159	
160	
161	3. Assign a Network Interface to a VRF
162	
163	   Network interfaces are assigned to a VRF by enslaving the netdevice to a
164	   VRF device:
165	       $ ip link set dev NAME master NAME
166	
167	   On enslavement connected and local routes are automatically moved to the
168	   table associated with the VRF device.
169	
170	   For example:
171	   $ ip link set dev eth0 master mgmt
172	
173	
174	4. Show Devices Assigned to a VRF
175	
176	   To show devices that have been assigned to a specific VRF add the master
177	   option to the ip command:
178	       $ ip link show vrf NAME
179	       $ ip link show master NAME
180	
181	   For example:
182	   $ ip link show vrf red
183	   3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
184	       link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
185	   4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP mode DEFAULT group default qlen 1000
186	       link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
187	   7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN mode DEFAULT group default qlen 1000
188	       link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
189	
190	
191	   Or using the brief output:
192	   $ ip -br link show vrf red
193	   eth1             UP             02:00:00:00:02:02 <BROADCAST,MULTICAST,UP,LOWER_UP>
194	   eth2             UP             02:00:00:00:02:03 <BROADCAST,MULTICAST,UP,LOWER_UP>
195	   eth5             DOWN           02:00:00:00:02:06 <BROADCAST,MULTICAST>
196	
197	
198	5. Show Neighbor Entries for a VRF
199	
200	   To list neighbor entries associated with devices enslaved to a VRF device
201	   add the master option to the ip command:
202	       $ ip [-6] neigh show vrf NAME
203	       $ ip [-6] neigh show master NAME
204	
205	   For example:
206	   $  ip neigh show vrf red
207	   10.2.1.254 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
208	   10.2.2.254 dev eth2 lladdr 5e:54:01:6a:ee:80 REACHABLE
209	
210	   $ ip -6 neigh show vrf red
211	   2002:1::64 dev eth1 lladdr a6:d9:c7:4f:06:23 REACHABLE
212	
213	
214	6. Show Addresses for a VRF
215	
216	   To show addresses for interfaces associated with a VRF add the master
217	   option to the ip command:
218	       $ ip addr show vrf NAME
219	       $ ip addr show master NAME
220	
221	   For example:
222	   $ ip addr show vrf red
223	   3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
224	       link/ether 02:00:00:00:02:02 brd ff:ff:ff:ff:ff:ff
225	       inet 10.2.1.2/24 brd 10.2.1.255 scope global eth1
226	          valid_lft forever preferred_lft forever
227	       inet6 2002:1::2/120 scope global
228	          valid_lft forever preferred_lft forever
229	       inet6 fe80::ff:fe00:202/64 scope link
230	          valid_lft forever preferred_lft forever
231	   4: eth2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master red state UP group default qlen 1000
232	       link/ether 02:00:00:00:02:03 brd ff:ff:ff:ff:ff:ff
233	       inet 10.2.2.2/24 brd 10.2.2.255 scope global eth2
234	          valid_lft forever preferred_lft forever
235	       inet6 2002:2::2/120 scope global
236	          valid_lft forever preferred_lft forever
237	       inet6 fe80::ff:fe00:203/64 scope link
238	          valid_lft forever preferred_lft forever
239	   7: eth5: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master red state DOWN group default qlen 1000
240	       link/ether 02:00:00:00:02:06 brd ff:ff:ff:ff:ff:ff
241	
242	   Or in brief format:
243	   $ ip -br addr show vrf red
244	   eth1             UP             10.2.1.2/24 2002:1::2/120 fe80::ff:fe00:202/64
245	   eth2             UP             10.2.2.2/24 2002:2::2/120 fe80::ff:fe00:203/64
246	   eth5             DOWN
247	
248	
249	7. Show Routes for a VRF
250	
251	   To show routes for a VRF use the ip command to display the table associated
252	   with the VRF device:
253	       $ ip [-6] route show vrf NAME
254	       $ ip [-6] route show table ID
255	
256	   For example:
257	   $ ip route show vrf red
258	   prohibit default
259	   broadcast 10.2.1.0 dev eth1  proto kernel  scope link  src 10.2.1.2
260	   10.2.1.0/24 dev eth1  proto kernel  scope link  src 10.2.1.2
261	   local 10.2.1.2 dev eth1  proto kernel  scope host  src 10.2.1.2
262	   broadcast 10.2.1.255 dev eth1  proto kernel  scope link  src 10.2.1.2
263	   broadcast 10.2.2.0 dev eth2  proto kernel  scope link  src 10.2.2.2
264	   10.2.2.0/24 dev eth2  proto kernel  scope link  src 10.2.2.2
265	   local 10.2.2.2 dev eth2  proto kernel  scope host  src 10.2.2.2
266	   broadcast 10.2.2.255 dev eth2  proto kernel  scope link  src 10.2.2.2
267	
268	   $ ip -6 route show vrf red
269	   local 2002:1:: dev lo  proto none  metric 0  pref medium
270	   local 2002:1::2 dev lo  proto none  metric 0  pref medium
271	   2002:1::/120 dev eth1  proto kernel  metric 256  pref medium
272	   local 2002:2:: dev lo  proto none  metric 0  pref medium
273	   local 2002:2::2 dev lo  proto none  metric 0  pref medium
274	   2002:2::/120 dev eth2  proto kernel  metric 256  pref medium
275	   local fe80:: dev lo  proto none  metric 0  pref medium
276	   local fe80:: dev lo  proto none  metric 0  pref medium
277	   local fe80::ff:fe00:202 dev lo  proto none  metric 0  pref medium
278	   local fe80::ff:fe00:203 dev lo  proto none  metric 0  pref medium
279	   fe80::/64 dev eth1  proto kernel  metric 256  pref medium
280	   fe80::/64 dev eth2  proto kernel  metric 256  pref medium
281	   ff00::/8 dev red  metric 256  pref medium
282	   ff00::/8 dev eth1  metric 256  pref medium
283	   ff00::/8 dev eth2  metric 256  pref medium
284	
285	
286	8. Route Lookup for a VRF
287	
288	   A test route lookup can be done for a VRF:
289	       $ ip [-6] route get vrf NAME ADDRESS
290	       $ ip [-6] route get oif NAME ADDRESS
291	
292	   For example:
293	   $ ip route get 10.2.1.40 vrf red
294	   10.2.1.40 dev eth1  table red  src 10.2.1.2
295	       cache
296	
297	   $ ip -6 route get 2002:1::32 vrf red
298	   2002:1::32 from :: dev eth1  table red  proto kernel  src 2002:1::2  metric 256  pref medium
299	
300	
301	9. Removing Network Interface from a VRF
302	
303	   Network interfaces are removed from a VRF by breaking the enslavement to
304	   the VRF device:
305	       $ ip link set dev NAME nomaster
306	
307	   Connected routes are moved back to the default table and local entries are
308	   moved to the local table.
309	
310	   For example:
311	   $ ip link set dev eth0 nomaster
312	
313	--------------------------------------------------------------------------------
314	
315	Commands used in this example:
316	
317	cat >> /etc/iproute2/rt_tables.d/vrf.conf <<EOF
318	1  mgmt
319	10 red
320	66 blue
321	81 green
322	EOF
323	
324	function vrf_create
325	{
326	    VRF=$1
327	    TBID=$2
328	
329	    # create VRF device
330	    ip link add ${VRF} type vrf table ${TBID}
331	
332	    if [ "${VRF}" != "mgmt" ]; then
333	        ip route add table ${TBID} unreachable default
334	    fi
335	    ip link set dev ${VRF} up
336	}
337	
338	vrf_create mgmt 1
339	ip link set dev eth0 master mgmt
340	
341	vrf_create red 10
342	ip link set dev eth1 master red
343	ip link set dev eth2 master red
344	ip link set dev eth5 master red
345	
346	vrf_create blue 66
347	ip link set dev eth3 master blue
348	
349	vrf_create green 81
350	ip link set dev eth4 master green
351	
352	
353	Interface addresses from /etc/network/interfaces:
354	auto eth0
355	iface eth0 inet static
356	      address 10.0.0.2
357	      netmask 255.255.255.0
358	      gateway 10.0.0.254
359	
360	iface eth0 inet6 static
361	      address 2000:1::2
362	      netmask 120
363	
364	auto eth1
365	iface eth1 inet static
366	      address 10.2.1.2
367	      netmask 255.255.255.0
368	
369	iface eth1 inet6 static
370	      address 2002:1::2
371	      netmask 120
372	
373	auto eth2
374	iface eth2 inet static
375	      address 10.2.2.2
376	      netmask 255.255.255.0
377	
378	iface eth2 inet6 static
379	      address 2002:2::2
380	      netmask 120
381	
382	auto eth3
383	iface eth3 inet static
384	      address 10.2.3.2
385	      netmask 255.255.255.0
386	
387	iface eth3 inet6 static
388	      address 2002:3::2
389	      netmask 120
390	
391	auto eth4
392	iface eth4 inet static
393	      address 10.2.4.2
394	      netmask 255.255.255.0
395	
396	iface eth4 inet6 static
397	      address 2002:4::2
398	      netmask 120
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.