About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / pktgen.txt




Custom Search

Based on kernel version 4.16.1. Page generated on 2018-04-09 11:53 EST.

1	
2	
3	                  HOWTO for the linux packet generator
4	                  ------------------------------------
5	
6	Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
7	or as a module.  A module is preferred; modprobe pktgen if needed.  Once
8	running, pktgen creates a thread for each CPU with affinity to that CPU.
9	Monitoring and controlling is done via /proc.  It is easiest to select a
10	suitable sample script and configure that.
11	
12	On a dual CPU:
13	
14	ps aux | grep pkt
15	root       129  0.3  0.0     0    0 ?        SW    2003 523:20 [kpktgend_0]
16	root       130  0.3  0.0     0    0 ?        SW    2003 509:50 [kpktgend_1]
17	
18	
19	For monitoring and control pktgen creates:
20		/proc/net/pktgen/pgctrl
21		/proc/net/pktgen/kpktgend_X
22	        /proc/net/pktgen/ethX
23	
24	
25	Tuning NIC for max performance
26	==============================
27	
28	The default NIC settings are (likely) not tuned for pktgen's artificial
29	overload type of benchmarking, as this could hurt the normal use-case.
30	
31	Specifically increasing the TX ring buffer in the NIC:
32	 # ethtool -G ethX tx 1024
33	
34	A larger TX ring can improve pktgen's performance, while it can hurt
35	in the general case, 1) because the TX ring buffer might get larger
36	than the CPU's L1/L2 cache, 2) because it allows more queueing in the
37	NIC HW layer (which is bad for bufferbloat).
38	
39	One should hesitate to conclude that packets/descriptors in the HW
40	TX ring cause delay.  Drivers usually delay cleaning up the
41	ring-buffers for various performance reasons, and packets stalling
42	the TX ring might just be waiting for cleanup.
43	
44	This cleanup issue is specifically the case for the driver ixgbe
45	(Intel 82599 chip).  This driver (ixgbe) combines TX+RX ring cleanups,
46	and the cleanup interval is affected by the ethtool --coalesce setting
47	of parameter "rx-usecs".
48	
49	For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):
50	 # ethtool -C ethX rx-usecs 30
51	
52	
53	Kernel threads
54	==============
55	Pktgen creates a thread for each CPU with affinity to that CPU.
56	Which is controlled through procfile /proc/net/pktgen/kpktgend_X.
57	
58	Example: /proc/net/pktgen/kpktgend_0
59	
60	 Running:
61	 Stopped: eth4@0
62	 Result: OK: add_device=eth4@0
63	
64	Most important are the devices assigned to the thread.
65	
66	The two basic thread commands are:
67	 * add_device DEVICE@NAME -- adds a single device
68	 * rem_device_all         -- remove all associated devices
69	
70	When adding a device to a thread, a corresponding procfile is created
71	which is used for configuring this device. Thus, device names need to
72	be unique.
73	
74	To support adding the same device to multiple threads, which is useful
75	with multi queue NICs, the device naming scheme is extended with "@":
76	 device@something
77	
78	The part after "@" can be anything, but it is custom to use the thread
79	number.
80	
81	Viewing devices
82	===============
83	
84	The Params section holds configured information.  The Current section
85	holds running statistics.  The Result is printed after a run or after
86	interruption.  Example:
87	
88	/proc/net/pktgen/eth4@0
89	
90	 Params: count 100000  min_pkt_size: 60  max_pkt_size: 60
91	     frags: 0  delay: 0  clone_skb: 64  ifname: eth4@0
92	     flows: 0 flowlen: 0
93	     queue_map_min: 0  queue_map_max: 0
94	     dst_min: 192.168.81.2  dst_max:
95	     src_min:   src_max:
96	     src_mac: 90:e2:ba:0a:56:b4 dst_mac: 00:1b:21:3c:9d:f8
97	     udp_src_min: 9  udp_src_max: 109  udp_dst_min: 9  udp_dst_max: 9
98	     src_mac_count: 0  dst_mac_count: 0
99	     Flags: UDPSRC_RND  NO_TIMESTAMP  QUEUE_MAP_CPU
100	 Current:
101	     pkts-sofar: 100000  errors: 0
102	     started: 623913381008us  stopped: 623913396439us idle: 25us
103	     seq_num: 100001  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
104	     cur_saddr: 192.168.8.3  cur_daddr: 192.168.81.2
105	     cur_udp_dst: 9  cur_udp_src: 42
106	     cur_queue_map: 0
107	     flows: 0
108	 Result: OK: 15430(c15405+d25) usec, 100000 (60byte,0frags)
109	  6480562pps 3110Mb/sec (3110669760bps) errors: 0
110	
111	
112	Configuring devices
113	===================
114	This is done via the /proc interface, and most easily done via pgset
115	as defined in the sample scripts.
116	You need to specify PGDEV environment variable to use functions from sample
117	scripts, i.e.:
118	export PGDEV=/proc/net/pktgen/eth4@0
119	source samples/pktgen/functions.sh
120	
121	Examples:
122	
123	 pg_ctrl start           starts injection.
124	 pg_ctrl stop            aborts injection. Also, ^C aborts generator.
125	
126	 pgset "clone_skb 1"     sets the number of copies of the same packet
127	 pgset "clone_skb 0"     use single SKB for all transmits
128	 pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
129	                         packet and update HW tx queue tail pointer once.
130	                         "burst 1" is the default
131	 pgset "pkt_size 9014"   sets packet size to 9014
132	 pgset "frags 5"         packet will consist of 5 fragments
133	 pgset "count 200000"    sets number of packets to send, set to zero
134	                         for continuous sends until explicitly stopped.
135	
136	 pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds
137	
138	 pgset "dst 10.0.0.1"    sets IP destination address
139	                         (BEWARE! This generator is very aggressive!)
140	
141	 pgset "dst_min 10.0.0.1"            Same as dst
142	 pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
143	 pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
144	 pgset "src_max 10.0.0.254"          Set the maximum source IP.
145	 pgset "dst6 fec0::1"     IPV6 destination address
146	 pgset "src6 fec0::2"     IPV6 source address
147	 pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
148	 pgset "srcmac 00:00:00:00:00:00"    sets MAC source address
149	
150	 pgset "queue_map_min 0" Sets the min value of tx queue interval
151	 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
152	                         To select queue 1 of a given device,
153	                         use queue_map_min=1 and queue_map_max=1
154	
155	 pgset "src_mac_count 1" Sets the number of MACs we'll range through.
156	                         The 'minimum' MAC is what you set with srcmac.
157	
158	 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
159	                         The 'minimum' MAC is what you set with dstmac.
160	
161	 pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
162	                         are: IPSRC_RND # IP source is random (between min/max)
163	                              IPDST_RND # IP destination is random
164	                              UDPSRC_RND, UDPDST_RND,
165	                              MACSRC_RND, MACDST_RND
166	                              TXSIZE_RND, IPV6,
167	                              MPLS_RND, VID_RND, SVID_RND
168	                              FLOW_SEQ,
169	                              QUEUE_MAP_RND # queue map random
170	                              QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
171	                              UDPCSUM,
172	                              IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
173	                              NODE_ALLOC # node specific memory allocation
174	                              NO_TIMESTAMP # disable timestamping
175	 pgset 'flag ![name]'    Clear a flag to determine behaviour.
176	                         Note that you might need to use single quote in
177	                         interactive mode, so that your shell wouldn't expand
178	                         the specified flag as a history command.
179	
180	 pgset "spi [SPI_VALUE]" Set specific SA used to transform packet.
181	
182	 pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
183	                         cycle through the port range.
184	
185	 pgset "udp_src_max 9"   set UDP source port max.
186	 pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
187	                         cycle through the port range.
188	 pgset "udp_dst_max 9"   set UDP destination port max.
189	
190	 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
191	                                         outer label=16,middle label=32,
192						 inner label=0 (IPv4 NULL)) Note that
193						 there must be no spaces between the
194						 arguments. Leading zeros are required.
195						 Do not set the bottom of stack bit,
196						 that's done automatically. If you do
197						 set the bottom of stack bit, that
198						 indicates that you want to randomly
199						 generate that address and the flag
200						 MPLS_RND will be turned on. You
201						 can have any mix of random and fixed
202						 labels in the label stack.
203	
204	 pgset "mpls 0"		  turn off mpls (or any invalid argument works too!)
205	
206	 pgset "vlan_id 77"       set VLAN ID 0-4095
207	 pgset "vlan_p 3"         set priority bit 0-7 (default 0)
208	 pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)
209	
210	 pgset "svlan_id 22"      set SVLAN ID 0-4095
211	 pgset "svlan_p 3"        set priority bit 0-7 (default 0)
212	 pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)
213	
214	 pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
215	 pgset "svlan 9999"       > 4095 remove svlan tag
216	
217	
218	 pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
219	 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
220	
221	 pgset "rate 300M"        set rate to 300 Mb/s
222	 pgset "ratep 1000000"    set rate to 1Mpps
223	
224	 pgset "xmit_mode netif_receive"  RX inject into stack netif_receive_skb()
225					  Works with "burst" but not with "clone_skb".
226					  Default xmit_mode is "start_xmit".
227	
228	Sample scripts
229	==============
230	
231	A collection of tutorial scripts and helpers for pktgen is in the
232	samples/pktgen directory. The helper parameters.sh file support easy
233	and consistent parameter parsing across the sample scripts.
234	
235	Usage example and help:
236	 ./pktgen_sample01_simple.sh -i eth4 -m 00:1B:21:3C:9D:F8 -d 192.168.8.2
237	
238	Usage: ./pktgen_sample01_simple.sh [-vx] -i ethX
239	  -i : ($DEV)       output interface/device (required)
240	  -s : ($PKT_SIZE)  packet size
241	  -d : ($DEST_IP)   destination IP
242	  -m : ($DST_MAC)   destination MAC-addr
243	  -t : ($THREADS)   threads to start
244	  -c : ($SKB_CLONE) SKB clones send before alloc new SKB
245	  -b : ($BURST)     HW level bursting of SKBs
246	  -v : ($VERBOSE)   verbose
247	  -x : ($DEBUG)     debug
248	
249	The global variables being set are also listed.  E.g. the required
250	interface/device parameter "-i" sets variable $DEV.  Copy the
251	pktgen_sampleXX scripts and modify them to fit your own needs.
252	
253	The old scripts:
254	
255	pktgen.conf-1-2                  # 1 CPU 2 dev
256	pktgen.conf-1-1-rdos             # 1 CPU 1 dev w. route DoS 
257	pktgen.conf-1-1-ip6              # 1 CPU 1 dev ipv6
258	pktgen.conf-1-1-ip6-rdos         # 1 CPU 1 dev ipv6  w. route DoS
259	pktgen.conf-1-1-flows            # 1 CPU 1 dev multiple flows.
260	
261	
262	Interrupt affinity
263	===================
264	Note that when adding devices to a specific CPU it is a good idea to
265	also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
266	to the same CPU.  This reduces cache bouncing when freeing skbs.
267	
268	Plus using the device flag QUEUE_MAP_CPU, which maps the SKBs TX queue
269	to the running threads CPU (directly from smp_processor_id()).
270	
271	Enable IPsec
272	============
273	Default IPsec transformation with ESP encapsulation plus transport mode
274	can be enabled by simply setting:
275	
276	pgset "flag IPSEC"
277	pgset "flows 1"
278	
279	To avoid breaking existing testbed scripts for using AH type and tunnel mode,
280	you can use "pgset spi SPI_VALUE" to specify which transformation mode
281	to employ.
282	
283	
284	Current commands and configuration options
285	==========================================
286	
287	** Pgcontrol commands:
288	
289	start
290	stop
291	reset
292	
293	** Thread commands:
294	
295	add_device
296	rem_device_all
297	
298	
299	** Device commands:
300	
301	count
302	clone_skb
303	burst
304	debug
305	
306	frags
307	delay
308	
309	src_mac_count
310	dst_mac_count
311	
312	pkt_size
313	min_pkt_size
314	max_pkt_size
315	
316	queue_map_min
317	queue_map_max
318	skb_priority
319	
320	tos           (ipv4)
321	traffic_class (ipv6)
322	
323	mpls
324	
325	udp_src_min
326	udp_src_max
327	
328	udp_dst_min
329	udp_dst_max
330	
331	node
332	
333	flag
334	  IPSRC_RND
335	  IPDST_RND
336	  UDPSRC_RND
337	  UDPDST_RND
338	  MACSRC_RND
339	  MACDST_RND
340	  TXSIZE_RND
341	  IPV6
342	  MPLS_RND
343	  VID_RND
344	  SVID_RND
345	  FLOW_SEQ
346	  QUEUE_MAP_RND
347	  QUEUE_MAP_CPU
348	  UDPCSUM
349	  IPSEC
350	  NODE_ALLOC
351	  NO_TIMESTAMP
352	
353	spi (ipsec)
354	
355	dst_min
356	dst_max
357	
358	src_min
359	src_max
360	
361	dst_mac
362	src_mac
363	
364	clear_counters
365	
366	src6
367	dst6
368	dst6_max
369	dst6_min
370	
371	flows
372	flowlen
373	
374	rate
375	ratep
376	
377	xmit_mode <start_xmit|netif_receive>
378	
379	vlan_cfi
380	vlan_id
381	vlan_p
382	
383	svlan_cfi
384	svlan_id
385	svlan_p
386	
387	
388	References:
389	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
390	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
391	
392	Paper from Linux-Kongress in Erlangen 2004.
393	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
394	
395	Thanks to:
396	Grant Grundler for testing on IA-64 and parisc, Harald Welte,  Lennert Buytenhek
397	Stephen Hemminger, Andi Kleen, Dave Miller and many others.
398	
399	
400	Good luck with the linux net-development.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.