About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / pktgen.txt




Custom Search

Based on kernel version 4.1. Page generated on 2015-06-28 12:13 EST.

1	
2	
3	                  HOWTO for the linux packet generator 
4	                  ------------------------------------
5	
6	Enable CONFIG_NET_PKTGEN to compile and build pktgen either in-kernel
7	or as a module.  A module is preferred; modprobe pktgen if needed.  Once
8	running, pktgen creates a thread for each CPU with affinity to that CPU.
9	Monitoring and controlling is done via /proc.  It is easiest to select a
10	suitable sample script and configure that.
11	
12	On a dual CPU:
13	
14	ps aux | grep pkt
15	root       129  0.3  0.0     0    0 ?        SW    2003 523:20 [pktgen/0]
16	root       130  0.3  0.0     0    0 ?        SW    2003 509:50 [pktgen/1]
17	
18	
19	For monitoring and control pktgen creates:
20		/proc/net/pktgen/pgctrl
21		/proc/net/pktgen/kpktgend_X
22	        /proc/net/pktgen/ethX
23	
24	
25	Tuning NIC for max performance
26	==============================
27	
28	The default NIC settings are (likely) not tuned for pktgen's artificial
29	overload type of benchmarking, as this could hurt the normal use-case.
30	
31	Specifically increasing the TX ring buffer in the NIC:
32	 # ethtool -G ethX tx 1024
33	
34	A larger TX ring can improve pktgen's performance, while it can hurt
35	in the general case, 1) because the TX ring buffer might get larger
36	than the CPU's L1/L2 cache, 2) because it allows more queueing in the
37	NIC HW layer (which is bad for bufferbloat).
38	
39	One should hesitate to conclude that packets/descriptors in the HW
40	TX ring cause delay.  Drivers usually delay cleaning up the
41	ring-buffers for various performance reasons, and packets stalling
42	the TX ring might just be waiting for cleanup.
43	
44	This cleanup issue is specifically the case for the driver ixgbe
45	(Intel 82599 chip).  This driver (ixgbe) combines TX+RX ring cleanups,
46	and the cleanup interval is affected by the ethtool --coalesce setting
47	of parameter "rx-usecs".
48	
49	For ixgbe use e.g. "30" resulting in approx 33K interrupts/sec (1/30*10^6):
50	 # ethtool -C ethX rx-usecs 30
51	
52	
53	Viewing threads
54	===============
55	/proc/net/pktgen/kpktgend_0 
56	Name: kpktgend_0  max_before_softirq: 10000
57	Running: 
58	Stopped: eth1 
59	Result: OK: max_before_softirq=10000
60	
61	Most important are the devices assigned to the thread.  Note that a
62	device can only belong to one thread.
63	
64	
65	Viewing devices
66	===============
67	
68	The Params section holds configured information.  The Current section
69	holds running statistics.  The Result is printed after a run or after
70	interruption.  Example:
71	
72	/proc/net/pktgen/eth1       
73	
74	Params: count 10000000  min_pkt_size: 60  max_pkt_size: 60
75	     frags: 0  delay: 0  clone_skb: 1000000  ifname: eth1
76	     flows: 0 flowlen: 0
77	     dst_min: 10.10.11.2  dst_max: 
78	     src_min:   src_max: 
79	     src_mac: 00:00:00:00:00:00  dst_mac: 00:04:23:AC:FD:82
80	     udp_src_min: 9  udp_src_max: 9  udp_dst_min: 9  udp_dst_max: 9
81	     src_mac_count: 0  dst_mac_count: 0 
82	     Flags: 
83	Current:
84	     pkts-sofar: 10000000  errors: 39664
85	     started: 1103053986245187us  stopped: 1103053999346329us idle: 880401us
86	     seq_num: 10000011  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
87	     cur_saddr: 0x10a0a0a  cur_daddr: 0x20b0a0a
88	     cur_udp_dst: 9  cur_udp_src: 9
89	     flows: 0
90	Result: OK: 13101142(c12220741+d880401) usec, 10000000 (60byte,0frags)
91	  763292pps 390Mb/sec (390805504bps) errors: 39664
92	
93	Configuring threads and devices
94	================================
95	This is done via the /proc interface, and most easily done via pgset
96	as defined in the sample scripts.
97	
98	Examples:
99	
100	 pgset "clone_skb 1"     sets the number of copies of the same packet
101	 pgset "clone_skb 0"     use single SKB for all transmits
102	 pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
103	                         packet and update HW tx queue tail pointer once.
104	                         "burst 1" is the default
105	 pgset "pkt_size 9014"   sets packet size to 9014
106	 pgset "frags 5"         packet will consist of 5 fragments
107	 pgset "count 200000"    sets number of packets to send, set to zero
108	                         for continuous sends until explicitly stopped.
109	
110	 pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds
111	
112	 pgset "dst 10.0.0.1"    sets IP destination address
113	                         (BEWARE! This generator is very aggressive!)
114	
115	 pgset "dst_min 10.0.0.1"            Same as dst
116	 pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
117	 pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
118	 pgset "src_max 10.0.0.254"          Set the maximum source IP.
119	 pgset "dst6 fec0::1"     IPV6 destination address
120	 pgset "src6 fec0::2"     IPV6 source address
121	 pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
122	 pgset "srcmac 00:00:00:00:00:00"    sets MAC source address
123	
124	 pgset "queue_map_min 0" Sets the min value of tx queue interval
125	 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
126	                         To select queue 1 of a given device,
127	                         use queue_map_min=1 and queue_map_max=1
128	
129	 pgset "src_mac_count 1" Sets the number of MACs we'll range through.  
130	                         The 'minimum' MAC is what you set with srcmac.
131	
132	 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
133	                         The 'minimum' MAC is what you set with dstmac.
134	
135	 pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
136	                         are: IPSRC_RND # IP source is random (between min/max)
137	                              IPDST_RND # IP destination is random
138	                              UDPSRC_RND, UDPDST_RND,
139	                              MACSRC_RND, MACDST_RND
140	                              TXSIZE_RND, IPV6,
141	                              MPLS_RND, VID_RND, SVID_RND
142	                              FLOW_SEQ,
143	                              QUEUE_MAP_RND # queue map random
144	                              QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
145	                              UDPCSUM,
146	                              IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
147	                              NODE_ALLOC # node specific memory allocation
148	
149	 pgset spi SPI_VALUE     Set specific SA used to transform packet.
150	
151	 pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
152	                         cycle through the port range.
153	
154	 pgset "udp_src_max 9"   set UDP source port max.
155	 pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
156	                         cycle through the port range.
157	 pgset "udp_dst_max 9"   set UDP destination port max.
158	
159	 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
160	                                         outer label=16,middle label=32,
161						 inner label=0 (IPv4 NULL)) Note that
162						 there must be no spaces between the
163						 arguments. Leading zeros are required.
164						 Do not set the bottom of stack bit,
165						 that's done automatically. If you do
166						 set the bottom of stack bit, that
167						 indicates that you want to randomly
168						 generate that address and the flag
169						 MPLS_RND will be turned on. You
170						 can have any mix of random and fixed
171						 labels in the label stack.
172	
173	 pgset "mpls 0"		  turn off mpls (or any invalid argument works too!)
174	
175	 pgset "vlan_id 77"       set VLAN ID 0-4095
176	 pgset "vlan_p 3"         set priority bit 0-7 (default 0)
177	 pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)
178	
179	 pgset "svlan_id 22"      set SVLAN ID 0-4095
180	 pgset "svlan_p 3"        set priority bit 0-7 (default 0)
181	 pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)
182	
183	 pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
184	 pgset "svlan 9999"       > 4095 remove svlan tag
185	
186	
187	 pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
188	 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
189	
190	 pgset stop    	          aborts injection. Also, ^C aborts generator.
191	
192	 pgset "rate 300M"        set rate to 300 Mb/s
193	 pgset "ratep 1000000"    set rate to 1Mpps
194	
195	Sample scripts
196	==============
197	
198	A collection of small tutorial scripts for pktgen is in the
199	samples/pktgen directory:
200	
201	pktgen.conf-1-1                  # 1 CPU 1 dev 
202	pktgen.conf-1-2                  # 1 CPU 2 dev
203	pktgen.conf-2-1                  # 2 CPU's 1 dev 
204	pktgen.conf-2-2                  # 2 CPU's 2 dev
205	pktgen.conf-1-1-rdos             # 1 CPU 1 dev w. route DoS 
206	pktgen.conf-1-1-ip6              # 1 CPU 1 dev ipv6
207	pktgen.conf-1-1-ip6-rdos         # 1 CPU 1 dev ipv6  w. route DoS
208	pktgen.conf-1-1-flows            # 1 CPU 1 dev multiple flows.
209	
210	Run in shell: ./pktgen.conf-X-Y
211	This does all the setup including sending.
212	
213	
214	Interrupt affinity
215	===================
216	Note that when adding devices to a specific CPU it is a good idea to
217	also assign /proc/irq/XX/smp_affinity so that the TX interrupts are bound
218	to the same CPU.  This reduces cache bouncing when freeing skbs.
219	
220	Enable IPsec
221	============
222	Default IPsec transformation with ESP encapsulation plus transport mode
223	can be enabled by simply setting:
224	
225	pgset "flag IPSEC"
226	pgset "flows 1"
227	
228	To avoid breaking existing testbed scripts for using AH type and tunnel mode,
229	you can use "pgset spi SPI_VALUE" to specify which transformation mode
230	to employ.
231	
232	
233	Current commands and configuration options
234	==========================================
235	
236	** Pgcontrol commands:
237	
238	start
239	stop
240	
241	** Thread commands:
242	
243	add_device
244	rem_device_all
245	max_before_softirq
246	
247	
248	** Device commands:
249	
250	count
251	clone_skb
252	debug
253	
254	frags
255	delay
256	
257	src_mac_count
258	dst_mac_count
259	
260	pkt_size 
261	min_pkt_size
262	max_pkt_size
263	
264	mpls
265	
266	udp_src_min
267	udp_src_max
268	
269	udp_dst_min
270	udp_dst_max
271	
272	flag
273	  IPSRC_RND
274	  IPDST_RND
275	  UDPSRC_RND
276	  UDPDST_RND
277	  MACSRC_RND
278	  MACDST_RND
279	  TXSIZE_RND
280	  IPV6
281	  MPLS_RND
282	  VID_RND
283	  SVID_RND
284	  FLOW_SEQ
285	  QUEUE_MAP_RND
286	  QUEUE_MAP_CPU
287	  UDPCSUM
288	  IPSEC
289	  NODE_ALLOC
290	
291	dst_min
292	dst_max
293	
294	src_min
295	src_max
296	
297	dst_mac
298	src_mac
299	
300	clear_counters
301	
302	dst6
303	src6
304	
305	flows
306	flowlen
307	
308	rate
309	ratep
310	
311	References:
312	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
313	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
314	
315	Paper from Linux-Kongress in Erlangen 2004.
316	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
317	
318	Thanks to:
319	Grant Grundler for testing on IA-64 and parisc, Harald Welte,  Lennert Buytenhek
320	Stephen Hemminger, Andi Kleen, Dave Miller and many others.
321	
322	
323	Good luck with the linux net-development.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.