About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Documentation / networking / pktgen.txt




Custom Search

Based on kernel version 4.0. Page generated on 2015-04-14 21:25 EST.

1	
2	
3	                  HOWTO for the linux packet generator 
4	                  ------------------------------------
5	
6	Date: 041221
7	
8	Enable CONFIG_NET_PKTGEN to compile and build pktgen.o either in kernel
9	or as module. Module is preferred. insmod pktgen if needed. Once running
10	pktgen creates a thread on each CPU where each thread has affinity to its CPU.
11	Monitoring and controlling is done via /proc. Easiest to select a suitable 
12	a sample script and configure.
13	
14	On a dual CPU:
15	
16	ps aux | grep pkt
17	root       129  0.3  0.0     0    0 ?        SW    2003 523:20 [pktgen/0]
18	root       130  0.3  0.0     0    0 ?        SW    2003 509:50 [pktgen/1]
19	
20	
21	For monitoring and control pktgen creates:
22		/proc/net/pktgen/pgctrl
23		/proc/net/pktgen/kpktgend_X
24	        /proc/net/pktgen/ethX
25	
26	
27	Tuning NIC for max performance
28	==============================
29	
30	The default NIC setting are (likely) not tuned for pktgen's artificial
31	overload type of benchmarking, as this could hurt the normal use-case.
32	
33	Specifically increasing the TX ring buffer in the NIC:
34	 # ethtool -G ethX tx 1024
35	
36	A larger TX ring can improve pktgen's performance, while it can hurt
37	in the general case, 1) because the TX ring buffer might get larger
38	than the CPUs L1/L2 cache, 2) because it allow more queueing in the
39	NIC HW layer (which is bad for bufferbloat).
40	
41	One should be careful to conclude, that packets/descriptors in the HW
42	TX ring cause delay.  Drivers usually delay cleaning up the
43	ring-buffers (for various performance reasons), thus packets stalling
44	the TX ring, might just be waiting for cleanup.
45	
46	This cleanup issues is specifically the case, for the driver ixgbe
47	(Intel 82599 chip).  This driver (ixgbe) combine TX+RX ring cleanups,
48	and the cleanup interval is affected by the ethtool --coalesce setting
49	of parameter "rx-usecs".
50	
51	For ixgbe use e.g "30" resulting in approx 33K interrupts/sec (1/30*10^6):
52	 # ethtool -C ethX rx-usecs 30
53	
54	
55	Viewing threads
56	===============
57	/proc/net/pktgen/kpktgend_0 
58	Name: kpktgend_0  max_before_softirq: 10000
59	Running: 
60	Stopped: eth1 
61	Result: OK: max_before_softirq=10000
62	
63	Most important the devices assigned to thread. Note! A device can only belong 
64	to one thread.
65	
66	
67	Viewing devices
68	===============
69	
70	Parm section holds configured info. Current hold running stats. 
71	Result is printed after run or after interruption. Example:
72	
73	/proc/net/pktgen/eth1       
74	
75	Params: count 10000000  min_pkt_size: 60  max_pkt_size: 60
76	     frags: 0  delay: 0  clone_skb: 1000000  ifname: eth1
77	     flows: 0 flowlen: 0
78	     dst_min: 10.10.11.2  dst_max: 
79	     src_min:   src_max: 
80	     src_mac: 00:00:00:00:00:00  dst_mac: 00:04:23:AC:FD:82
81	     udp_src_min: 9  udp_src_max: 9  udp_dst_min: 9  udp_dst_max: 9
82	     src_mac_count: 0  dst_mac_count: 0 
83	     Flags: 
84	Current:
85	     pkts-sofar: 10000000  errors: 39664
86	     started: 1103053986245187us  stopped: 1103053999346329us idle: 880401us
87	     seq_num: 10000011  cur_dst_mac_offset: 0  cur_src_mac_offset: 0
88	     cur_saddr: 0x10a0a0a  cur_daddr: 0x20b0a0a
89	     cur_udp_dst: 9  cur_udp_src: 9
90	     flows: 0
91	Result: OK: 13101142(c12220741+d880401) usec, 10000000 (60byte,0frags)
92	  763292pps 390Mb/sec (390805504bps) errors: 39664
93	
94	Configuring threads and devices
95	================================
96	This is done via the /proc interface easiest done via pgset in the scripts
97	
98	Examples:
99	
100	 pgset "clone_skb 1"     sets the number of copies of the same packet
101	 pgset "clone_skb 0"     use single SKB for all transmits
102	 pgset "burst 8"         uses xmit_more API to queue 8 copies of the same
103	                         packet and update HW tx queue tail pointer once.
104	                         "burst 1" is the default
105	 pgset "pkt_size 9014"   sets packet size to 9014
106	 pgset "frags 5"         packet will consist of 5 fragments
107	 pgset "count 200000"    sets number of packets to send, set to zero
108	                         for continuous sends until explicitly stopped.
109	
110	 pgset "delay 5000"      adds delay to hard_start_xmit(). nanoseconds
111	
112	 pgset "dst 10.0.0.1"    sets IP destination address
113	                         (BEWARE! This generator is very aggressive!)
114	
115	 pgset "dst_min 10.0.0.1"            Same as dst
116	 pgset "dst_max 10.0.0.254"          Set the maximum destination IP.
117	 pgset "src_min 10.0.0.1"            Set the minimum (or only) source IP.
118	 pgset "src_max 10.0.0.254"          Set the maximum source IP.
119	 pgset "dst6 fec0::1"     IPV6 destination address
120	 pgset "src6 fec0::2"     IPV6 source address
121	 pgset "dstmac 00:00:00:00:00:00"    sets MAC destination address
122	 pgset "srcmac 00:00:00:00:00:00"    sets MAC source address
123	
124	 pgset "queue_map_min 0" Sets the min value of tx queue interval
125	 pgset "queue_map_max 7" Sets the max value of tx queue interval, for multiqueue devices
126	                         To select queue 1 of a given device,
127	                         use queue_map_min=1 and queue_map_max=1
128	
129	 pgset "src_mac_count 1" Sets the number of MACs we'll range through.  
130	                         The 'minimum' MAC is what you set with srcmac.
131	
132	 pgset "dst_mac_count 1" Sets the number of MACs we'll range through.
133	                         The 'minimum' MAC is what you set with dstmac.
134	
135	 pgset "flag [name]"     Set a flag to determine behaviour.  Current flags
136	                         are: IPSRC_RND # IP source is random (between min/max)
137	                              IPDST_RND # IP destination is random
138	                              UDPSRC_RND, UDPDST_RND,
139	                              MACSRC_RND, MACDST_RND
140	                              TXSIZE_RND, IPV6,
141	                              MPLS_RND, VID_RND, SVID_RND
142	                              FLOW_SEQ,
143	                              QUEUE_MAP_RND # queue map random
144	                              QUEUE_MAP_CPU # queue map mirrors smp_processor_id()
145	                              UDPCSUM,
146	                              IPSEC # IPsec encapsulation (needs CONFIG_XFRM)
147	                              NODE_ALLOC # node specific memory allocation
148	
149	 pgset spi SPI_VALUE     Set specific SA used to transform packet.
150	
151	 pgset "udp_src_min 9"   set UDP source port min, If < udp_src_max, then
152	                         cycle through the port range.
153	
154	 pgset "udp_src_max 9"   set UDP source port max.
155	 pgset "udp_dst_min 9"   set UDP destination port min, If < udp_dst_max, then
156	                         cycle through the port range.
157	 pgset "udp_dst_max 9"   set UDP destination port max.
158	
159	 pgset "mpls 0001000a,0002000a,0000000a" set MPLS labels (in this example
160	                                         outer label=16,middle label=32,
161						 inner label=0 (IPv4 NULL)) Note that
162						 there must be no spaces between the
163						 arguments. Leading zeros are required.
164						 Do not set the bottom of stack bit,
165						 that's done automatically. If you do
166						 set the bottom of stack bit, that
167						 indicates that you want to randomly
168						 generate that address and the flag
169						 MPLS_RND will be turned on. You
170						 can have any mix of random and fixed
171						 labels in the label stack.
172	
173	 pgset "mpls 0"		  turn off mpls (or any invalid argument works too!)
174	
175	 pgset "vlan_id 77"       set VLAN ID 0-4095
176	 pgset "vlan_p 3"         set priority bit 0-7 (default 0)
177	 pgset "vlan_cfi 0"       set canonical format identifier 0-1 (default 0)
178	
179	 pgset "svlan_id 22"      set SVLAN ID 0-4095
180	 pgset "svlan_p 3"        set priority bit 0-7 (default 0)
181	 pgset "svlan_cfi 0"      set canonical format identifier 0-1 (default 0)
182	
183	 pgset "vlan_id 9999"     > 4095 remove vlan and svlan tags
184	 pgset "svlan 9999"       > 4095 remove svlan tag
185	
186	
187	 pgset "tos XX"           set former IPv4 TOS field (e.g. "tos 28" for AF11 no ECN, default 00)
188	 pgset "traffic_class XX" set former IPv6 TRAFFIC CLASS (e.g. "traffic_class B8" for EF no ECN, default 00)
189	
190	 pgset stop    	          aborts injection. Also, ^C aborts generator.
191	
192	 pgset "rate 300M"        set rate to 300 Mb/s
193	 pgset "ratep 1000000"    set rate to 1Mpps
194	
195	Example scripts
196	===============
197	
198	A collection of small tutorial scripts for pktgen is in examples dir.
199	
200	pktgen.conf-1-1                  # 1 CPU 1 dev 
201	pktgen.conf-1-2                  # 1 CPU 2 dev
202	pktgen.conf-2-1                  # 2 CPU's 1 dev 
203	pktgen.conf-2-2                  # 2 CPU's 2 dev
204	pktgen.conf-1-1-rdos             # 1 CPU 1 dev w. route DoS 
205	pktgen.conf-1-1-ip6              # 1 CPU 1 dev ipv6
206	pktgen.conf-1-1-ip6-rdos         # 1 CPU 1 dev ipv6  w. route DoS
207	pktgen.conf-1-1-flows            # 1 CPU 1 dev multiple flows.
208	
209	Run in shell: ./pktgen.conf-X-Y It does all the setup including sending. 
210	
211	
212	Interrupt affinity
213	===================
214	Note when adding devices to a specific CPU there good idea to also assign 
215	/proc/irq/XX/smp_affinity so the TX-interrupts gets bound to the same CPU.
216	as this reduces cache bouncing when freeing skb's.
217	
218	Enable IPsec
219	============
220	Default IPsec transformation with ESP encapsulation plus Transport mode
221	could be enabled by simply setting:
222	
223	pgset "flag IPSEC"
224	pgset "flows 1"
225	
226	To avoid breaking existing testbed scripts for using AH type and tunnel mode,
227	user could use "pgset spi SPI_VALUE" to specify which formal of transformation
228	to employ.
229	
230	
231	Current commands and configuration options
232	==========================================
233	
234	** Pgcontrol commands:
235	
236	start
237	stop
238	
239	** Thread commands:
240	
241	add_device
242	rem_device_all
243	max_before_softirq
244	
245	
246	** Device commands:
247	
248	count
249	clone_skb
250	debug
251	
252	frags
253	delay
254	
255	src_mac_count
256	dst_mac_count
257	
258	pkt_size 
259	min_pkt_size
260	max_pkt_size
261	
262	mpls
263	
264	udp_src_min
265	udp_src_max
266	
267	udp_dst_min
268	udp_dst_max
269	
270	flag
271	  IPSRC_RND
272	  IPDST_RND
273	  UDPSRC_RND
274	  UDPDST_RND
275	  MACSRC_RND
276	  MACDST_RND
277	  TXSIZE_RND
278	  IPV6
279	  MPLS_RND
280	  VID_RND
281	  SVID_RND
282	  FLOW_SEQ
283	  QUEUE_MAP_RND
284	  QUEUE_MAP_CPU
285	  UDPCSUM
286	  IPSEC
287	  NODE_ALLOC
288	
289	dst_min
290	dst_max
291	
292	src_min
293	src_max
294	
295	dst_mac
296	src_mac
297	
298	clear_counters
299	
300	dst6
301	src6
302	
303	flows
304	flowlen
305	
306	rate
307	ratep
308	
309	References:
310	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/
311	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/examples/
312	
313	Paper from Linux-Kongress in Erlangen 2004.
314	ftp://robur.slu.se/pub/Linux/net-development/pktgen-testing/pktgen_paper.pdf
315	
316	Thanks to:
317	Grant Grundler for testing on IA-64 and parisc, Harald Welte,  Lennert Buytenhek
318	Stephen Hemminger, Andi Kleen, Dave Miller and many others.
319	
320	
321	Good luck with the linux net-development.
Hide Line Numbers
About Kernel Documentation Linux Kernel Contact Linux Resources Linux Blog

Information is copyright its respective author. All material is available from the Linux Kernel Source distributed under a GPL License. This page is provided as a free service by mjmwired.net.