2016-11-22 99 views
1

我有一个HTTP线程代理,即从客户的每个请求我产生一个线程统计:现在我想收集一些统计像每秒(PPS)比特每秒(BPS)和包。使用libpcap的收集上的连接

我喜欢我的代码做的只有一两件事,所以如果一个线程处理的连接,将不计算BPS和PPS为每个数据包过,我要离开这个给另一个线程。

我为来自客户端的每个HTTP请求创建一个线程,并且如果代理成功连接到请求的远程服务器,proxt将实际的HTTP请求发送到服务器,并且在路由数据之前,连接线程创建一个日志线程:将计算bps和pps,直到连接打开。连接线程提供了关于要过滤哪些数据包(本地IP地址,本地端口,远程IP地址,远程端口)的日志线程信息,以便每个日志记录线程仅过滤来自父连接线程的数据包。

我有问题,计算为每个数据包BPS和PPS。

下面是记录线程包我的循环拍摄的伪代码:

// pcap variables 
pcap_t *handle; 
struct pcap_pkthdr *header; 
const u_char *pkt_data; 
// timevals used to calculate delay from last filtered packet 
struct timeval oldTimevalUpload; 
struct timeval oldTimevalDownload;  
memset(&oldTimevalUpload, 0, sizeof(oldTimevalUpload)); 
memset(&oldTimevalDownload, 0, sizeof(oldTimevalDownload)); 

// stopLogging is a boolean flag declared in connection "parent" thread: 
// it is set to false when connection thread has done sending and 
// receiving data and connection is going to be closed 
while (((res = pcap_next_ex(handle, &header, &pkt_data)) >= 0) && (!stopLogging)) { 
    // check res 
    if (packet is upload) { 
     struct timeval difference; 
     timeval_subtract(&difference, &(header->ts), &oldTimevalUpload); 
     long long delay = (difference.tv_sec * 1000000) + difference.tv_usec; 
     long long acceptedPackets = ((long long)(pkt_data)) * 1000000; 
     long long acceptedBits = ((long long)(pkt_data+8)) * 8 * 1000000; 
     long long pps = acceptedPackets/delay; 
     long long bps = acceptedBits/delay; 
     debugRed(host << ", UPLOAD DIVIDE ACCEPTED PKTS " << acceptedPackets << 
      " AND ACCEPTED BITS " << acceptedBits << " PER DELAY " << delay << 
      " IS PPS " << pps << " AND BPS " << bps); 
     oldTimevalUpload.tv_sec = header->ts.tv_sec; 
     oldTimevalUpload.tv_usec = header->ts.tv_usec; 
    } else if (packet is download) { 
     // basically the same as above 
    } 
} 
debug("Quit logging connection " << localIPaddr << ":" << localPort << " and " 
    << remoteIPaddr << ":" << remotePort); 
pcap_close(handle); 

下面是一个示例输出:

www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1479811349890053 IS PPS 0 AND BPS 0 
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1479811350032141 IS PPS 0 AND BPS 0 
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 4344 IS PPS 22845174953 AND BPS 182761414364 
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 146464 IS PPS 677568822 AND BPS 5420551015 
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 2815 IS PPS 35253797513 AND BPS 282030402841 
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 2808 IS PPS 35341680911 AND BPS 282733470085 
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1120 IS PPS 88606642857 AND BPS 708853200000 
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 1134 IS PPS 87512733686 AND BPS 700101925925 
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 39658 IS PPS 2502381360 AND BPS 20019052498 
www.netflix.com DOWNLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 176317 IS PPS 562846690 AND BPS 4502773890 
www.netflix.com UPLOAD, DIVIDE ACCEPTED PKTS 99239440000000 AND ACCEPTED BITS 793915584000000 PER DELAY 136687 IS PPS 726034224 AND BPS 5808274261 

我从来不知道我的家庭网络能够承受超过500 MBps的,所以一定是错的。

This page显示了如何计算bps和pps,并且解释了在acceptedBits中的移位,但是我要反正报告它。他说的话基本上

Starting from the beginning of <code>pkt_data</code>, you can see the first <code>long integer</code> bytes represent accepted bytes

,我所做的一切:在这里你可以看到的功能pcap_next_ex第二和第三个参数!为什么我会变得如此庞大而奇怪的bps和pps?

在Ubuntu 14.04的工作;不知道如何检查libpcap版本,但locate libpcap给出了这样的:

/home/dexter/Desktop/wireshark-1.99.9/wiretap/libpcap.c 
/home/dexter/Desktop/wireshark-1.99.9/wiretap/libpcap.h 
/usr/lib/x86_64-linux-gnu/libpcap.a 
/usr/lib/x86_64-linux-gnu/libpcap.so 
/usr/lib/x86_64-linux-gnu/libpcap.so.0.8 
/usr/lib/x86_64-linux-gnu/libpcap.so.1.5.3 
/usr/share/doc/libpcap-dev 
/usr/share/doc/libpcap0.8 
/usr/share/doc/libpcap0.8-dev 
/usr/share/doc/libpcap-dev/changelog.Debian.gz 
/usr/share/doc/libpcap-dev/copyright 
/usr/share/doc/libpcap0.8/CREDITS.gz 
/usr/share/doc/libpcap0.8/README.Debian 
/usr/share/doc/libpcap0.8/README.gz 
/usr/share/doc/libpcap0.8/changelog.Debian.gz 
/usr/share/doc/libpcap0.8/copyright 
/usr/share/doc/libpcap0.8-dev/changelog.Debian.gz 
/usr/share/doc/libpcap0.8-dev/copyright 
/var/lib/dpkg/info/libpcap-dev.[list,md5sums] 
/var/lib/dpkg/info/libpcap0.8-dev.[list,md5sums,preinst] 
/var/lib/dpkg/info/libpcap0.8:amd64.[list,md5sums,postinst,postrm,shlibs,symbols] 

回答

1

在您的代码:

long long acceptedPackets = ((long long)(pkt_data)) * 1000000; 
long long acceptedBits = ((long long)(pkt_data+8)) * 8 * 1000000; 

虽然pkt_data是一个指针。

你正在做的是基本上获取分组数据的地址,将其转换为long long,加上8(对于第二行),将其乘以一个常数,并认为这是你的值,这在语义上是不正确的。您应该取消引用该指针,并考虑您的数据类型(将pkt_data转换为指向long long的指针)。

在代码:

long long acceptedPackets = (*(long long*)(pkt_data)) * 1000000; 
long long acceptedBits = (*(long long*)(pkt_data+8)) * 8 * 1000000; 
// this also works: 
//long long acceptedPackets = *(long long*)pkt_data * 1000000; 
//long long acceptedBits = *((long long*)pkt_data + 1) * 8 * 1000000; 

举一个例子,看http://ideone.com/JqmRre

编辑

this guide

最后一个参数是最有趣的人全部,最多 混乱的平均新手pcap程序员。本发明的另一个指针 到u_char,并指向数据块的第一个字节含有 整个分组

这意味着pkt_data是数据包的内容本身。除非数据包的前16个字节包含所需的信息(这是不真实的,因为它包含原始数据包,因此它具有ETH,IP和TCP/UDP标头),否则不能使用该数据。为了获得PPS指标,你必须在你的循环中实现一个简单的计数器(因为你每帧都要打印该指标,简单的long long pps = (long long)(1.0/delay);应该就足够了 - 请注意,除法处于浮点数。 。,你应该使用帧头的信息,以便long long bps = (long long)(header->caplen * 8.0/delay);应该做

作为一个侧面说明,时间指标,因为你正在使用C++ 11,请尝试使用chrono它比timeval中更清晰,更安全:

添加一个#include <chrono>

您的最终代码应该如下所示:

// pcap variables 
pcap_t *handle; 
struct pcap_pkthdr *header; 
const u_char *pkt_data; 
// Use of high_resolution_clock 
std::chrono::high_resolution_clock::time_point oldTimeUpload = std::chrono::high_resolution_clock::now(); 
std::chrono::high_resolution_clock::time_point oldTimeDownload = std::chrono::high_resolution_clock::now(); 

// stopLogging is a boolean flag declared in connection "parent" thread: 
// it is set to false when connection thread has done sending and 
// receiving data and connection is going to be closed 
while (((res = pcap_next_ex(handle, &header, &pkt_data)) >= 0) && (!stopLogging)) { 
    // check res 
    if (packet is upload) { 
     std::chrono::high_resolution_clock::time_point now = std::chrono::high_resolution_clock::now(); 
     long long delay = std::chrono::duration_cast<std::chrono::nanoseconds>(now - oldTimeUpload).count(); 
     long long pps = (long long)(1000000000.0/delay); 
     long long bps = (long long)(header->caplen * 8 * 1000000000.0/delay); 
     debugRed(host << ", UPLOAD DIVIDE PER DELAY " << delay << 
      " IS PPS " << pps << " AND BPS " << bps); 
     oldTimevalUpload.tv_sec = header->ts.tv_sec; 
     oldTimevalUpload.tv_usec = header->ts.tv_usec; 
    } else if (packet is download) { 
     // basically the same as above 
    } 
} 
debug("Quit logging connection " << localIPaddr << ":" << localPort << " and " 
    << remoteIPaddr << ":" << remotePort); 
pcap_close(handle); 
+0

我也跟着你的建议:我现在有负数XD 'www.youtube.com下载DIVIDE被接受PKTS -9153173178195537408及接受BITS -2444556543073261568 PER DELAY 1479847752267586 IS PPS -6185与BPS -1651 www.youtube.com DOWNLOAD DIVIDE ACCEPTED PKTS -9153173178195537408 AND ACCEPTED BITS -2444556543073261568 PER DELAY 2745 IS PPS -3334489318104020与BPS -890548831720678 www.youtube.com UPLOAD DIVIDE ACCEPTED PKTS -1866508824056485888 AND接受位-2458024441489261568 PER DELAY 1479847752270362是PPS -1261和BPS -1660'和s o on ... – elmazzun

+0

请检查我的编辑 –

0

如何在另一个进程而不是另一个线程上执行pps和bps计算?我可以推荐HttpAnalyzer实用程序,它可以直接从网络接口捕获HTTP数据包,并计算pps,bps和更多的统计数据。

由于它是开源的,您可以更改代码以符合您的目的或按原样使用它。

下面是该实用程序的输出示例:

STATS SUMMARY 
============= 

General stats 
-------------------- 

Sample time:          18.374 [Seconds] 
Number of HTTP packets:       5662 [Packets] 
Rate of HTTP packets:       291.910 [Packets/sec] 
Number of HTTP flows:        55 [Flows] 
Rate of HTTP flows:        2.836 [Flows/sec] 
Number of HTTP pipelining flows:      0 [Flows] 
Number of HTTP transactions:      322 [Transactions] 
Rate of HTTP transactions:      16.601 [Transactions/sec] 
Total HTTP data:        5916120 [Bytes] 
Rate of HTTP data:       305011.600 [Bytes/sec] 
Average packets per flow:      102.945 [Packets] 
Average transactions per flow:     5.963 [Transactions] 
Average data per flow:      107565.818 [Bytes] 

HTTP request stats 
-------------------- 

Number of HTTP requests:       323 [Requests] 
Rate of HTTP requests:       16.653 [Requests/sec] 
Total data in headers:       188596 [Bytes] 
Average header size:       583.889 [Bytes] 

HTTP response stats 
-------------------- 

Number of HTTP responses:       332 [Responses] 
Rate of HTTP responses:       17.117 [Responses/sec] 
Total data in headers:       119577 [Bytes] 
Average header size:       360.172 [Bytes] 
Num of responses with content-length:    320 [Responses] 
Total body size (may be compressed):   5409410 [Bytes] 
Average body size:       16904.406 [Bytes] 

HTTP request methods 
-------------------- 

| Method | Count | 
--------------------- 
| GET  | 321 | 
| POST  | 2  | 
--------------------- 

Hostnames count 
-------------------- 

| Hostname         | Count | 
---------------------------------------------------- 
| images1.teny.co.qq      | 180 | 
| www.teny.co.qq       | 82 | 
| go.teny.co.qq       | 14 | 
| www.niwwin.co.qq       | 8  | 
| az835984.vo.msecnd.net     | 5  | 
| asset.pagefair.com      | 3  | 
| b.scorecardresearch.com     | 3  | 
| cdn.oolala.com       | 3  | 
| asset.pagefair.net      | 2  | 
| dy2.teny.co.qq       | 2  | 
| ecdn.firstimpression.io     | 2  | 
| pagead2.googlesyndication.com   | 2  | 
| server.exposebox.com      | 2  | 
| totalmedia2.teny.co.qq     | 2  | 
| vrp.mybrain.com       | 1  | 
| trc.oolala.com       | 1  | 
| zdwidget3-bs.sphereup.com    | 1  | 
| vrt.mybrain.com       | 1  | 
| www.googletagmanager.com     | 1  | 
| a.visualrevenue.com      | 1  | 
| tpc.googlesyndication.com    | 1  | 
| static.dynamicyield.com     | 1  | 
| st.dynamicyield.com      | 1  | 
| sf.exposebox.com       | 1  | 
| mediadownload.teny.co.qq     | 1  | 
| cdn.firstimpression.io     | 1  | 
| ajax.googleapis.com      | 1  | 
---------------------------------------------------- 

Status code count 
-------------------- 

| Status Code     | Count | 
---------------------------------------- 
| 200 OK      | 327 | 
| 204 No Content    | 1  | 
| 301 Moved Permanently  | 1  | 
| 302 Moved Temporarily  | 1  | 
| 304 Not Modified    | 2  | 
---------------------------------------- 

Content-type count 
-------------------- 

| Content-type     | Count | 
------------------------------------------ 
| application/javascript   | 11 | 
| application/json    | 1  | 
| application/x-javascript  | 23 | 
| image/gif      | 22 | 
| image/jpeg      | 157 | 
| image/png      | 85 | 
| text/css      | 9  | 
| text/html      | 8  | 
| text/javascript    | 13 | 
------------------------------------------