Question
Why is the actual CDN billing traffic different from the log monitoring traffic?
Possible Reasons
The traffic recorded in the log monitoring is the traffic counted by the application layer logs, but the actual network traffic generated is 7%-15% higher than the traffic counted by the application layer. The main reasons are as follows:
- Consumption of TCP/IP packet headers
HTTP requests are based on the TCP/IP protocol, and the maximum number of bytes per packet in the Internet is 1500. These 1500 bytes include 40 bytes of packet headers inserted by the TCP and IP protocols. The packet header part also generates traffic, and since adding packet headers is done by the protocol stack at the kernel layer and cannot be counted by the application layer, the traffic of these 40 bytes will not be recorded in the logs, and the traffic of this part will account for more than 2.74% (40/1460) of the traffic calculated through the logs. Under normal circumstances, it would account for about 3%.
- TCP retransmission
Depending on the load on the physical network of the Internet, 3-10% of the packets sent will be dropped by the Internet under normal circumstances. After being dropped, the server will retransmit the dropped part, the retransmission action is handled by the kernel layer protocol stack, and the application layer cannot be counted. This part of the traffic accounts for the proportion of our logs calculated traffic. Depending on the network performance, in the early morning, Internet load is light, so the retransmission rate will be low; in the evening, the Internet load is heavy, so the retransmission will rise. In general, this rate is between 3%-7% .
Therefore, in the industry standard, a network consumption of 7%-15% will be added to the original traffic for billing traffic statistics, and we take the average value of 10% for network consumption statistics.