Linux Network Administrators Guide
Prev
Chapter 20. Netnews
Next
20.3. How Does Usenet Handle News?
Today, Usenet has grown to enormous proportions. Sites that carry the whole of Netnews usually transfer something like a paltry 60 MB a day.[1] Of course, this requires much more than pushing files around. So let's take a look at the way most Unix systems handle Usenet news.
News begins when users create and post articles. Each user enters a message into a special application called a newsreader, which formats it appropriately for transmission to the local news server. In Unix environments the newsreader commonly uses the inews command to transmit articles to the newsserver using the TCP/IP protocol. But it's also possible to write the article directly into a file in a special directory called the news spool. Once the posting is delivered to the local news server, it takes responsibility for delivering the article to other news users.
News is distributed through the net by various transports. The medium used to be UUCP, but today the main traffic is carried by Internet sites. The routing algorithm used is called flooding . Each site maintains a number of links (news feeds ) to other sites. Any article generated or received by the local news system is forwarded to them, unless it has already been at that site, in which case it is discarded. A site may find out about all other sites the article has already traversed by looking at the Path: header field. This header contains a list of all systems through which the article has been forwarded in bang path notation.
To distinguish articles and recognize duplicates, Usenet articles have to carry a message ID (specified in the Message-Id: header field), which combines the posting site's name and a serial number into <serial @site >. For each article processed, the news system logs this ID into a history file, against which all newly arrived articles are checked.
The flow between any two sites may be limited by two criteria. For one, an article is assigned a distribution (in the Distribution: header field), which may be used to confine it to a certain group of sites. On the other hand, the newsgroups exchanged may be limited by both the sending and receiving systems. The set of newsgroups and distributions allowed to be transmitted to a site are usually kept in the sys file.
The sheer number of articles usually requires that improvements be made to the above scheme. On UUCP networks, systems collect articles over a period of time and combine them into a single file, which is compressed and sent to the remote site. This is called batching .
An alternative technique is the ihave/sendme protocol that prevents duplicate articles from being transferred, thus saving net bandwidth. Instead of putting all articles in batch files and sending them along, only the message IDs of articles are combined into a giant "ihave" message and sent to the remote site. The remote site reads this message, compares it to its history file, and returns the list of articles it wants in a "sendme" message. Only the requested articles are sent.
Of course, ihave/sendme makes sense only if it involves two big sites that receive news from several independent feeds each, and that poll each other often enough for an efficient flow of news.
Sites that are on the Internet generally rely on TCP/IP-based software that uses the Network News Transfer Protocol (NNTP). NNTP is described in RFC-977; it is responsible for the transfer of news between news servers and provides Usenet access to single users on remote hosts.
NNTP knows three different ways to transfer news. One is a real-time version of ihave/sendme, also referred to as pushing news. The second technique is called pulling news, in which the client requests a list of articles in a given newsgroup or hierarchy that have arrived at the server's site after a specified date, and chooses those it cannot find in its history file.
* License
