0-471-31499-4, $39.99 US
Available October 1998
by Geoff HustonChapter 14 Interacting with other ISPs
The Internet is not, and never has been, a single network. The Internet is a collection of interconnected component networks that share a common addressing structure, a common view of routing, and a common view of a naming system. This interconnection environment spans more than 60,000 component networks, and this number continues to grow and grow.
Therefore, every ISP must not only coexist with other ISPs but also must operate in cooperation with other ISPs. Underneath the competitive retail environment is a somewhat different environment, in which every Internet network must interoperate with neighboring Internet networks in order to produce an outcome of efficient and comprehensive connectivity and end-to end-service—the essential attribute of the total client offering.
Within this chapter, we explore this environment of interaction, examining the issues from a business perspective. The three areas covered here are the structure of ISP interaction within the Internet in terms of roles, the physical structures of ISP interaction, and the financial issues of ISP interaction.
ISP Roles: Retailing, Reselling, and Wholesaling
At some stage in the evolution of the ISP business, each ISP must confront the resale of their services. The initial business model of many ISPs is one based on a retail model of a direct relationship between the user and the ISP. This relationship is relatively sustainable within a dial-up host network access service. However, after the ISP expands to encompass clients who use a permanent connection to the ISP and who peer their network to that of the ISP, then sustaining a retail-only operation becomes difficult for the ISP. A natural outcome of the Internet model is that the control of the service environment rests with the client of the service rather than with the service provider. Therefore, a network client of an ISP access service has the ability to resell the access service to third-party clients. In this environment, reselling and wholesaling are very natural developments within the ISP activity sector, with or without the explicit concurrence of the provider ISP. The original ISP may see this reselling as an additional channel to the market for its own Internet carriage services. The ISP could take a positive stance by actively encouraging resellers into the market as a means of overall market stimulus, while tapping into the marketing, sales, and support resources of other entities to continue to drive the underlying Internet carriage service portfolio.
Given that a retail operation can become a provider to resellers at the discretion of the reselling retail client, is a wholesale transit ISP restricted from undertaking retail operations? No such restriction exists.
Internet carriage service is a commodity service, which does not allow for a significant level of intrinsic product discrimination. The relative low level of value added by a wholesale operation implies a low rate of financial return for that operation. This low financial return forces a wholesale provider into the retail sector as a means of improving the financial performance of the service operation. Many retail ISPs first move into greater levels of provisioning capacity to support an increased scope of retail operations. The low barriers to entry to the wholesale market allow for another means of increasing the scope of the operation. To lift business cash-flow levels, the business enters into wholesale agreements that effectively resell the carriage components of the operation without the bundling of other services normally associated with the retail operation. This process allows the ISP to gain high carriage volumes to gain access to lower unit tariffs from carriage providers.
As a reseller, a network operates both as a client and as a provider. This role ambiguity is by no means uncommon on the Internet. Few, if any, reasonable technical-based characterizations draw a clear and unambiguous distinction between a client and service provider when access services to networks are considered. A campus network may be a client of one or more service providers, while the network is a service provider to campus users. Indeed most networks in a similar situation take on the dual role of client and provider, and the ability to resell an access service can extend to almost arbitrary depths of the reselling hierarchy. From this technical perspective, very few natural divisions of the market support a stable segmentation into exclusively wholesale and exclusively retail market sectors. The overall structure of roles is indicated in Figure 14.1.
Figure 14.1 ISP roles and relationships.
The resultant business environment is one characterized by a reasonable degree of fluidity, in which no clear delineation of relative roles or markets exists. The ISP market environment is, therefore, one of competitive market forces in which each ISP has to create a market following using a market positioning strongly based on the service price and some aspects of service quality.
However, no ISP can operate in isolation. Each client has the expectation of universal and comprehensive reachability, such that any client of any other Internet ISP can reach the client, and the client can reach a client of any other ISP. The client of an ISP is not undertaking a service contract that limits connectivity only to other clients of the same ISP. Because no provider can claim ubiquity of access, every provider relies on every other provider to complete the user-provided picture of comprehensive connectivity. Because of this dependent relationship, an individual provider's effort to provide substantially superior quality of carriage may have little overall impact on the totality of client-delivered service quality. Quality of service becomes something that can be impacted negatively by poor local engineering but cannot be improved beyond the quality provided by the network's peers, and their peers in turn. Internet wholesale carriage services, therefore, are a commodity service, in which scant opportunity exists for service-based differentiation. The wholesale activity becomes a price-based service with low levels of added value.
The implication in terms of ISP positioning is that the retail operation (rather than the wholesale activity) is the major area where the ISP can provide discriminating service quality. Within the retail operation, the ISP can offer a wide variety of services with a set of associated service levels and base a market positioning on factors other than commodity carriage pricing.
Accordingly, the environment of interconnection between ISPs does not break down into a model of a set of wholesale carriage providers and associated retail service providers. The environment currently is one with a wide diversity of retail-oriented providers, where each provider may operate both as a retail service operator, and a wholesale carriage provider to other retailers.
Peer or Client?
One of the significant issues that arises here is whether an objective determination can be made of whether an ISP is a peer to, or a client of, another ISP. If a completely objective determination cannot be made, the question then becomes one of who is responsible for making such a determination.
This question is an inevitable outcome of the reselling environment, where the reseller starts to make multiple upstream service contracts, with a growing number of downstream clients of the reselling service. At this point, the profile of the original reseller is little distinguished from that of the original provider. The original reseller sees no unique value being offered by the original upstream provider and may conclude that it is adding value to the original upstream provider by offering the upstream provider volume carriage and close access to the reseller's client base. From the perspective of the original reseller, the roles have changed, and the reseller now perceives itself as a peer ISP to the original upstream ISP provider.
This role reversal is perhaps most significant when the generic interconnection environment is one of zero sum financial settlement, in which the successful assertion by one party of a change to peer interconnection status, from a previous relationship of a client, to the second party results in the dropping of client service revenue without any net change in the cost base of the operation. The party making the successful assertion of peer interconnection sees the opposite, with an immediate drop in the cost of the ISP operation with no net revenue change.
The traditional public regulatory resolution of such matters has been through an administrative process of licensed service providers, who become peer entities through a process of administrative fiat. In this model, an ISP becomes a licensed service provider through the payment of license fees to the communications regulatory body. The license then allows the enterprise access to interconnection arrangements. A client is an entity that operates without a carrier license, and a peer is one that has been granted such an instrument. However, such heavily regulated environments are quite artificial in their delineation of the entities that operate in this market, and this process acts as a depressant to large-scale private investment. The regulatory environment is changing worldwide to shift the burden of communications infrastructure investment from the public sector, or a uniquely positioned small segment of the private sector, to an environment that encourages private investment. The Internet industry is at the leading edge of this trend, and the ISP domain typically operates within a deregulated valued-added communications service provider regulatory environment. Individual licenses are replaced with generic class licenses or similar deregulated structures, in which formal applications or payments of license fees to operate in this domain are unnecessary. Therefore, no authoritative external entity makes the decision as to whether the relationship between two ISPs is that of a provider and client or that of peers.
If no public regulatory body wants to make such a determination, is there a comparable industry body? The early attempts of the Commercial Internet eXchange (CIX) arrangements were based on a description of the infrastructure of each party, in which acknowledgments of peer capability were based on the operation of a national transit infrastructure of a minimum specified capability. This specification of peering within the CIX subsequently was modified so that CIX peer status for an ISP was based on payment of the CIX Association membership fee. This CIX model is not one that intrinsically admits bilateral peer relationships. The relationship is an multilateral one, in which each ISP executes a single agreement with the CIX Association and then effectively is a component of the Association. The consequence of this multilateral arrangements is that the peering settlements can be regarded as an instance of zero sum financial settlement peering.
Other models use a functional peer specification.. If the ISP attaches to a nominated physical exchange structure, then the ISP is in a position to open bilateral negotiations with any other ISP also directly attached to the structure. This model is inherently more flexible, as the bilateral exchange structure enables each represented ISP to make their own determination of whether to agree to a peer relationship or not. This model also enables each bilateral peer arrangement to be executed individually, admitting a wider diversity of financial settlement arrangements.
The bottom line is that a peer relationship is based on the supposition that either party can terminate the interconnection relationship and that the other party does not consider such an action a competitively hostile act. If one party has a high reliance on the connection arrangement and the other does not, then the most stable business outcome is that this reliance is expressed in terms of a service contract with the other party, and a provider/client relationship is established. If a balance of mutual requirement exists between both parties and if the ability to address these requirements in other ways is open to both parties, then a stable basis for a peer relationship exists. Such a statement has no intrinsic metrics that allow the requirements to be quantified. Peering in such an environment is best expressed as the balance of perceptions, in which each party perceives an acceptable approximation of equal benefit in the interconnection relationship in their own terms.
This conclusion leads to the various tiers of accepted peering that are evident in the Internet today. Local ISPs see a rationale to view local competing ISPs as peers, and they still admit the need to purchase trunk transit services from one or more upstream ISPs under terms of a client contract with the trunk provider ISP. Trunk ISPs see an acceptable rationale in peering with ISPs with a similar role profile in trunk transit but perceive an inequality of relationship with local ISPs. Of course, this balance of perceptions becomes clouded when a trunk ISP also participates in local markets as a peer retail player. We examine this in further detail when looking at peering options later in the book.
Physical Interaction: Exchanges and NAPs
One of the physical properties of electromagnetic propagation is that the power required to transmit an electromagnetic pulse over a distance varies in accordance with this distance. The shorter the distance between the transmitter and the receiver, the lower the power budget required; closer is cheaper. This statement holds true not only for power budgets but also for data protocol efficiency. Minimizing the delay between the sender and receiver allows the protocol to operate faster and operate more efficiently; closer is faster, and closer is more efficient.
These observations imply that distinct and measurable advantages are gained by localizing data traffic, that is by ensuring that the physical path traversed by the packets passed between the sender and the receiver is as short as possible. These advantages are realizable in terms of performance, efficiency, and cost. How then are such considerations factored into the structure of the Internet?
The Exchange Model
A strictly hierarchical model of Internet structure is one in which a small number of global ISP transit operators is at the top; a second tier is of national ISP operators; and a third tier consists of local ISPs. At each tier the ISPs are clients of the tier above as shown in Figure 14.2. If this hierarchical model were strictly adhered to, traffic between two local ISPs would be forced to transit a national ISP, and traffic between two national ISPs would transit a global ISP, even if both national ISPs operate within the same country. In the worst case, traffic between two local ISPs would need to transit a national ISP, a global ISP from one hierarchy, then a second global ISP, and a second national ISP to reach the local ISP. If the two global providers interconnect at a remote location, the transit path of the traffic between these two local ISPs can be very long indeed.
Figure 14.2 A purely hierarchical structure for the Internet.
On the campus where I worked in the late 1980s, data traffic between the campus and one of its technology vendors had to be passed from Australia to the east coast of the United States and back again, just to cross the street to the vendor's office in Australia. That this process worked at all was a constant source of wonder. That the ISP managed to deliver messages in seconds was a source of amazement at the time. At the same time the majority of traffic between Internet service networks in Europe included two Atlantic transits. Such stories of well-travelled packets that transit large amounts of the global communications infrastructure to arrive within a few meters of where the trafficoriginated are still, unfortunately, all too common in the Internet today. Ultimately, the cost of such inefficient traffic engineering is expressed in the price of goods and services on the Internet.
Such extended paths are inefficient, and the extended transit delay yields poor performance. Such paths are also costly, and such costs are ultimately part of the cost component of the price of Internet access. In a competitive market, strong pressure always is applied to reduce costs. Within a hierarchical environment, strong pressure is applied for the two national providers, who operate within the same market domain, to modify this strict hierarchy and directly interconnect their networks. Such a local interconnection allows the two networks to service their mutual connectivity requirements without payment of transit costs to their respective global transit ISP providers. At the local level is a similar incentive for the local ISPs to reduce their cost base, and a local interconnection with other local ISPs would allow local traffic to be exchanged without the payment of transit costs to the respective transit providers.
Although constructing a general interconnection regime based on point-to-point bilateral connections is possible, this approach does not exhibit good scaling properties. Between N providers, who want to interconnect, the outcome of such a model of single interconnecting circuits is N2 [ms] N circuits and N2 [ms] N routing interconnections as indicated in Figure 14.3. Given that interconnections exhibit the greatest leverage within geographical local situations, simplifying this picture within the structure of a local exchange is possible. In this scenario each provider draws a single circuit to the local exchange and then executes interconnections at this exchange. Between N providers who want to interconnect, the same functionality of complete interconnection can be constructed N point-to-point circuits.
Figure 14.3 Fully meshed peering.
The Exchange Router
One model of an exchange is to build the exchange itself as a router, as indicated in Figure 14.4. Each provider's circuit terminates on the exchange router, and each provider’s routing system peers with the routing process on the exchange router. This structure also simplifies the routing configuration, so that full interconnection is effected with N routing peer sessions. However, the exchange router model does become an active component of the interconnect peering policy environment. In effect, each provider must execute a multilateral interconnection peering with all of the other connected providers. Selectively interconnecting with a subset of the providers present at such a router-based exchange is not easily achieved. This type of exchange must execute its own routing policy. When two or more providers are advertising a route to the same destination, the exchange router must execute a policy decision as to which provider's route is loaded in the router's forwarding table, making a policy choice of transit provider on behalf of all other exchange-connected providers. Because the exchange is now an active policy element in the interconnection environment, the exchange is no longer completely neutral to all participants. This imposition on the providers may be seen as unacceptable, in that some of their ability to devise and execute an external transit policy is usurped by the exchange operator's policies.
Figure 14.4 An exchange router.
Typically, providers have a higher expectation of flexibility of policy determination from exchange structures than this base level. Providers want the flexibility to execute interconnections on a bilateral basis at the exchange and to make policy decisions as to which provider to prefer when the same destination is advertised by multiple providers. They require the exchange to be neutral with respect to such individual policy decisions.
The Exchange LAN
The modification to the inter-provider exchange structure is to use a LAN as the exchange element. In this model a participating provider draws a circuit to the exchange and locates a dedicated router on the exchange LAN. This structure is indicated in Figure 14.5. Each provider executes a bilateral peering agreement with another provider by initiating a router peering session with the other party's router. When the same network destination is advertised by multiple peers, the provider can execute a policy-bed preference as to which peer’s route will be loaded in the local forwarding table. Such a structure preserves the cost efficiency of using N circuits to effect interconnection at the exchange, while admitting the policy flexibility of N2 [ms] N potential routing peer sessions.
Figure 14.5 An exchange LAN.
Early inter-provider exchanges, such as the US interagency interconnection points FIX-East and FIX-West (FIX stands for Federal Internet eXchange), were based on an Ethernet LAN as the common interconnection element. This physical structure was simple and not all that robust under the pressures of growth. The LAN become congested easily. Subsequent refinements to the model have included the use of Ethernet switches as a higher capacity LAN and the use of FDDI rings, switched FDDI hubs, fast Ethernet hubs, and switched fast Ethernet hubs.
Exchanges are very high traffic concentration points. The desire to manage ever higher traffic volumes has lead to the anticipation of gigabit Ethernet switches as the next evolutionary technology step within exchanges. The model of the exchange location accommodates a model of diversity of access media, in which the provider's router undertakes the media translation between the access link protocol and the common exchange protocol.
The local traffic exchange hub does represent a critical point of failure with the local Internet topology. Accordingly, the exchange should be engineered in the most resilient fashion possible, using standards associated with a premium quality data center. This structure may include multiple power utility connections, uninterruptible power supplies, multiple trunk fiber connections, and excellent security measures.
The exchange should operate neutrally with respect to every participating ISP, with the interests of all the exchange clients in mind. Therefore, exchange facilities, which are operated by an entity that is not also a local or trunk ISP, enjoy higher levels of trust from the clients of the exchange.
Distributed Exchanges
Distributed exchange models also have been deployed in various locations. This deployment can be as simple as a metropolitan FDDI extension, in which the exchange comes to the provider's location rather than the reverse, as indicated in Figure 14.6. Other models that use an ATM-based switching fabric also have been deployed. Distributed exchange models attempt to address the cost of operating a single colocation environment with a high degree of resilience and security but do so at a cost of enforcing the use of a uniform access technology.
Figure 14.6 A distributed exchange.
However, the major challenge of such distributed models is that of switching speed. Switching requires some element of contention resolution, in which two ingress data elements that are addressed to a common egress path require the switch to detect the resource contention and then resolve it by serializing the egress. Switching, therefore, requires signaling, in which the switching element must inform the ingress element of switch contention. To increase the speed of the switch, the latency of this signaling must be reduced. The dictates of increased switching speed have the corollary of requiring the switch to exist within the confines of a single location.
Besides speed, we must consider the cost shift. In a distributed exchange model, the exchange operator operates the set of access circuits that form the distributed exchange. This process increases costs to providers, while it prevents the provider from using a specific access technology that matches their business requirements of cost and supportable traffic volume. Not surprisingly, to date the most prevalent form of exchange remains the third-party hosted colocation model. This model admits a high degree of diversity in access technologies, while still providing the substrate of an interconnection environment that can operate at high speed and therefore manage high traffic volumes.
Other Exchange-Located Services
The colocation environment is also broadened to include other functions, in addition to a pure routing and traffic exchange role. For a high-volume content provider, the exchange location offers minimal transit distance to a large user population distributed across multiple local service providers, as well as allowing the content provider to exercise a choice in selecting a nonlocal transit provider.
We also can add value to the exchange environment by providing additional functions and services, as well as terminating providers’ routers and large-volume content services. The exchange location within the overall network topology is an ideal location for hosting multicast services, because the location is quite optimal in terms of multicast carriage efficiency. Similarly, Usenet trunk feed systems can exploit the local hub created by the exchange. The overall architecture of a colocation environment that permits value-added services, which can productively use the unique environment created at an exchange, is indicated in Figure 14.7.
Figure 14.7 Exchange-located service platforms.
Network Access Points
The role of the exchange was broadened with the introduction of the Network Access Point (NAP) in the NSF-proposed post-NSFNET architecture of 1995.
The NAP was seen to undertake two roles: the role of an exchange provider between regional ISPs who want to execute bilateral peering arrangements and the role of a transit purchase venue, in which regional ISPs could execute purchase agreements with one or more of a set of trunk carriage ISPs also connected at the NAP. The access point concept was intended to describe access to the trunk transit service. The original four NAPs were established within the framework of a NSF solicitation. These four were located in San Francisco, operated by Pacific Bell, Chicago, operated by Ameritech, New Jersey, operated by Spring, and Washington DC, operated by Metropolitan Fiber Systems. Similar hybrid local exchange and transit purchase facilities have been set up in other North American cities as well as within cities in Europe and the Asia Pacific area.
This mixed role of both local exchange and transit operations leads to considerable operational complexity, in terms of the transit providers being able to execute a clear business agreement. What is the bandwidth of the purchased service in terms of requirements for trunk transit versus the access requirements for exchange traffic? If a local ISP purchases a transit service at one of the NAPs, does that imply that the trunk provider is then obligated to present all the ISP’s routes at remote NAP? How can a trunk provider distinguish between traffic presented to it on behalf of a remote client versus traffic presented to it by a local service client?
We also should consider the issue that the quality of the purchased transit service is colored by the quality of the service provided by the NAP operator. Although the quality of the transit provider's network may remain constant, and the quality of the local ISP’s network and ISP’s NAP access circuit may be acceptable, the quality of the transit service may be negatively impacted by the quality of the NAP transit.
One common solution is to use the NAP colocation facility to execute transit purchase agreements and use so-called backdoor connections for the transit service provision role. This usage restricts the NAP exchange network to a theoretically simpler local exchange roles. Such a configuration is illustrated in Figure 14.8.
Figure 14.8 Peering and transit purchase.
Exchange Business Models
For the ISP industry, a number of attributes are considered highly desirable for an exchange facility. The exchange should be as follows:
A continuing concern exists about the performance of exchanges and the consequent issue of quality of services that traverse the exchange. Many of these concerns stem from an exchange business model that may not be adequately robust under pressures of growth from participating ISPs.
The exchange business models typically are based on a flat-fee structure. The most basic model uses a fee structure based on the number of rack units used by the ISP to colocate equipment at the exchange. When an exchange participant increases the amount of traffic presented over an access interface, under a flat-fee structure, this increased level of traffic is not accompanied by any increase in exchange fees. However, the greater traffic volumes do imply that the exchange itself is faced with a greater traffic load. This greater load places pressure on the exchange operator to deploy further equipment to augment the switching capacity, without any corresponding increase in revenue levels.
For an exchange operator to base tariffs on the access bandwidths is not altogether feasible, given that such access facilities are leased by the participating ISPs and may not be known to the exchange operator. Nor is using a traffic-based funding model possible given that an exchange operator should refrain from monitoring individual ISP traffic across the exchange and given the unique position of the exchange operator. Accordingly, the exchange operator has to devise a fiscally prudent tariff structure at the outset that enables the exchange operator to accommodate large-scale traffic growth, while maintaining the highest possible traffic throughput metrics.
Alternatively are business models in which the exchange is structured as a cooperative entity between a number of ISPs. In these models the exchange is a nonprofit common asset of the cooperative body. This model is widely used but also one prone to the economic condition of the Tragedy of the Commons. It is in everyone's interest to maximize their exploitation of the exchange, while no single member wants to underwrite the financial responsibility for ensuring that the quality of the exchange is maintained.
Fourteenth-century Britain was organized as a loosely aligned collection of villages, each with a common pasture for villagers to graze horses, cattle, and sheep. Each household attempted to gain wealth by putting as many animals on the commons as it could afford. As the village grew in size, more and more animals were placed on the commons, and the overgrazing ruined the pasture. No stock could be supported on the commons thereafter. As a consequence of population growth, greed, and the tragedy of the commons, village after village collapsed.
The conclusion that can be drawn is that the exchange is an important component of Internet infrastructure, and the quality of the exchange is of paramount importance if it is to be of any relevance to ISPs. Using an independent exchange operator whose income is derived from the utility of the exchange is one way of ensuring that the exchange is managed proficiently and that the service quality is maintained for the ISP clients of the exchange.
A Structure for Connectivity
Enhancing the Internet infrastructure is quantified by the following objectives:
We have reached a critical point within the evolution of the Internet. The natural reaction of the various network entities will be to increase the complexity of the structure to preserve direct connectivity requirements. Today, we are in the uncomfortable position of increasingly complex connectivity. The inability to reach stable cost distribution models creates an environment in which each ISP attempts to optimize its position by undertaking as many direct 1:1 connections with peer ISPs as it possibly can. Some of these connections are managed via the exchange structure. Many more are implemented as direct links between the two entities. Given the relative crudity of the inter-AS routing policy tools that we use today, this structure must be a source of some considerable concern. The result of a combination of an increasingly complex mesh of inter-AS connections together with very poor tools to manage the resultant routing space is overall instability of the Internet environment. In terms of meeting critical immediate objectives, however, such dire general predictions do not act as an effective deterrent to these actions.
The result is a situation in which the inter-AS space is the critical component of the Internet. This space can be viewed correctly as the demilitarized zone within the politics of today's ISP-based Internet. In the absence of any coherent policy, or even a commonly accepted set of practices, the lack of administration of this space is a source of paramount concern.
Interaction Financials: Peering and Settlements
Any large multiprovider distributed service sector has to address the issue of cost distribution at some stage in its evolution. Cost distribution is the means by which various providers can participate in the delivery of a service to a customer who purchases a service from a single provider and be compensated for their costs in an equitable structure of interprovider settlement.
When an airline ticket is purchased from one provider, various other providers and service enterprises may play a role in the delivery of the service. The customer does not separately pay each airport baggage handler or other service provider. The customer’s original fare, paid to the original service provider, then is distributed by the service provider to other providers who incurred cost for providing components of the total service . These costs were incurred through a set of service contracts and inter-provider financial settlements, all of which are invisible to the customer.
The Internet is in a very similar situation. Some 60,000 constituent networks must interconnect in one fashion or another to provide comprehensive end-to-end service to each client. In supporting a data transaction between two clients, the two parties often are not clients of the same network. Indeed, the two client service networks often do not directly interconnect, and one or more additional networks must act in a transit provider role to service the transaction. Within the Internet environment, how do all the service parties to a transaction, who incur cost in supporting the transaction, receive compensation for their cost? What is the cost distribution model of the Internet?
Here, we examine the basis for Internet inter-provider cost distribution models and then look at the business models currently used in the inter-provider Internet environment. This area commonly is termed financial settlement, a term the Internet picked up from the telephony industry.
The Currency of Interconnection
What exactly is being exchanged between two ISP’s who want to interconnect? In the sense of the meaning of currency as the circulating medium, the question is what precisely is being circulated at the exchange and within the realm of interconnection? The answer to the question is: routing entries. When two parties exchange routing entries, the outcome is that traffic flows in response to the flow of routing entries. The flows are in opposite directions, as indicated in Figure 14.9, so that a bilateral routing-mediated flow occurs only when routes are passed in both directions.
Figure 14.9 Routing and traffic flows.
Within the routing environment of an ISP are a number of different route types, based predominately on the way in which the route has been acquired by the ISP:
How then should the ISP export routes so that the inbound traffic flow matches the outbound flows implied by this route structure?
Clients. All available routes in the preceding four categories, with the exception of internal ISP service functions, should be passed to clients, either in the form of a default route or as explicit route entries passed via a BGP session
Upstream providers. All client routes and all internal ISP routes corresponding to Internet-wide services should be passed to upstream providers. Some clients may want some restrictions placed on their routes being advertised in such a fashion. The ability for a client to specify such caveats on the routing structure and the mechanism to allow it to happen should be clearly indicated in the service contract.
Peer ISPs. All client routes and all ISP routes corresponding to Internet-wide service should be passed to peer ISPs. Again the client may want to place a restriction on such an advertisement of their routes.
This structure is shown in Figure 14.10.
Figure 14.10 External routing interaction.
The implicit outcome of this structure is that the ISP does not act in a transit role to peer ISPs and does not permit peer-to-peer transit nor peer-to-upstream transit. Peer ISPs have visibility only to clients of the ISP. From the service visibility perspective, client-only services are not visible to peer ISPs or upstream ISPs. Therefore, value-added client services are implicitly visible only to clients and only when they access the service through a client channel.
Settlement Options
Financial settlements have been a continual topic of discussion within the domain of Internet interconnection. To look at the Internet settlement environment, let’s first look at the use of inter-provider financial settlements within the international telephony service industry. Then, we will look at the application of these generic principles to the Internet environment.
Within the traditional telephony model, inter-provider peering takes place within one of three general models:
The accounting settlement call rate is measured in units of call minutes, and the actual rate used is negotiated bilaterally between the two parties. The Federal Communications Commission of the United States (FCC) asserts that U.S. telephone operators paid the United States $5.6 billion in settlement rates in 1996, and the FCC is voicing the view that accounting rates have shifted into areas of noncost-based settings.
This accounting settlement issue is one of the drivers behind the increasing interest in Voice over IP solutions, because typically no settlement component exists in such solutions, and the call termination charges are cost-based, without bilateral price setting.
The telephony settlement model is by no means stable, and currently significant pressure is being placed on the international accounting arrangements to move away from bilaterally negotiated call accounting rates to a more general adoption of an SKA model. Simultaneously, communications deregulation within many national environments is changing the transit fee model, as local providers extend their network into the long-distance area and commence peering arrangements with similar entities. Criticism also has been directed at the bilaterally negotiated settlement rates, because the rates are not cost-based rates but are based on a desire to create a revenue stream from accounting rate settlements.
Internet Settlements
A number of critical differences exist between the telephony models of interconnection and the Internet environment, which have confounded all attempts to cleanly map telephony interconnection models into the Internet environment.
Internet interconnection accounting is a packet-based accounting issue, because there is no call in the Internet. Therefore, the most visible difference between the two environments is the replacement of the call with the packet as the currency unit of interconnection. Althoughwe can argue that a TCP session has much in common with a call, this concept is not readily identified within the packet forwarding fabric, thus it is not readily apparent to the network who initiated the TCP session. Unlike a telephony call, no concept of state initiation exists to pass a call request through a network and lock down a network transit path in response to a call response. The network undergoes no state change in response to a TCP session, and therefore,no means is readily available to identify that a call has been initiated and by which party.
Packets may be dropped. When a packet is passed across an interconnection from one provider to another, no firm guarantee is given by the second provider that the packet will definitely be delivered to the destination. The second provider (or subsequent providers in the transit path) may drop the packet for quite legitimate reasons and will remain within the protocol specification in so doing. Indeed, the TCP protocol uses packet drop as a rate-control signal. For the efficient operation of the TCP protocol, some level of packet drop is a useful and anticipated event. However, if a packet is used as the accounting unit in a general cost distribution environment, should the provider who receives and subsequently drops the packet be able to claim an accounting credit within the interconnection? The logical response is that such accounting credits should apply only to delivered packets, but such an accounting structure is highly challenging to implement accurately.
Packet header contents are within the explicit control of the end user, not the provider. Users can exercise some significant level of control of the path a packet takes to transit the Internet if source routing is honored so that the relative packet flows between two providers can be manipulated by the client if so desired.
Packet forwarding is not a verified operation. A provider may chose to forward a packet to a second provider without reference to the particular routes the second provider is advertising to the first party. A packet also may be forwarded to the second provider with a source address that is not being advertised to the second provider. Given that the generic Internet architecture strives for robustness under extreme conditions, attempts to forward a packet to its addressed destination are undertaken irrespective of how the packet may have arrived at this location in the first place, and irrespective of how a packet with reverse header IP addresses will transit the network.
Routing information is not uniformly available. Complete information is not available to the Internet regarding the status and reachability of every possible Internet address. Only as a packet is forwarded closer to the addressed destination does more complete information regarding the status of the destination address become apparent to the provider. Accordingly a packet may have incurred some cost of delivery before its ultimate undeliverability becomes evident. An intermediate also provider can never be completely assured that a packet is deliverable.
These points indicate that a packet-based accounting system for interconnection is the only available settlement mechanism and that such a model is prone to levels of abuse because packets can be passed across an interconnection in either direction under the explicit direction of an end user or a provider. This foundation is not the most stable for a large-scale and high-value monetary settlement structure.
Settlement Models for the Internet
Where a wholesale or retail service agreement is in place, one ISP is in effect a customer of the other ISP. In this relationship, the customer ISP (downstream ISP) is purchasing transit and connectivity services from the supplier ISP (upstream ISP). The downstream ISP resells this service to its clients. The upstream ISP must announce the downstream ISP’s routes to all other customers and other egress points of the ISP’s networks to honor the service contract to the downstream ISP customer.
However, given two ISP‘s who interconnect, which party should assume the upstream provider role and which party should assume the downstream customer role is not always immediately obvious to either party or to an outside observer. Greater geographic coverage may be the discriminator here that allows the customer/provider determination. However, this factor is not the only possible one within the scope of the discussion. One ISP may host significant content and may observe that access to this content adds value to the other party’s network, which may be used as an offset against a more uniform customer relationship. In a similar vein, an ISP with a very large client population within a limited geographic locality may see this large client base as an offset against a more uniform customer relationship with the other provider. In many ways, the outcome of these discussions can be likened to two animals meeting in the jungle at night. Each animal sees only the eyes of the other, and from this limited input, the animals must determine which animal should attempt to eat the other!
An objective determination of which ISP should be the provider and which the client is not always possible. In many contexts, the question is inappropriate, given that for some traffic classes the respective roles of provider and client may swap over. The question often is rephrased along the lines of, " Can two providers interconnect without the implicit requirement to cast one as the provider and the other as the client?" Exploration of some concepts of how the question could possibly be answered is illustrative of the problem here.
Packet Cost Accounting
One model to use is that a packet incurs cost as it is passed through the network. For a small interval of time, the packet occupies the entire transmission capacity of each circuit over which it passes. Similarly, for a brief interval of time, the packet is exclusively the switching fabric of the router. The more routers the packet passes through, and the greater the number and distance of transmission hops the packet traverses, the greater the incurred cost in carrying the packet.
A potential settlement model could be constructed from this observation. The strawman model is that whenever a packet is passed across a network boundary, the packet is effectively sold to the next provider. The sale price increases as the packet transits through the network, accumulating value in direct proportion to the distance the packet traverses within the network. Each boundary packet sale price reflects the previous sale price, plus the value added in transiting the ISP’s infrastructure. Ultimately, the packet is sold to the destination client. This model is indicated in Figure 14.11.
Figure 14.11 Financial inter-provider settlement via packet cost accounting.
As with all strawmen models, this one has a number of critical weaknesses, but let’s look at the strengths first. An ISP gains revenue from a packet only when delivered on egress from the network, rather than in network ingress. Accordingly, a strong economic incentive exists to accept packets that will not be dropped in transit within the ISP, given that the transmission of the packet only generates revenue to the ISP on successful delivery of the packet. This factor places strong pressure on the ISP to maintain quality in the network, because dropped packets imply foregone revenue on local transmission. Because the packet was already purchased from the previous provider in the path, packet loss also implies financial loss. Strong pressure also is exerted to price the local transit function at a commodity price level, rather than attempt to undertake opportunistic pricing. If the chosen transit price is too great, the downstream provider has the opportunity to extend the network to reach further upstream in the path, bypassing the ISP and purchasing the traffic directly from the upstream source. Accordingly, this model of per-packet pricing, using a settlement model of egress volumes and local value added to the per-packet price based on transmission costs, does allow for reasonable stability and cost distribution.
However, weaknesses of this potential model cannot be ignored. First, some level of packet drop is inevitable irrespective of traffic load. Generally, the more remote the sender from the destination, the less able the sender is to ascertain that the destination address is a valid IP address, and the destination host is available. To minimize the liability from such potential packet loss, the ISP should maintain a relatively complete routing table and only accept packets in which a specific route is maintained for the network. More critical is the issue that the mechanism is open to abuse. Packets, which are generated by the upstream ISP, can be transmitted across the interface, which in turn results in revenue being generated for the ISP. Of course, per-packet accounting within the core of the network is a significant refinement of existing technology. Within a strict implementation of this model, packets require the concept of an attached value that ISPs augment on an ingress-to-egress basis, which could be simplified to a hop-by-hop value increment. Implementations feasibly can use a level of averaging to simplify this by using a tariff for domestic transit and a second for international transit.
TCP Session Accounting
These traffic-based metrics do exhibit some weaknesses because of their inability to resist abuse and likelihood of exacting a price even when the traffic is not delivered to an ultimate destination. Does an alternative settlement structure that can address this exist? One approach is to perform significantly greater levels of analysis of the traffic as it transits a boundary between a client and the provider or between two providers and to adopt financial settlement measures that match the type of traffic being observed. As an example, the network boundary could detect the initial TCP SYN handshake, and all subsequent packets within the TCP session could be accounted against the session initiator, while UDP traffic could be accounted against the UDP source.
Although such settlement schemes are perhaps limited more by imagination in the abstract, however, very real technical considerations must be borne to bear on this speculation. For a client-facing access router to detect a TCP flow and correctly identify the TCP session initiator requires the router to correctly identify the initial SYN handshake, the opening packet, and then all packets within this TCP flow against this accounting element. This identification may be completely impossible within the network at an interprovider boundary. The outcome of the routing configuration may be an asymmetric traffic path, on which a boundary may see only traffic passing in a single direction.
However, the greatest problem with this model is the diversity of retail pricing structures that exist within the Internet today. Some ISPs use pricing based on received volume, some on sent volume, some on a mix of sent and received volume, and some use pricing based on the access capacity irrespective of volume. This discussion leads to the critical question when considering financial settlements: Considering that the end client is paying the local ISP for comprehensive Internet connectivity, when a client's packet is passed from one ISP to another at an interconnection point, where is the revenue for the packet? Is the revenue model one the packet sender pays or one the packet receiver pays? The model described here assumes a uniform retail model in which the receiver pays for Internet packets. This is simply not mirrored within the retail environment of the Internet today.
Although this session-based settlement model does promote a quality environment with fair carriage pricing, it cannot address the fundamental issue of financial settlements.
Criteria for Settlements
For a financial settlement structure to be viable and stable, the settlement structure must be a uniform abstraction of a relatively uniform retail tariff structure. This conclusion is critically important.
The financial structure of interconnection must be an abstraction of the retail models used by the two ISPs. If the uniform retail model is used, the party originating the packet pays the first ISP a tariff to deliver the packet to its destination within the second ISP; then the first ISP is in a position to fund the second ISP to complete the delivery through an interconnection mechanism. If, on the other hand, the uniform retail model is used in which the receiver of the packet funds its carriage from the sender, then the second ISP funds the upstream ISP. If no uniform retail model is used, when a packet is passed from one provider to the other, no understanding exists about which party receives the revenue for the carriage of the packet and accordingly which party settles with the other party for the cost incurred in transmission of the packet.
The answer to these issues within the Internet environment has been to commonly adopt just two models of interaction. These models sit at the extreme ends of the business spectrum, where one is a customer/provider relationship, and the other is a peering relationship without any form of financial settlement, or SKA. These approximately correspond to the second and third models described previously from traditional models of interconnection within the communications industry. However, an increasing trend has moved towards models of financial settlement in a bilaterally negotiated basis within the Internet. Observing the ISP industry repeat the same well-trodden path of the international telephony world is somewhat interesting.
SKA Settlement
SKA Peering arrangements are those in which traffic is exchanged between two or more ISPs without mutual charge (an interconnection arrangement with no financial settlement). Within a national structure, typically the marginal cost of international traffic transfer to and from the rest of the Internet is significantly higher than domestic traffic transfer. In such cases, any SKA peering is likely to relate to only domestic traffic, and international transit would either be provided by a separate agreement or provided independently by each party.
This SKA peering model is most stable where the parties involved perceive equal benefit from the interconnection. This interconnection model generally is used in the context of interconnection or with providers with approximate equal dimension, as in peering regional providers with other regional providers, national providers with other national providers, and so on. Oddly enough, the parties themselves do not have to agree on what that value or dimension may be in absolute terms. Each party makes an independent assessment of the value of the interconnection, in terms of the perceived size and value of the ISP and the value of the other ISP. If both parties reach the conclusion that in their terms a net balance of value is achieved, then the interconnection is on a stable basis. If one party believes that it is larger than the other and SKA interconnection would result in leverage of its investment by the smaller party, then an SKA interconnection is unstable.
Even with the mutual perception of equality of dimension and value, SKA peering is unlikely to remain stable unless both networks provide infrastructure functions independently or share the operational load of provision of infrastructure functions. These functions include USENET news flows, NTP reference signals, DNS forwarders and caching services, information caches, multicast services, and similar infrastructure services.
The essential criteria for a stable SKA peering structure is perceived equality in the peering relationship. This can be achieved in a number of ways, including the use of entry barrier pricing into the peering environment or the use of peering criteria (such as the specification of ISP network infrastructure or network level of service and coverage areas as eligibility for peering).
A typical feature of the SKA peering environment is to define an SKA peering in terms of traffic peering at the client level only. This definition forces each peering ISP to be self sufficient in the provision of transit services and ISP infrastructure services (such as the requirement to provide services including DNS, NTP, USENET, and so on) that would not be provided across a peering point. This process may not result in the most efficient or effective Internet infrastructure, but it does create a level of approximate parity and reduces the risks of leverage within the interconnection. In this model, each ISP presents at each interconnection or exchange only those routes associated with the ISP’s customers and accepts only traffic from peering ISPs at the interconnection or exchange directed to such customers. The ISP does not accept transit traffic destined to other remote exchange locations, nor to upstream ISPs, nor traffic directed to the ISP‘s infrastructure services. Equally, the ISP does not accept traffic, which is destined to peering ISPs, from upstream transit providers. The business model here is that each client of an ISP is contracting the ISP to present their routes to all other customers of the ISP, to the upstream providers of the ISP, and to all exchange points where the ISP has a presence. The particular tariff model chosen by the ISP in servicing the customers is not material to this interconnection model. Traffic passed to a peer ISP at the exchange becomes the responsibility of the peer ISP to pass to their customers at their cost.
Another means of generating equity within an SKA peering is to peer only within the terms of a defined locality. In this model, an ISP would present routes to an SKA peer in which the routes corresponded to customers located at a particular access POP, or a regional cluster of access POPs. The SKA peer’s ability to leverage advantage from the greater level of investment (assuming that the other party is the smaller party) is now no longer a factor, because the smaller ISP sees only those parts of the larger ISP that sit within a well-defined local or regional zone. This form of peering is indicated in Figure 14.12.
Figure 14.12 SKA peering using local cells.
The probable outcome of widespread use of SKA interconnections is a generalized ISP domain along the lines of Figure 14.13. Here, the topology is segregated into two domains—a set of transit ISPs whose predominate investment direction is in terms of high-capacity carriage infrastructure and high-capacity switching systems and a collection of local ISPs, whose predominate investment direction is in service infrastructure supporting a string retail focus. Local ISPs participate at exchanges and announce local routes at the exchange on an SKA basis of interconnection with peer ISPs. Such ISPs are strongly motivated to prefer to use all routes presented at the exchange within such peering sessions, as the ISP is not charged any transit cost for the traffic under an SKA settlement structure. The exchange does not provide comprehensive connectivity to the ISP, and this connectivity needs to be complemented with a separate purchase of transit services. In this role, the local ISP becomes a client of one or more transit ISPs explicitly for the purpose of access to transit connectivity services.
Figure 14.13 ISP structure of local and transit operations.
In this model, the transit ISP must have established a position of broad ranging connectivity, with a well-established and significant market share of the wholesale transit business. A transit ISP also must be able to present customer routes at a carefully selected set of major exchange locations and have some ability to exchange traffic with all other transit ISPs. This latter requirement has typically been implemented using private interconnection structures, and the associated settlements often are negotiated bilaterally. These settlements possibly may include some element of financial settlement.
Financial Settlement
The alternative to SKA and provider/client role selection is the adoption of a financial settlement structure. The settlement structure is based on both parties effectively selling services to each other across the interconnection point, with the financial settlement undertaking the task of balancing the relative sales amounts.
The simplest form of undertaking this settlement is to measure the volume of traffic being passed in each direction across the interconnection and to use a single accounting rate for all traffic. At the end of each accounting period, the two ISPs would financially settle based on the agreed accounting rate applied to the net traffic flow.
Which way the money should flow in relationship to traffic flow is not immediately obvious. One model assumes that the originating provider should be funding the terminating provider to deliver the traffic, and therefore, money should flow in the same direction as traffic. The reverse model assumes that the overall majority of traffic, to the level of 80–85 percent of all ISP traffic, is traffic generated by an action of the receiver, such as Web page retrieval or the downloading of software. Therefore, the network cost should be imposed on the discretionary user, so that the terminating provider should fund the originating provider. This latter model has some degree of supportive evidence, because a larger provider will, generally, provide more traffic to a smaller attached provider than it receives from that provider. Observation of traffic statistics would bear this out, indicating that traffic-received volumes are a preferable means of determining the relative interconnection benefit to two providers.
The accounting rate can be negotiated to be any amount. Although in the telephony industry settlement domain, in which a rate is not cost-based, business instability issues arise. The agreed settlement traffic unit rate would have to match the average marginal cost of transit traffic in both ISP networks for the settlement to be attractive to both parties.
Refinements to this approach can be introduced, although they are accompanied by significant expenditure on traffic monitoring and accounting systems. The refinements are intended to address the somewhat arbitrary determination of financial settlement based on the receiver or the sender. One way is to undertake flow-based accounting, in which the cost accounting for the volume of all packets associated with a TCP flow is directed to the initiator of the TCP session. Here, the cost accounting for all packets of a UDP flow is directed to the UDP receiver. The session-based accounting is significantly more complex than simple volume accounting, and such operational complexity would be reflected in the cost of undertaking such a form of accounting. However, asymmetric paths are a common feature of the inter-AS environment, so that it may not always be possible to see both sides of a TCP conversation and perform an accurate determination of the session initiator.
Another refinement is to use a different rate for each provider, where the base rate is adjusted by some agreed size factor to ensure that the larger provider is not unduly financially exposed by the arrangement. The adjustment factor can be the number of POPs, range of the network, volume carried on the network, number of routes advertised to the peer, or any other metric related to the ISP’s investment and market share profile. Alternatively, a relative adjustment factor can be a number to which both parties agree.
Of course, traffic flows on the Internet are not as rigidly structured as those within the telephony environment, and within this Internet interconnection environment, readily determining which party’s client generated a bidirectional traffic flow across the peering structure. This is why the coarse metric of traffic volumes in each direction is often chosen as the basis of the financial settlement. Of course, such a relative traffic volume balance is not very robust either, and the metric is one infinitely adjustable and potentially infinitely abuseable. The capability to adjust the relative traffic balance comes from the direct relationship between the routes advertised and the volume of traffic received. To reduce the amount of traffic received, the ISP reduces the number of routes advertised to the corresponding peer. Increasing the number of routes, and at the same time increasing the number of specific routes increases the amount of received traffic. Where there is a rich mesh of connectivity, there is a strong financial incentive for each party to adjust the routing parameters to match the lowest financial expenditure by using restricted route advertisements with the greatest levels of revenue by using a local preference for received routes (with the highest preference for client-advertised routes and the next level of preference for financial settled peer advertised routes). Such settings of the routing system may not necessarily correspond to the optimal traffic path in network engineering terms, nor will these settings necessarily result in a highly stable routing and traffic configuration.
Of far greater concern is the ability to abuse the interconnection arrangements. One party can generate and then direct large volumes of traffic to the other party. Although overt abuse of the arrangements is often easy to detect, greed is a wonderful stimulant to ingenuity, and more subtle forms of abuse of this interconnection are always possible. However, both parties typically indicate in an interconnection agreement their undertaking not to indulge in such forms of deliberate abuse of the interconnection. Third parties can still abuse the interconnection in various ways. Loose source routing can allow traffic, which passes across the interconnection in either direction, to be generated. The ability to remotely trigger traffic flows through source address spoofing is possible even where loose source routing is disabled. This window of financial vulnerability is far wider than many ISPs are comfortable with, because it opens the provider to a significant liability over which it has a limited ability to detect and control. Consequently, financial settlement structures based on traffic flow metrics are not a commonly deployed mechanism, as they introduce unacceptable financial risks to the ISP into the interconnection environment.
No Interconnection
Examining the option of complete autonomy of operation, without any form of interaction with other local or regional ISPs, is instructive.
One scenario for a group of ISPs is that a mutually acceptable peering relationship cannot be negotiated, and all ISPs operate disconnected network domains with dedicated upstream connections and no interconnection. The outcome of such a situation is that third-party connectivity would take place, with transit traffic flowing between the local ISPs being exchanged within the domain of a mutually connected third-party ISP (or via transit across a set of third-party ISPs). For example, for an Asian Pacific country, this structure would result in traffic between two local entities, both located within the same country, being passed across the Pacific, routed across a number of network domains within the United States, and then passed back across the Pacific. Not only is this inefficient in terms of resource utilization, this structure also adds a significant cost to the operation of the ISPs, a cost that ultimately is passed to the consumer in higher prices for Internet traffic.
Note that this situation is not entirely novel; the Internet has seen such arrangements appear in the past; and such situations are still apparent in today's Internet. Such arrangements have arisen, in general, as the outcome of an inability to negotiate a stable local peering structure.
However, such positions of no interconnection have proved to be relatively short-lived due to the high cost of operating such international transit environments, the instability of the significantly lengthened interconnection paths, and the unwillingness of foreign third-party ISPs to act (often unwittingly) as agents for domestic interconnection in the longer term. As a result of these factors such off-shore connectivity structures generally have been augmented with domestic peering structures.
The resultant general operating environment of the Internet is that effective isolation is not in the best interests of the ISP, nor is isolation in the interests of other ISPs, nor in the best interests of the consumers of the ISPs’ services. In the interests of a common desire to undertake rational and cost-effective use of communications’ resources, each national (or regional) collection of ISPs act to ensure local interconnectivity between such ISPs. A consequent priority is to reach acceptable ISP peering arrangements.
Futures
A number of aspects of the ISPs’ interconnection environment may have ramifications in terms of the future profile of the ISP industry worldwide.
Currently, the growth of the Internet market is acting as the major impetus to the ISP market. When this level of growth tapers off, widespread expectation calls for some level of rationalization to occur, with the number and diversity of ISPs tapering off as a part of this rationalization. The rationale behind this expectation is that in a more static market, larger players can use various economies of scale to ensure that their operation is cost efficient. This efficiency can be reflected in their retail pricing structure. Also, larger investors can afford to take longer term positions on the market, sustaining an initial period of operating loss to realize desired longer term market share positions
However, another factor is at play in the longer term future of the industry. This factor is this basic issue of cost distribution within a multiprovider environment. Without a robust mechanism that allows for the incremental costs associated with the carriage of traffic to be apportioned to each provider within the transit path, a fundamental barrier exists to the stability of a well-populated diverse ISP market place servicing vertical market sectors. The business outcomes from this lack of effective cost distribution are that smaller industry players are unable to establish a stable revenue stream from the transit costs associated with interconnection.
Cost distribution is an essential attribute of a diverse provider environment. In the absence of cost distribution is a challenge to sustain a diverse provider industry, and the outcome may be a consolidation into a very small number of ISP transit operators providing service on a global scale. This very small number probably will never become just one, but a set of global alliances may see an outcome along these lines of between two and six major global transit providers, with each transit operator undertaking private peering using SKA settlement with its transit peers.
Within the local ISP environment is more scope for a diversity of players in an SKA-based interconnection environment, which can be sustained relatively easily. Consolidation, if occurring in this sector, will be a result of economies of scale rather than an outcome of the economics of interconnection.
Strong pressure to change the technology base to accommodate more sophisticated settlement structures is unlikely to emerge. The fundamental observation is that any financial settlement structure is robust only where a relatively widely accepted retail model exists. Although a diversity of retail mechanisms is available, the stability of a financial settlement structure is somewhat dubious.