UGA's Next-Generation Network

(Revised 5-11-05)

 

Overview

 

UGA's network consists of three layers – core, distribution and access. The core layer is composed of communications devices interconnected via fiber optic cabling to which each building network connects.  The distribution layer consists of the communications equipment located in main wiring closets (called MDFs) in buildings that connect to the core network via fiber cabling and to distributed wiring closets (called IDFs) either via copper or fiber cabling.  Networked devices typically connect via copper cabling to communications equipment in IDFs which constitutes the access layer of the network.

 

Network Review Process

 

In the Fall of 2003, EITS initiated a process of reviewing the UGA campus network with the goal of planning for a next-generation network.  A network review team (consisting of Mike Dennis, Stan Gatewood, Paul Keck and David Matthews-Morgan) worked together throughout the process to insure that due diligence practices were followed and that network and security needs were addressed.

 

The first phase in the process was to document how the current network is put together and to send that document to several networking vendors for feedback.  The resulting Request for Information (RFI) document was reviewed by Gartner and was given to six vendors – Cisco, Foundry, Enterasys, Extreme, Nortel and HP – in late December 2003.  The vendors were asked to critique our network and provide a vision of how their equipment can best serve our network and why their equipment/vision is the best solution for UGA's network of the future.

 

Five of the six vendors (excluding HP) responded in writing to UGA's RFI document by February 6, 2004.  Each of the vendors also provided formal presentations of their responses to EITS staff between late February and early March.  After all of the presentations were given and subsequently reviewed, a meeting was held with a Gartner network analyst.  The Gartner analyst recommended that UGA migrate to a collapsed, fully-meshed, 10 Gigabit per second, 4-device core network from the current, partially-meshed, 2 Gigabit per second, 16-device Foundry core.  (Note: The current Foundry core devices can no longer be purchased from the vendor.)  Gartner's view was that this recommended core network topology represented a “best practice” in the industry.  This was also the recommendation of at least two of the five vendors during their presentations.  The following are some of the advantages of having a four-core network:

 

·        Capital and maintenance cost savings

·        Easier to manage

·        Simpler design for future upgrades

·        Future network technologies more affordable and easier to implement

 

One of the goals in planning for the next-generation network has been to have a single vendor for the new core network layer, one networking vendor for the distribution layer and one for the access layer of the UGA network.  A second RFI document was developed which described the current UGA network in terms of core, distribution and access layers; the desired, next-generation, layered network; and the functional network and security requirements needed at each layer.  (The next-generation core and distribution networks are depicted in Appendix A.  The functional network and security requirements are documented in Appendices B and C, respectively.)  It also described the RFI deliverables expected of the vendors from whom the University wished to solicit a response.  In order to keep this phase of the review process manageable, it was decided that three vendors would be provided a copy of the second RFI.  Gartner recommended that the RFI be sent to Cisco and Nortel, and UGA chose Foundry as the third vendor because of our positive experience with Foundry both as our current core network equipment provider as well as a provider of network equipment in both the distribution and access layers within some buildings on campus.

 

Each vendor who responded to this RFI was asked to document the “carrier-class” capabilities of the device they recommend for the core network.  They were also asked to describe how their proposed product(s) meet the functional network and security requirements listed in the RFI document at the core, distribution and access layers.  In addition, the vendors were asked to demonstrate how their solution can reduce the network and security operations and management costs associated with the proposed UGA network infrastructure.  Vendors provided aggressive pricing information for the solutions they recommend and proposed training opportunities for UGA network and security personnel.  RFI responses were to be provided in two forms – a written document covering the issues articulated above and a formal presentation of the vendor's solution and its ability to satisfy UGA's design requirements.  The written document was due by Friday, October 8, 2004.  Formal vendor presentations were to be scheduled after that date.

 

Based on the written responses to the second RFI, the review team decided that Nortel's response did not meet all of the network and security requirements.  Therefore, only Cisco and Foundry were invited to give formal presentations highlighting their RFI responses.  After these presentations, the review team analyzed the responses and sought the advice of another Gartner network analyst in late December 2004 regarding those responses.  In addition, a technical team consisting of Network Operations and Infrastructure staff met with both Cisco and Foundry technical personnel to discuss and clarify their proposed core solutions.  Based on the review team's assessment and the feedback from the Gartner analyst, the technical team and other universities who have implemented Cisco and Foundry products, the review team has developed the following recommendations for the core, distribution and access layers of UGA's next-generation network (NGN).

 

NGN Core Network

 

The NGN core layer will consist of four locations – Boyd (Data Center), Peabody, Life Sciences and Stegeman Coliseum – each containing a fully-meshed Ethernet switch device interconnected via 10 Gigabit per second links.  Since the core layer will be collapsed from 16 to 4 devices, it will be imperative for the core switches to be “carrier-class” meaning that every component in a core switch must be fully-redundant and fault-tolerant.  This will be especially important since each core device will have from 60 to 83 buildings connected to it in a redundant fashion.  IPv4, IPX and AppleTalk communications protocols are currently routed across the core.  However, to simplify the core network and reduce the processing strain on core devices as well as reduce the technical knowledge needed to support them, only IPv4 traffic will be permitted across the NGN core.

 

Both Cisco and Foundry offered solutions that met the functional network and security requirements for the core.  However, Foundry's solution is recommended for the core network for the following reasons:

 

 

It is estimated that $900,000 will be needed for the four core devices and spare parts.

 

NGN Distribution Network

 

At the distribution layer, there are a total of 143 building networks directly connected to the current 16-device core (with anywhere from 2 to 14 building networks connected to a given core device).  Most of the building networks connect at the distribution layer using one fiber Gigabit Ethernet port through an Ethernet switch (predominantly Enterasys equipment).  A few building networks have one fiber Gigabit Ethernet connection to a core through an Ethernet router (also called a layer-three switch) from either Foundry, Cisco, Extreme or HP.  The layer-three switches usually only support IPv4 traffic.  The current Enterasys switches have components that can no longer be purchased and will not be supported after July 1, 2007.  Although Enterasys sells a next-generation Ethernet switch, because of financial and corporate instability (compared with the other network vendors being considered), it would not be wise to purchase replacement components from that vendor.

 

Both Cisco's and Foundry's solutions satisfied the functional network and security requirements at the distribution layer.  However, since Foundry is recommended for the core layer and it is also recommend for the access layer, it would be best to utilize Foundry layer-three switches within the distribution layer as well.  In addition, having Foundry devices in the core and distribution layers will provide flexible, interoperable building connectivity options.  EITS and other departments have successfully deployed Foundry equipment in a number of building MDFs.  It is estimated that $2 million will be needed to replace the current Enterasys equipment in building MDFs with Foundry switches that provide the same connectivity to building IDF communications devices.

 

In order to provide greater redundancy given that there will be only four core devices, each Foundry layer-three switch in building MDFs will connect to two different core devices.  Because there is insufficient fiber optic cabling to provide these dual connections, additional fiber will need to be installed in the existing conduit system.  The conduit system is laid out such that a North-South fiber corridor connecting the Peabody and Coliseum core locations and an East-West fiber corridor connecting the Boyd and Life Science core locations would constitute an optimal fiber design.  The cost of installing the additional fiber (with some spare fiber for specialized connectivity needs) is estimated to be $1 million.

 

NGN Access Network

 

As mentioned previously, the access layer deals with the manner in which individual devices connect to building networks via IDFs.  Based on survey data provided by departmental UGA network personnel, the equipment in IDFs range from unmanaged, 10 Megabit per second shared Ethernet hubs to a variety of managed and unmanaged Ethernet switches from at least 16 vendors (the predominant switch vendors being Enterasys and Foundry).  A few departments have deployed some form of hardware firewall to protect networked devices.

 

Although networking devices in the access layer (i.e., in IDFs) are currently a hodgepodge of equipment from a number of vendors with varying capabilities, it is UGA's desire to eventually replace undesirable networking hubs and switches with function-rich, security-aware, centrally-supported devices from one networking vendor.  As was the case with the core and distribution layers, both Cisco's and Foundry's solutions claimed to meet the functional network and security requirements at the access layer.

 

One of the key security requirements at the access layer is the ability for the network to defend itself in the event that a compromised or inadequately protected/configured device is connected to the building network.  This self-defending network capability is achieved by interrogating the connected device to insure that appropriate operating system patches have been applied, that up-to-date versions of antivirus software are installed, and that other security software such as desktop firewalls are installed and activated.  Although Cisco has been talking about its self-defending network architecture for some time, it does not currently have this capability in switches that would be placed in building IDFs.  Foundry, on the other hand, has announced this self-healing capability in their current switch products through partnerships with third-party security software providers such as Checkpoint and Sygate.

 

Another feature Foundry has in their switches that Cisco lacks is the ability to monitor traffic patterns (or flows) to and from individually connected devices.  Having the ability to monitor individual device flows would enable security monitoring tools to proactively detect inappropriate network behavior from a device and (if desired) automatically disconnect the device from the network.  Gartner felt that this feature gave Foundry an edge over Cisco in terms of securing monitoring.  Since Foundry has additional security features that Cisco doesn't have, it is recommended that Foundry switches become the standard for building IDFs.

 

Pricing Considerations

 

Both Cisco and Foundry were asked to provide competitive discounts for both hardware procurements and maintenance for a period of three years with the understanding that EITS would standardize its network equipment purchases on the selected vendor and would strongly encourage the rest of the UGA community to purchase their network equipment from that vendor.  Cisco and Foundry provided substantive discounts on both hardware and maintenance over state contract pricing. Cisco's discount applied to “certain Cisco products and maintenance services for its [UGA's] Core Switching Upgrade Project” and committed to an additional trade-in discount for this project.  After analyzing the cost of comparable Cisco and Foundry equipment for the Boyd core and Southern Light Rail, even with the additional trade-in discount offered by Cisco, the cost of the Foundry equipment was nearly $20,000 less that the Cisco equivalent.  Without the trade-in, Foundry would have been approximately $50,000 less expensive than Cisco for the Boyd core equipment.

 

Summary

 

The next-generation UGA network will consist of four core devices interconnected in a fully-meshed fashion via 10 Gigabit per second links.  Building networks will connect to two core devices utilizing layer-three switches in MDFs.  Additional fiber will be installed to provide redundant connections from the MDF devices to the core.  Feature-rich, security-aware switches will be used in IDFs connecting individual devices to building networks.  For the reasons cited above, Foundry equipment is recommended for all layers of the network (core, distribution and access).  Underscoring this recommendation is the fact that a number of peer institutions (University of California – Davis, Michigan State University, and University of Nebraska) and aspirant institutions (University of California – Berkeley, University of California – San Diego, University of Illinois – Urbana/Champaign, and University of Washington) have large-scale deployments of Foundry equipment.  In addition, Emory University, University of Miami, University of Alabama – Birmingham, Harvard University, Princeton University, UCLA, University of Southern California and University of California – San Francisco also have significant deployments of Foundry equipment within their network infrastructures.

 

Appendix A

Next-Generation Core and Distribution Networks

 




Appendix B

Functional Network Requirements

The following table constitutes a list of functional network requirements for each of the layers (C=core, D=distribution, A=access) and where noted, whether a layer-three (L3) switch is indicated:

 

Functional Requirement Description

Layers Required

IPv4 (RFC 1812 & RFC 2644)

C, D(L3)

IPv6

C, D(L3)

OSPFv2 (RFC 2328)

C, D(L3)

OSPF NSSA (RFC 1587)

C, D(L3)

IPX

C, D(L3)

IPX RIP

C, D(L3)

AppleTalk

C, D(L3)

RTMP

C, D(L3)

Differentiated Services  (RFC 2475)

C, D(L3)

Classification and marking of layer 4 traffic for timing-sensitive applications

C, D, A

VRRP per redundant, L2-connected building (RFC 3768)

C

Fully-redundant and fault-tolerant switch components

C

Sufficient backplane capacity for non-blocking frame forwarding

C, D, A

Wire-speed frame forwarding on all ports

C, D, A

MAC Bridging (IEEE 802.1D)

C, D, A

VLAN Tagging/Prioritization (IEEE 802.1p)

C, D, A

Virtual LANs (IEEE 802.1Q)

C, D, A

Multi-Link Trunking (IEEE 802.3ad)

C, D, A

100BaseTX/FX Ethernet (IEEE 802.3u)

C, D, A

Flow Control (IEEE 802.3x)

C, D, A

1000BaseSX/LX/ZX Ethernet (IEEE 802.3z)

C, D, A

Gigabit over Category 5 Wiring (IEEE 802.ab)

D, A

AAA (RFC 2989)

C, D, A

BGP4 (RFC 1771, RFC 1745, RFC 1965, RFC 1997, RFC 2385, RFC 2439, RFC 2798, RFC 2842)

C

BOOTP (RFC 951, RFC 1542)

C, D, A

BOOTP/DHCP Relay (RFC 2131)

C, D(L3)

DVMRP V3 (RFC 1075)

C, D(L3)

IEEE 802.1x Authentication

D, A

IGMP (RFC 1812)

C, D, A

IGMP V2 (RFC 2236)

C, D, A

IP Forward Table MIB (RFC 1354)

C, D(L3)

NTP (RFC 1305)

C, D(L3)

PIM-Dense Mode (RFC 2362)

C, D(L3)

PIM-Sparse Mode (RFC 2117)

C, D(L3)

Radius (RFC 2865)

C, D(L3)

RIP V1 (RFC 1058)

C, D(L3)

RIP V2 (RFC 2453)

C, D(L3)

RSVP (RFC 2205, RFC 2206, RFC 2207, RFC 2208, RFC 2209, RFC 2210)

C, D(L3)

RMONv1 (RFC 1757 - Groups 1,2,3,9)

C, D, A

SNMPv3 (RFC )

C, D, A

SNMP MIB II (RFC 1907)

C, D, A

SSH Version 2

C, D, A

Syslog Support

C, D, A

TFTP (RFC 783)

C, D, A

Gigabit port mirroring

C, D, A

Secondary IP addressing on routed interfaces

C

VLANs based on non-standard subnet masks

C, D, A

 

Appendix C

Functional Security Requirements

 

The following table constitutes a list of functional security requirements for each of the layers (C=core, D=distribution, A=access) and where noted, whether a layer-three (L3) switch is indicated:

 

Functional Requirement Description

Layers Required

Support central security policy enforcement capability, e.g., check security baseline:

  • Antivirus (AV) software version
  • endpoint security software version
  • OS patch level
  • registered MAC address
  • appropriate IP address
  • Inappropriate or unauthorized services

of any device connecting to the network and allow access only after baseline has been verified (or place the device in a quarantine zone, as needed, containing remediation software that can be installed on it)

C, D, A

Work with UGA's AV provider to verify AV policy compliance

D, A

Provide a solution to allow for planned firmware update management in communications devices

C, D, A

Support anomaly-based network intrusion protection system (IPS) sensors by simultaneously mirroring all ports on switch to IPS sensor port

C

Trigger security ACL changes based on IPS detection

C, D, A

Identify network attacks:

  • DDOS & DOS
  • Violations of RFCs

and stop unknown/unauthorized traffic

C, D, A

Event and audit logging to Security Information Management System (SIMS):

  • Communications device configuration modifications
  • Failed communications device authentication attempts
  • OSPF update tampering (only for OSPF participating devices)
  • Unauthorized access (ports & services)

C, D, A

Security Assertion Markup Language (SAML) information exchange between communications devices and SIMS:

  • Network thresholds
  • SNMP traps
  • Device overflows
  • Host configuration (services, password policy)
  • RFC violations

C, D, A