A key feature of the MC3810 Multiservice Access Concentrator is its ability to support voice and video streams in addition to the traditional data streams typically supported by IOS router platforms. These new information streams are real-time streams and originate from synchronous devices. As such, special attention must be paid to the clocking architecture of networks that carry voice and video services. The MC3810 provides several unique features that allow network designers to properly distribute and control the clocking architecture of their multiservice networks. This paper provides an overview of the specific and unique clocking implementations on the MC3810.

Why Synchronous Clocking Is Important.

Voice and video streams are unique in that they are real-time and continuous. Typically, a voice or video source generates information at a fixed rate, which needs to be quickly and continuously transported to the receiver, which accepts the information at a fixed fate. Voice and video devices tend to be duplex systems, where the rate at which a device generates data, is the same as the device’s ability to receive information. If two devices generate information at different rates, there will be loss of information as one side over-runs and the other under-runs. It is essential in these architectures that a single clock source be utilized throughout the network, to minimize the loss of information. Use of a single clock source makes the network a synchronous system.

It is important to understand that the use of synchronous Layer 1 media types (e.g. V.35 or DS1) does not guarantee that the overall system will be synchronous. For example two digital voice devices may be interconnected with a synchronous V.35 link, such that information is exchanged between the two devices with a common synchronous clock. However, if the two voice devices do not generate the information (e.g. digitized voice) with a common clock source, then the overall system is still not synchronous, and data loss may result. The diagram illustrates a simple case where data loss occurs in a non-synchronous back-to-back voice system. The same concept can be extrapolated to a system that crosses a public frame-relay network, or a system that is using circuit emulation to transport video stream through an ATM network.

The important concept to grasp is that voice and video information must be generated and received at the same rate on opposite ends of a network. The concept also applies even if voice is compressed. Using the previous diagram, consider if the voice codecs were performing an 8:1 compression of the voice. If a synchronous system was not employed, Device A might generate compressed voice information at the rate of 64.1/8 Kbps, while Device B generates compressed voice at the rate of 63.9 Kbps.

It should be noted that clocking mismatches can manifest in several ways. A simple case is diagramed above, where data loss occurs as voice is being D-A or A-D digitized. However, in practice on the MC3810, this case is not always catastrophic. The voice coders employed on the MC3810 expect to have to deal with cases where data loss has occurred due to network perturbations such as delay and jitter. The coders will interpolate for missing compressed data, and the resulting voice waveform is often not noticeable degraded. In this special case, the clocking mismatch results in a conflict at the voice codec, which is able to intelligently exploit the predictable nature of voice, to mitigate (but not correct) the conflict.

However, in the case of a Circuit Emulation connection, there is no "magic" interpolation that can be performed to deal with a data over-run or under-run situation. In the CES case, data will simply be corrupted when a non-synchronous system is designed. Typically, when data corruption occurs due to over-runs or under-runs, the video devices attached to the MC3810 CES ports will lose frame synchronization and enter a frame-search mode. This results in even more data loss, which is readily noticeable and often catastrophic.

Clocking mismatches can also result in a conflict at the Layer 1 level. This is especially true if an MC3810 with two MFTs is placed at the border of two separately clocked T1 or E1 networks, and is forced to attempt to resolve the clock difference between the networks. DS1 Clock and Frame slips can occur, which can result in lengthy re-frame times, and can cause an attached DS1 device to declare the line down.

Persons familiar with router installations are sometimes surprised when they first install an MC3810 with two MFTs and are immediately faced with Layer 1 problems. After all, the MC3810 is not the first IOS router platform with two DS1 interfaces, and dual DS1 installations with other routers do not seem to need as much careful planning. The difference is that most other IOS router platforms do not yet have to deal with an overall synchronous network. It is worth noting that platforms that implement CES, such as the LS1010 with 7200, also face the same clocking challenges, and provide similar methods to recover and distribute a synchronous clock through the network.

Inattention to the source of system clock can also lead to more subtle problems that may manifest themselves in ways other than clock slips on the T1/E1 controllers. Various portions of the system depend on a stable clock source to perform properly, and can fail to perform if driven with an unstable clock source. For example, the fax-relay mode of the voice DSPs must demodulate the V.29 fax signal in order to decode the digital fax image data. If there is significant clock jitter between the system clock that drives the DSPs and the fundamental clock used to modulate the fax signal, the DSPs may fail to train on the V.29 signal. This will result in fax failures – even though no clock slips are evidenced on the T1/E1 controllers.

How to Design a Synchronous MC3810 Network

One can ensure a synchronous system by employing a master clock somewhere within the network and distributing and recovering that clock throughout the network. In this manner the end devices at opposite ends of the network are referenced to a common source. Multiple clock sources can be used in a synchronous network, if they are of sufficient accuracy where one can be confident that the end devices will have the same reference. The MC3810 offers the following features that can be used to design a synchronous network:

1. Synchronous clock recovery from Controller T1 0, or Controller E1 0, and distribution to all ports (i.e remaining controller, and both UIO ports)
2. Synchronous clock recovery from Controller T1 1, or Controller E1 1, and distribution to all ports.
3. Synchronous clock recovery from UIO A and distribution to all ports.
4. Clock generation from an internal Stratum 4 oscillator and distribution to all ports.
5. All MFT types can provide internal or line clocking methods when in T1 or E1 mode,
6. All MFT types can provide loop clocking methods when in T1 mode.
7. Specially ordered E1 MFTs can provide loop clocking methods when in E1 mode.
8. A hierarchy of potential clock sources may be defined, in order to provide a backup clock source in the event of failure of the primary clock source.

Clock Distribution Design of MC3810

The MC3810 achieves its synchronous clocking capabilities through the efficient use of components and circuitry already required for basic functions. The diagram below describes the clock flow through the MC3810. For simplicity, data flow is not shown on the diagram.

A central feature of the MC3810 clock system is a 2Mhz Phase-Lock-Loop circuit that is used to drive most of the interfaces and components of the system. By synchronizing interfaces to this central clock, the MC3810 can distribute a common network clock source to each of its external interfaces. Hence this PLL circuit is called the ‘network-clock’ source.

Another key feature is a clock divider that can step the 2Mhz network-clock down to either 56Khz or 64Khz. This 56/64Khz clock can then drive individual clock multipliers associated with each UIO, which is the mechanism by which the UIOs are synchronized to the network-clock source. The UIOs may also be driven off a Baud Rate Generator that is a standard feature of the Serial Communication Controller of the CPU. The BRG is the typical clock source for most IOS router platforms, but has the drawback that the BRG is not guaranteed to be accurately synchronized to any particular clock source.

Choosing the Source of Network-Clock

The network-clock PLL may be driven from one of two sources. The first source is a common circuit that delivers recovered clock from either of the T1/E1 controllers. The second network-clock source is a circuit that delivers recovered clock from Serial 0. Recovered clock cannot be derived from Serial 1.

Note that even though there are only two physical choices to drive the network-clock PLL, the system can nonetheless be configured to source the network-clock from one of four sources: controller T1/E1 0, controller T1/E1 1, Serial 0, or a free-running internal source. The remainder of this section describes how that is possible.

The system defaults to switching the network-clock PLL to be driven by the common circuit from the T1/E1 controllers. It is important to recognize that it is through the configuration of the clock recovery mode of the T1/E1 controllers that the majority of the functionality of the network-clock PLL is determined.

Use Internal Clock from System Control Board. If both MFTs have been configured to not recover clock by entering clock source internal under the controller configuration level, then no input signal is provided to the network-clock PLL. As a result, the network-clock PLL free runs. This results in a system clock that is generated entirely internally. In this mode, the internal PLL is accurate to a Stratum 4 level.

Use Clock from Network Device Attached to Controller T1/E1 0. If controller T1/E1 0 is set to recover clock from the attached network device by entering clock source line under the controller configuration level, then the clock recovery circuit on controller T1/E1 0 will place a recovered 2Mhz clock on the common circuit toward the network-clock PLL. The controller’s clock recovery circuit generates a 2 MHz clock whether the controller is configured for T1 or E1. Once the network-clock PLL circuit is receiving a valid 2Mhz clock from an MFT, it synchronizes to the recovered clock and redistributes it to the rest of the system.

Use Clock from Network Device Attached to Controller T1/E1 1. Likewise, if controller T1/E1 1 is set to recover clock from the attached network device by entering clock source line under the controller configuration level, then the clock recovery circuit on controller T1/E1 1 will place a recovered 2 MHz clock on the common circuit toward the network-clock PLL. Again, this recovered clock will eventually be redistributed to the rest of the system

Use Clock from Network Device Attached to Serial 0. If Serial 0 is configured as a DTE, it can accept clock from the attached DTE, and use that to drive the network-clock PLL. Because the input to the network-clock PLL must be 2Mhz, a clock multiplier circuit is used to multiply the incoming clock on Serial 0 to 2Mhz, in increments of 8Khz. This multiplier is configured by entering clock rate line <rate> at the Serial 0 interface level, where <rate> is the rate of the incoming clock, and must be a multiple of 8Khz. This command is only accepted when Serial 0 is configured as a DTE.

Additionally, the network-clock PLL must be switched to use the multiplied 2Mhz clock from Serial 0, through the network-clock-select 1 serial 0 global configuration command. (The network-clock-select command is described in more detail later).

Redistributing Network-Clock to External Interfaces

Once the network-clock PLL has been synchronized to a particular clock source, the synchronized clock may be redistributed to both T1/E1 controllers, and both UIO serial interfaces (Serial 0 and Serial 1). This is the primary method by which an overall synchronous network is implemented, in that the MC3810 is able to extend a synchronous master clock source from one portion of the network to another.

Redistributing Network-clock to a Network Device Attached to T1/E1 Controllers. Either T1/E1 controller can be set to use the network-clock as the Layer 1 framer’s reference clock, by entering clock source internal at the controller configuration level. In this mode, the transmitted T1/E1 signal is driven with the network-clock, which is also used to attempt to frame-sync the received T1/E1 signal. The network device should be configured to derive its clock from the T1/E1 signal transmitted by the MC3810. If the T1/E1 signal received from the attached network device is not synchronous with the MC3810 network-clock, then frame and clock slips will result at the T1/E1 controller. Data is lost when frame or clock slips occur.

Redistributing Network-clock to a Network Device Attached to Serial Interfaces. When a DCE cable is attached to either UIO serial port, the serial port can be configured to generate a clock that is synchronously derived from the network-clock. The 2 MHz network clock from the network-clock PLL must first be divided down to a 56Khz or 64Khz base rate. This is configured using the network-clock base-rate [ 56 | 64] global configuration command. The system defaults to a 56Khz base-rate.

Once the network-clock has been divided down to a 56Khz or 64Khz base-rate, it may then be multiplied in integer steps for output at the serial ports. This is configured using the clock rate network <rate> command at the serial interface configuration level. The desired rate must be an integer multiple of the current network-clock base-rate, or will be rejected by the CLI.

There is only one base-rate clock divider, so it is not possible to configure one serial port for a Nx56Khz synchronous clock rate, and the other for a Nx64Khz synchronous clock rate. It is however, possible to configure one port for a network-clock rate while configuring the other port with a clock rate derived from the Baud Rate Generator.

Understanding clocking modes of the MultiFlex controller

The Multiflex controller has three clocking options, internal, line, and loop, which are configured with the clock source <internal | line | loop> command under the controller configuration section.

Clock Source Internal Mode. As explained previously, when an MC3810 Multiflex controller is configured for internal clocking, it chooses the internal 2MHz network-clock as its reference. It uses this clock to transmit the T1/E1 signal. It also uses this clock to receive and frame sync the T1/E1 signal. If the received T1/E1 signal is not based upon this same clock, then the T1/E1 framer will periodically lose synchronization on the incoming T1/E1 signal. This results in clock slips and frame errors, which can be disruptive to data and voice flows. When in internal mode, the MFT does not attempt to recover and redistribute a clock from the received T1/E1 signal to the 2MHz Network-clock PLL.

In this mode, the T1/E1 device attached to the MFT should be configured to derive its clock from the MC3810 (or from a common source).

Clock Source Line Mode. As explained previously, when an MC3810 Multiflex controller is configure for line clocking, it recovers a 2Mhz clock from the incoming T1/E1 signal, and chooses that as its reference. It uses this recovered clock to transmit the T1/E1 signal, as well as to receive and frame sync the T1/E1 signal, so that the transmitted T1/E1 signal will be clocked at the same rate as the received T1/E1 signal. No clock slips or frame errors should occur. In this mode, the recovered 2Mhz clock is also passed to the 2Mhz network-clock PLL for redistribution to the rest of the MC3810 system.

In this mode, the T1/E1 device attached to the MFT should be configured to derive its clock from some other source, and supply it to the MC3810.

Clock Source Loop Mode. Loop timing mode is similar to Line timing mode, except that the MC3810 MFT does not pass on the recovered 2Mhz clock to the network-clock PLL. This allows an MFT to be timed independently by the received signal, while the rest of the MC3810 system can be timed from a different source (such as by the T1/E1 signal received on another MFT). Because the MFT is using the same clock as the incoming T1/E1 signal, no clock slips or frame-errors will occur.

In this mode, the T1/E1 device attached to the MFT may be configured to derive its clock from some other source, and supply it to the MC3810.

All MC3810 T1 MFTs have the ability to be loop timed. However, due to a limitation in the Siemens framer chip employed on the MFT, not all E1 configured MFTs have the ability to be loop timed. Cisco has redesigned the Siemens framer chip for Siemens and is waiting for Siemens to begin delivery. Software can determine whether the E1 MFT can support loop timing or not.

Data Flow Relationship with Clock Source Mode: It is important to understand the relationship that MFT clocking mode has with the method by which data is passed between the MFT and the rest of the MC3810 system. Data is exchanged between the MFT and the system through a PCM bus. This bus is always timed by the network-clock PLL. When the MFT reference clock and the system clock are the same, then data exchange between the MFT and the system occurs synchronously.

When an MFT is set for clock source internal, it uses the network-clock as its reference so data exchange is synchronous.

When an MFT is set for clock source line, it drives the network-clock from the recovered T1/E1 clock, so again data exchange between the MFT and the system is synchronous.

When an MFT is set for clock source loop, it uses the recovered T1/E1 clock, but does not drive the network-clock. As a result, the network-clock and the MFT clock are different and data exchange between the MFT and the system is asynchronous. Data is clocked in and out of a frame buffer by both the MFT and the system using independent clocks. At some point, buffer overflow or underflow can occur and data loss will result. However, the data loss occurs internally, and does not affect the framing condition on the outgoing T1/E1 signal. Hence, the data loss is controlled and minimized, without having to incur a long re-synchronization period. This is known as a controlled slip.

Configuring a Hierarchy of Clocking Sources

In addition to providing great flexibility in configuring the overall system clocking architecture, the MC3810 allows the definition of a hierarchy of clocking sources. A hierarchical clocking configuration allows the MC3810 to switch to another clock source, when the current clock source fails. Up to four clock sources can be identified in a prioritized chain. When a higher priority clock source fails, the MC3810 can switch to the next prioritized source. Likewise, when a higher priority source is restored, the MC3810 can return to the higher priority source.

Commands to configure this mode include:

Essentially, these commands determine which source the MC3810 will use to drive the network-clock 2MHz PLL, and the wait period before it switches to another source. The hierarchy is defined by entering up to four separate network-clock-select commands with different priorities.

In order to switch away from a clock source, the module or signal from which the clock source originates must fail. To maintain economical positioning, the MC3810 does not have a clock qualification circuit. If a clock is being received but it is jittered or non-nominal, the MC3810 will not disqualify the clock signal and switch to another clock source. However, if the module providing the clock experiences a failure (e.g the T1/E1 controller experiences Loss-of-Signal or Loss-of-Frame), then the clock source will be switched. Note also that shutting down a controller will not cause the clock source to be switched. Shutting down a controller causes AIS to be transmitted on the controller and suppresses any interrupts from the controller, thereby ceasing data flow. However the controller continues to frame-sync the incoming signal and recover clock. To manually switch the network-clock-source to the next source in the chain, the chain could be redefined. This behavior is consistent with network-clock-select functions on other Cisco devices (such as the LS1010). It is also consistent with the backup interface function – whereby manually shutting down a primary interface does not cause a backup interface to come out of standby state.

By default the MC3810 does not use a hierarchical clock chain. It defaults to setting the network-clock 2MHz PLL to be driven by which ever MFT is configured for clock source line. This allows the most common clocking configurations to be effected simply by defining the appropriate clock mode on the controllers. In other words, it is not necessary to use the network-clock-select commands to cause the MC3810 to derive clock from Controller T1 1. If additional control is required, or if a non-nominal source is required (such as Serial0), then the network-clock-select command may be used to override the default configuration.

Some rules to remember when dealing with the network-clock-select commands:

0. In the absence of a network-clock-select command, system clock is derived from whichever controller is set for clock source line.
1. Only one controller should actually be set for clock source line at any one time (note: there is a difference between set and configured).
2. If a controller is to be the object of a network-clock-select command, the controller must be configured for line timing.
3. Multiple controllers can be objects of network-clock-select commands.
4. Because rule 3 could cause a conflict between rule 2 and rule 1, any controller that is the object of a network-clock-select command, and is not the current source of system clock, will be auto-temporarily set to loop-timing by software (the line-timing configuration remains unchanged).
5. Hence, any controller that is the object of a network-clock-select command must be physically capable of being loop-timed. System error-messages will result if a network-clock-select command is applied to a controller which cannot be set to loop-timing, or if network-clock-select command is applied to a controller which is configured for internal timing.
6. Current shipping TEBs cannot be set to loop-timing when configured for E1 (due to a design limitation of the Siemens chip).

In summary, any potential network-clock source must be pre-configured into a mode that allows the MC3810 to derive clock from that source. However, these pre-configurations cannot all be active at the same time, otherwise a clocking conflict could occur. So, when a hierarchy of clock sources are defined with the network-clock-select commands, software will preserve the configurations, but will temporarily set the sources into a mode that allows harmoniously co-existence.

For illustration consider the following configuration:

The table below indicates how each potential clock source would be pre-configured in order to be the source of network-clock, and how each potential clock source would be temporarily set when it is an active or standby clock source. Boldface indicates the temporary changes made automatically by software.

Because the desired configuration defines that both controller T1 0 and controller T1 1 are potential clock sources, they are both pre-configured to line timing.

When the software chooses controller T1 0 to be the active clock source, it temporarily sets controller T1 1 to loop timing, and it throws the Network-Clock PLL switch in the direction of the controllers. Since controller T1 1 does not provide recovered clock while in loop timing mode, the system clock is driven from controller T1 0.

When the software chooses controller T1 1 to be the active clock source, it temporarily sets controller T1 0 to loop timing, and throws the Network-Clock PLL switch in the direction of the controllers. Since controller T1 0 does not provide recovered clock while in loop timing mode, the system clock is driven from controller T1 1.

When the software chooses Serial 0 to be the active clock source, it temporarily sets both controller T1 0 and T1 1 to loop timing, and throws the Network-Clock PLL switch in the direction of serial port. The system clock is then driven by a clock recovered from the DTE serial 0 port, which has been multiplied from 64000Hz to 2MHs.

When the software chooses the internal system clock to be the active clock source, it temporarily sets both controller T1 0 and T1 1 to loop timing, and throws the Network-Clock PLL switch in the direction of the controllers. Since the controllers are both set for loop timing, neither provides a recovered clock to drive the PLL - resulting in a free-running, or internally timed, system clock.