Preamble

The Shanghai High Repetition Rate XFEL and Extreme Light Facility (SHINE) is currently under construction as one of the world's most advanced hard X-ray free-electron laser facilities. The timing system, as an essential part of the free-electron laser facility, provides precise timing of trigger pulse signals for a range of devices to ensure that particles are generated and accelerated to the designed energy while enabling the precise measurement of beam parameters. To precisely distribute and synchronize the 1.003086 MHz (1300/1296) timing signals over a distance of approximately 3.1 km based on White Rabbit technology, three technical routes have been proposed. This paper begins with a description of the design and development process of the timing system for the SHINE project, which culminates with the determination of the design scheme. During the installation and commissioning of the timing system, the jitter accuracy of the timing signal was tested and found to be less than 10 ps, which meets the requirements of the project. Furthermore, the precise clock synchronization signal provided by the timing system supported the joint debugging of various related systems and realization of beam acquisition.

Keywords: SHINE, Timing System, White Rabbit

INTRODUCTION

The Shanghai High Repetition Rate XFEL and Extreme Light Facility (SHINE), as the first hard X-ray free electron laser facility in China, is located in Zhangjiang High-Tech Park, Shanghai, with a burial depth of 29 m and a length of 3.11 km \cite{1,2,3}. The facility will comprise a superconducting linear accelerator with an electron energy of 8 GeV, three undulator lines, three optical beam lines, a 100 PW ultra-intense ultra-short laser system, and the first 10 experimental stations \cite{4,5}. The photon energy range provided by SHINE will cover 0.4–25 keV, with excellent characteristics such as ultrahigh peak and average brightness \cite{6}, high repetition rate (1.003086 MHz), and femtosecond ultrafast pulses. Furthermore, SHINE will exhibit an ultrahigh spatial resolution capability at the nanometer-level and an ultrafast time-resolution capability at the femtosecond-level.

The timing system is a crucial component of the control system in a free-electron laser facility. It acts as a metronome, providing precise clock pulse signals for lasers, modulators, low-level radio frequency (LLRF) devices, beam diagnostics, optical measurements, and other devices that require synchronized triggering. It ensures that an electron bunch is generated and accelerated to the required energy to produce a free-electron laser that meets the requirements while simultaneously conducting beam and optical parameter measurements. In addition, the timing system provides hardware bunch numbers for beam diagnostics and optical measurements, which serve as the fundamental basis for beam parameter analysis and failure diagnosis. Two dominant schemes exist for timing systems in large accelerator facilities: the event timing system and the White Rabbit (WR)-based timing system. After comprehensive consideration, SHINE adopted a timing system based on WR technology.

WR technology \cite{7} is a new distributed synchronous timing technology that integrates Synchronous Ethernet, Precision Time Protocol version 2 (IEEE1588v2), and Dual Mixer Time Difference (DMTD) measurement \cite{8} to achieve sub-nanosecond clock synchronization over a range of tens of kilometers and more than two thousand nodes. WR technology was initially developed as an open-source project by the European Organization for Nuclear Research (CERN) in collaboration with the Helmholtz Association of the German Research Centre (GSI) in Germany \cite{9}. It is characterized by its high reliability and flexible topology and has been adopted by a number of large accelerator facilities in China and around the world, including ESRF, FAIR, HIAF, and LHC \cite{10,11,12,13}. In addition, WR technology has been successfully applied in distributed detector array projects such as LHAASO, KM3NeT, and SKA \cite{14,15,16}.

II. DESIGN PROPOSAL

2.1. Technical Routes

SHINE requires to precisely distribute and synchronize 1.003086 MHz (1300/1296 MHz) timing signals over a long distance of approximately 3.1 km with a jitter accuracy of less than 30 ps. To meet these requirements based on WR technology, three different technological routes have been proposed: random triggering transmission, radio frequency (RF) signal transmission, and non-standard clock transmission.

2.1.1. Random Triggering Transmission

Random trigger distribution, also called White Rabbit trigger distribution (WRTD), is a generic framework for distributing triggers between nodes over a WR network \cite{17}. In WRTD, an incoming trigger pulse is time-stamped. The master node receives these trigger pulses and distributes them to the slave nodes through the WR network in the form of a message containing the timestamps of the rising edges and metadata identifying the source of the original trigger pulses. The slave node receives a message and filters it to output a specific signal with a fixed delay. The Principle of the WRTD system is illustrated in Fig. 1(a).

The WRTD system is currently operational at both CERN \cite{18} and the Shanghai soft X-ray Free-Electron Laser User Facility (SXFEL-UF). In the case of the SXFEL-UF, the system comprises a master node and three slave nodes connected to each other through a network consisting of WR switches and optical fibers. The master node collects the timestamps of the rising edges of the trigger inputs through the time-to-digital converter (TDC) block, which serves as the time base for all channels in the system. The signals can be output directly from the slave node with a configurable delay time, or alternatively fan-out to various devices through a fan-out module. The jitter between the output of the slave node and the external reference signal is less than 120 ps for both laboratory and field tests \cite{19,20}. Thus, the theoretical feasibility of applying the WRTD system to SHINE has been established. However, the ultrahigh repetition rate of 1.0030864 MHz is constrained by the current 1 Gbps WR network bandwidth, which necessitates the deployment of 10 Gbps WR switches. Furthermore, the increased jitter resulting from this frequency does not satisfy the requirement of less than 30 ps jitter accuracy.

2.1.2. RF Signal Transmission

For RF signal transmission, as shown in Fig. 1(b), the phase and frequency information of the RF input is locked and recorded by a phase detector, proportional integral (PI) control, and direct digital synthesis (DDS) block \cite{21,22}. This information is corrected by feedback and encoded into data packets that are digitally transmitted to the slave node through the WR network. The slave node utilizes the received data to decode the corresponding frequency and phase information, thereby enabling recovery of the RF output by the DDS. Because all the nodes have the same reference frequency and time from the WR network, the DDS in the slave node can accurately reproduce the same clock output as the master node's RF clock \cite{17}. The efficacy of RF signal transmission in SHINE remains to be validated, and further studies are required to ascertain the efficacy of signal recovery and jitter accuracy control during pulse editing. The development of a prototype verified this technological route \cite{23}, with the jitter accuracy of the timing signal output being less than 20 ps. Compared with the nonstandard clock transmission proposed below, RF signal transmission requires additional hardware, such as a phase detector, PI controller, and DDS block, to convert analog signals to digital signals, which results in a more complex and costly system architecture. Consequently, this route was not selected for the SHINE project.

2.1.3. Non-standard Clock Transmission

In WR standard clock transmission, the master node utilizes the local clock to encode the data stream, and the slave nodes obtain a recovery clock of the same frequency through the clock and data recovery (CDR) circuit. To ensure compatibility with Gigabit Ethernet, WR devices are limited to operating at either 62.5 or 125 MHz. Meanwhile, a phase locked loop (PLL) circuit was built into the WR device to generate a phase-adjustable local clock with the same frequency using the recovery clock as a reference. Compared with the WR network's 125/62.5 MHz operating clock and 10 MHz reference signal in standard clock transmission, SHINE requires a repetition rate of 1.0030864 MHz. The 1.3 GHz RF synchronization signal must be divided by a special factor (1300/1296) to a frequency of 1.0030864 MHz or its integer multiple frequency to be used as a reference signal for non-standard clock transmission. Correspondingly, the operating frequency in a standard WR network must be adjusted accordingly to ensure synchronization of the entire WR network, for instance, when considering the operating frequency conversion of WR devices from 125/62.5 MHz to 125.385/62.693 MHz (a multiple of 1.003086 MHz).

The timing system based on the WR technique employing nonstandard clock transmission to achieve clock synchronization currently lack reference cases. Consequently, the jitter accuracy of timing systems implemented via nonstandard clock transmission remains uncertain and requires further development and validation. In a standard WR network, the internal voltage-controlled crystal oscillator (VCXO) has a center frequency of 25 MHz. The initial solution was to replace the VCXO of the switches and nodes with a customized VCXO (27.083 MHz) to convert the synchronous reference frequency to 67.708 MHz (1.003086 MHz135/2). However, the frequency coefficient (135/2) is not conducive to the implementation of functions such as phase calculations and pulse delays. Meanwhile, to be compatible with the standard WR framework and reduce the modification of the standard WR device, the operating frequency was modified to 64.1975 MHz (1.003086 MHz64). This operating frequency can be obtained from the original standard VCXO (25 MHz*208/81), which also offers the advantage of facilitating the generation of 2N frequency division.

Considering the alteration in the operational frequency, it is necessary to redefine the original timescale structure. For example, when the operating frequency is 64.1975 MHz, the clock counting will increment the 'seconds' count by one for every approximately 64,197,531 clock cycles, which is different from the clock cycles at the standard frequency (62.5 MHz). As time increases, the discrepancy accumulates, and the time reference within the WR devices gradually deviates. For this reason, pseudoseconds (∼0.9969 s, 1/1.003086 MHz) and standard seconds with a clear proportionality relationship were designed, allowing the phase relationship of the clock output to be determined. Through a prototype development verification, the jitter accuracy obtained from the test was less than 10 ps, representing an improvement over the previous two technical routes \cite{24}. Therefore, the SHINE timing system was designed based on non-standard clock transmission.

2.2. Bunch ID

For the majority of data acquisition equipment, the timing system must provide electron bunch number (Bunch ID) information, which is used to tag the same electron bunch and its resulting laser pulse measurements for diagnosis and analysis. The bunch ID is the unique identifier for each bunch. Because the generation time of each bunch has a one-to-one correspondence with the synchronized time scale, it can be considered as a clock count of 1.003086 MHz. The bunch ID is recorded at the instant the triggered device detects the rising edge of the trigger pulse. The bunch ID is incremented sequentially when the device is operated at 1.003086 MHz. Conversely, when the device operates at a reduced frequency, the bunch ID is increased by a multiple of the reduced frequency.

III. SYSTEM ARCHITECTURE

The SHINE timing system is illustrated in Fig. 2 [FIGURE:2], which shows the star topology of the master and slave nodes combined with a WR transmission network. It comprises one master node, approximately 750 slave nodes, and approximately 100 WR switches distributed across the three layers. The slave nodes are further classified into FANOUT and embedded FMC nodes.

3.1. Basic Function

The timing system must perform two crucial functions: beam-synchronous and random-event trigger signal distribution. Beam-synchronous trigger signal distribution is the basic function of a timing system that provides synchronized clock pulse signals (triggers) for the various accelerator devices. In this function, the slave node outputs a clock pulse signal that is phase locked to the grandmaster reference signal from a femtosecond synchronization system \cite{25}. All slave nodes are required to output simultaneously, and the skew between the pulse signals output by different slave nodes is quantified as the jitter. The random-event trigger signal distribution is utilized to transmit various event signals, including beam loss and machine snapshots. This is a significant expansion of accelerator timing systems. In a random-event trigger signal distribution, the WR network enables all slave nodes to output simultaneously with a brief delay once the master node receives a periodic or single pulse signal.

The SHINE timing system has specific requirements for the performance indices of its node outputs. In the beam-synchronous trigger signal distribution function, the jitter between the output signal of the slave node and the external reference signal must be less than 30 ps root mean square (RMS), and the phase change after rebooting is less than 120 ps peak-to-peak. The leakage trigger rate is less than 1/10^6. In the random trigger function, the jitter between the output signal from the node and the external reference signal must be less than 100 ps RMS.

3.2. Hardware

3.2.1. Embedded FMC Slave Node

The embedded FMC slave node is illustrated in Fig. 3(a) [FIGURE:3], and it primarily comprises the following core circuits: an XILINX Artix 7 series field-programmable gate array (FPGA), a flash chip for storing FPGA firmware, a WR clock circuit, a D-flip-flop chip, and a delay chip required for the clock fan-out frame. In addition, the external interfaces of an embedded FMC slave node include two small form pluggable (SFP) module interfaces, one standard FPGA Mezzanine Card low pin count (FMC LPC) interface \cite{26}, and two SMA signal output interfaces. Fig. 3(b) illustrates the internal framework and the external interfaces of the embedded FMC slave node. The specific functions of the different interfaces are as follows:

1. SFP0 interface: an optical module communication interface adapted to the frequency of SHINE, which is connected to the superior WR switch for clock synchronization and data transmission.
2. SFP1 interface: a standard Gigabit network communication interface through which user data from the FMC interface can be accessed.
3. SMA interface: outputs a beam-synchronous trigger signal with an adjustable fine delay for the corresponding channel.
4. FMC-LPC Interface: the interface that complies with the FMC-VITA specification represents the primary interface between the node and the underlying hardware, as well as the node's power supply interface. In addition to the two fine delay adjustable beam-synchronous clocks and the bi-directional data communication interface, the FMC also provides differential signals with only coarse delay regulation directly from the FPGA pins, which can be employed to generate both beam-synchronous and random-event trigger signals. Furthermore, a high-performance reference clock is provided to drive the D-flip-flops on the carrier board. The time information and the bunch ID are transmitted using a custom serial encoding.

The embedded FMC slave nodes provide comprehensive interfaces for timing and communication. In practice, different carrier boards can be utilized in conjunction with the embedded FMC slave nodes to combine the FANOUT slave and master nodes as required.

3.2.2. FANOUT Slave Node

The core function of the FANOUT slave node is to complement the pulse output framework of multiple beam-synchronous and random-event trigger signals. The low-noise clock fan-out chip receives the high-performance reference clock provided by the embedded FMC slave node and inputs it into the D-flip-flop clock terminals, which are used for signal de-jittering and noise reduction. The FPGA is responsible for decoding the information from the control bus of the delay chip and then using it to control the dual-channel delay chips. The multichannel fan-out carrier board enables the user to adjust the fine delay between multiple beam-synchronous and random event trigger signals.

3.2.3. Master Node

The core of the master node's carrier board is an XILINX SPARTAN 6 series FPGA that implements a 4-channel TDC \cite{27,28} for measuring external event pulses. The carrier board acquires the high-performance reference clock from the FMC interface and the synchronized time information from the embedded slave nodes, which is used as the time reference for the TDC \cite{29,30}. The timing-pulse acquisition logic packages the acquired event pulse information with a fixed delay according to the Etherbone protocol and broadcasts it to the WR network through the SFP interface of the embedded FMC board assembled on the master node. The master node serves as the global parameter-configuration management unit for the SHINE timing system. It provides real-time output-enabled and bunch-ID configurations for all the timing nodes.

3.3. Software

The Simple Network Management Protocol (SNMP) \cite{31}, which is a standard protocol for Internet Protocol (IP) network management, specifies a standardized management framework for monitoring and managing devices in a network environment, public language for communication, and related security and access control mechanisms. It is primarily used to monitor the status information of devices in the network and to support the writing of device parameters. As an extension to standard Ethernet, the WR timing network is compatible with the SNMP protocol for monitoring and management of the network and provides device-level support to EPICS for SNMP communication via the specific EPICS SNMP module. In the SHINE timing system, SNMP is mainly used to monitor the operation and synchronization status of switches and nodes, such as the current hardware and firmware version information, temperature of the device, lock status of the White Rabbit phase-locked loop (WR-PLL), operation status of the White Rabbit precision time protocol (WR-PTP), round-trip link latency between the current device and the parent device, and calibrated fixed latency parameter. It covers all the parameters required for debugging. Simultaneously, parameters such as the fixed delay and pulse width of the slave node can also be modified via SNMP.

A WR timing monitoring system was developed using the devSNMP, which is an EPICS SNMP device support module \cite{32}. devSNMP provides SNMP device-layer support for EPICS and accesses SNMP-based devices using the NET-SNMP software tool. The NET-SNMP software suite comprises a set of SNMP utilities and a comprehensive SNMP development library that enables the management and monitoring of controlled devices through a range of SNMP utilities, including snmpgets, snmpsets, and snmpwalks. By integrating the module into the EPICS development environment \cite{33}, EPICS soft IOCs can retrieve and modify the EPICS process variables (PVs) in the EPICS database for the management of WR devices via SNMP. To meet the demand for remote control of various devices in the SHINE timing system, we further developed a control interface using the Python Display Manager (PyDM) software. Users can read the current operating status of the system from the interface, enable or disable the corresponding channel, and adjust the delay and frequency of the channel according to the physical requirements.

3.4. Calibration

To ensure the synchronization of the timing system, it is essential to calibrate the fixed delay deviations between WR devices, including WR switches and nodes. The Precision Time Protocol version 2 (IEEE 1588v2) proposes a periodic delay request–response mechanism for measuring clock skews between master and slave devices. This mechanism defines four types of messages for time synchronization: SYNC, FOLLOW UP, DELAY REQ, and DELAY RESP. The master node periodically sends a SYNC message, records the sending time as t1, and sends this t1 timestamp to the slave node via a FOLLOWUP message. The slave node receives the SYNC message and records the reception time as t2 and receives the FOLLOWUP message to obtain the t1 timestamp. The slave node sends the DELAY REQ message and records the local sending time t3. The master node receives the DELAY REQ message, records the reception moment t4, and sends it to the slave node via the DELAY RESP message. The slave node receives the DELAY RESP message, obtains the t4 timestamp, and calculates the transmission delay and clock skew between the master and slave nodes by using t1, t2, t3, and t4.

This protocol assumes that the message transmission delay from master to slave (Delay_{MS}) is equal to that from slave to master (Delay_{SM}). The slave collects the four timestamps and calculates the round-trip latency (Delay_{MM}). The one-way transmission latency between the devices is given by the following equation:

$$Delay_{MS} = Delay_{SM} = Delay_{MM} = \frac{(t4 - t1) - (t3 - t2)}{2}$$

In accordance with Eq. (1), the clock skew (Offset_{MS}) between the master and slave nodes is calculated as:

$$Offset_{MS} = t2 - t1 - Delay_{MS}$$

In the WR link model, a dedicated calibration process \cite{34} is employed to calibrate the following parameters: fixed delays for data sending and receiving between the master and slave devices (ΔT_{XM}, Δ_{RXM}, ΔT_{XS}, Δ_{RXS}), total transmission delay of the fiber optic (δ, Sum of δ_{MS} and δ_{SM}), logical delay for data alignment during data reception on the master and slave devices when serial data is converted to parallel data (ε_M and ε_S), and asymmetry coefficient of the fiber-optic link connecting the master and slave (α). Based on these parameters, the round-trip latency (Delay_{MM}) is defined as the sum of the above delay factors.

$$Delay_{MM} = ΔT_{XM} + Δ_{RXM} + ε_M + ΔT_{XS} + Δ_{RXS} + ε_S + δ_{MS} + δ_{SM}$$

IV. PERFORMANCE TEST

Performance testing of the SHINE timing system was completed for the technical indices related to both the beam-synchronous and random-event trigger signal distribution functions. To assist in the construction of the test platform and the performance test of the timing system, several essential test equipment must be employed. The list of performance test instruments includes a MT6000-XPRO GPS rubidium clock, a DG645 digital delay/pulse generator, an AFG31252 arbitrary function generator, a ZFRSC-42-S+ power splitter, a Keysight 53230A universal frequency counter/timer, an Agilent E5052B signal source analyzer, and a 6 GHz Keysight MSOS604A high-definition oscilloscope.

4.1. Beam-synchronous Trigger Signal Distribution

4.1.1. Test Scheme

A block diagram of the performance test for the beam-synchronous trigger signal distribution is shown in Fig. 4(a) [FIGURE:4]. At the beginning of the test, the GPS/rubidium clock provides a stable 10 [TABLE:10] MHz external reference signal for the arbitrary function generator. The two output channels of the arbitrary function generator are synchronized to produce a phase-locked periodic square-wave signal (1.003086 MHz*16). The output signal from CH1 is divided by the DG645 to approximately 1 Hz and then fed to the first-layer WR switch (GrandMaster Mode) as the WR network pulse-per-second (PPS) reference signal. The output signal from CH2 is divided into two signals by a power splitter. One channel is input to the first-layer WR switch as a reference clock signal, whereas the other channel is input to the oscilloscope as a test reference signal. In addition, the master and slave nodes must be connected to the WR switch through an optical fiber and then form a complete WR network through the switch. To be able to read or set the parameters of each node remotely, the host computer needs to be connected to each master and slave node through a piece of network converter capable of converting a standard Ethernet network to a WR network.

Because of the single-fiber wavelength division multiplexing (WDM) transmission method, the physical lengths of the WR upstream and downstream fiber-optic links are the same. However, there are differences in the propagation speeds of the different wavelengths, resulting in differences in the upstream and downstream link delays. The optical fiber asymmetry coefficient (α) expresses the ratio of the propagation speeds of the upstream and downstream wavelengths in an optical fiber of a particular structure. This coefficient can be calibrated in the laboratory. Using upstream and downstream delay scaling on the fiber, the WR slave can accurately determine the downstream transmission delay on the fiber. Combined with the fixed delay in the downlink and the data alignment delay, the total delay from the master to the slave (Delay_{MS}) can be determined as follows:

$$Delay_{MS} = \frac{α + 1}{α + 2}(δ_{MS} + δ_{SM}) + ΔT_{XM} + Δ_{RXS} + ε_S$$

In the actual laboratory calibration process, it is first necessary to select the master and slave devices required for calibration and connect the two devices through an optical fiber. After waiting for them to be synchronized and phase-locked to each other, an oscilloscope is used to measure the skew of the clock signal outputs of the two devices, which is recorded as Delay_{MS}. The slave device calculates the clock skew from the master by substituting the value of Delay_{MS} into Eq. (2) and then compensates for it by adjusting the phase of the local time count and local clock to achieve precise time synchronization with the master device and complete the device calibration.

4.1.2. Test Result

The jitter accuracy is a crucial metric for evaluating the performance of a timing system. One of the most important parameters used to measure the jitter accuracy of a signal is the time interval error (TIE), which is expressed as the error of the clock with respect to the reference clock during the measurement interval \cite{35}. In the actual testing process, the output of the beam-synchronous trigger signal from one of the slave nodes in the WR network was selected, and the skew value, also known as the TIE, between the output and external reference signal was measured using a 6 GHz bandwidth oscilloscope. The standard deviation (Std Dev) of these TIE values was then identified as a result of the jitter, and the test required more than 10^4 sampling times.

For the beam-synchronous trigger signal distribution, the jitter between the output signal of the slave node and the external reference signal was less than 10 ps, as illustrated in Fig. 5(a) [FIGURE:5]. The orange signal in Fig. 5(a) shows the external reference signal (1.003086 MHz*16) generated using an arbitrary function generator. The other signals are beam-synchronous trigger signals output from two FANOUT slave nodes. The jitter between the external reference signal and the beam-synchronous trigger signal obtained by the oscilloscope was 9.417 ps. The jitter between the two beam-synchronous trigger signals was 3.894 ps. Furthermore, as shown in Fig. 5(b), the phase noise of the operational clock from the WR timing device was evaluated using an Agilent E5052B signal source analyzer. The test results indicated that the cumulative jitter of the device was less than 2 ps within the interval of 10 Hz to 10 MHz. The peak-to-peak phase change after rebooting was less than 75 ps peak-to-peak, whereas the adjustable step size of the delay was less than 50 ps.

Moreover, a highly accurate timing system is sensitive to temperature changes \cite{36}. Thus, the output delay of the signal is affected by temperature fluctuations. The test results for the effect of the operating temperature on the time skew between the output signals are shown in Fig. 5(c) [FIGURE:5]. In the test, as the heating cabinet was heated from 25°C to 50°C, the time skew between the clock pulse outputs from the node and reference signal outside the heating cabinet was 26 ps.

In the missed-trigger test, the number of trigger pulses per second was quantified using a frequency counter. By increasing the measurement interval of the frequency counter to 10 s, the accumulated historical data of 1000 measurements are presented in Fig. 5(d) [FIGURE:5]. These results demonstrate that the missed trigger rate reaches the requirement of less than 1/10^6. For the mask-mode test, the slave node can output a pulse signal at a specific frequency by modifying the repetition rate of the trigger signal, valid/invalid bunch cluster, and arbitrary editable unit. As shown in Fig. 6 [FIGURE:6], a unique beam-synchronous trigger signal output with a mask of 10110010 is generated by configuring the aforementioned parameters.

4.2. Random-event Trigger Signal Distribution

4.2.1. Test Scheme

A block diagram of the performance test for the random event trigger signal distribution is shown in Fig. 4(b) [FIGURE:4]. Another arbitrary function generator is used to generate two phase-locked periodic square-wave signals with a frequency of 1 KHz. The output from CH1 is fed to the WR master node as input for the random-event trigger. The output from CH2 is input to the oscilloscope as an external reference signal.

4.2.2. Test Result

Using an oscilloscope to measure the jitter between the output of the slave node and the reference signal, this test also required more than 10^4 sampling times, and the results showed that the jitter was less than 20 ps.

INSTALLATION AND COMMISSIONING

The SHINE facility is in a complex underground environment. To achieve high-precision synchronous timing signal distribution in such a special installation environment, the timing system must comprise a variety of timing devices at different locations in the tunnel and working shafts, according to the number and location of timing signals required by other systems. These devices must then be connected through optical fibers to form a WR network. The installation of the timing system was successfully completed, and commissioning of the system in the SHINE injector is currently underway. The system consists of one master node, nine WR switches, five FANOUT slave nodes, and several embedded FMC slave nodes.

As illustrated in Fig. 7(a) [FIGURE:7], when an oscilloscope was employed to assess the 1.003086 MHz beam-synchronous trigger signal outputs of the slave nodes from the SHINE timing system that was currently operational onsite, the oscilloscope indicated that the jitter between the timing signal outputs of the two slave nodes was 4.4492 ps. The control interface for one slave node in the timing system is shown in Fig. 7(b) [FIGURE:7]. In the joint debugging process with the beam instrumentation system, the FMC slave node embedded in the electronic equipment can not only provide high-precision synchronous clocks to the SBPM, CBPM, and other equipment but also output the bunch ID corresponding to the rising edge of the triggering pulse of the electron bunch. The analog simulation of the messages obtained from the embedded FMC slave node through the electronic equipment enabled the beam-synchronous trigger signal, random-event trigger signal, and bunch ID corresponding to the current trigger input to be read. The implementation employs 48-bit wide data to store the bunch IDs, thereby ensuring that the bunch ID will not be repeated for a period of approximately 9 years. In addition, the bunch ID is linked to the beam data and can be accessed through the EPICS PV.

The installation and operation of the timing system in the SHINE electronic gun section assisted in obtaining the first photoelectron beam and the first 1.003086 MHz 100 pC beam. Over a period of approximately 90 min, the timing system increased the frequency of the beam-synchronous trigger signal fed to the laser and beam instrumentation systems from ∼10 Hz to ∼1 MHz. At this point, the attainment of the electron bunch was observed by means of the beam position monitor (BPM).

VI. CONCLUSION

In response to the 1.003086 MHz beam-synchronous clock distribution demanded by SHINE, based on WR technology, the timing system realized the nonstandard clock transmission technical route. As an important part of the accelerator control system, the design and development of the timing system has been completed. In terms of the indicators, the jitter between the output of the clock signal from the node and external reference signal is less than 10 ps, and the jitter between the output of the clock signal from different slave nodes is less than 5 ps, which satisfies the requirements. Furthermore, the timing system provides real-time bunch ID, which establishes a data-alignment basis for the beam parameter analysis and failure diagnosis. The timing system has been installed and is being currently commissioned in the injector section.

VII. ACKNOWLEDGEMENT

We would like to thank our colleagues at Tsinghua University, University of Science and Technology of China, and CERN for their discussions and collaboration.

REFERENCES

[1] Z.T. Zhao, C. Feng, K.Q. Zhang, Two-stage EEHG for coherent hard X-ray generation based on a superconducting linac. Nucl. Sci. Tech. 28, 117 (2017). doi:10.1007/s41365-017-0258-z
[2] C. Feng, H.X. Deng, Z, Review of fully coherent free-electron lasers. Nucl. Sci. Tech. 29, 160 (2018). doi:10.1007/s41365-
[3] N.S. Huang, H.X. Deng, B. Liu et al., Features and futures of X-ray free-electron lasers. The Innovation. 2(2), 100097 (2021). doi:10.1016/j.xinn.2021.100097
[4] Z.T. Zhao, D. Wang, L. Yin et al., SCLF: An 8-GeV CW SCRF Linac-Based X-Ray FEL Facility in Shanghai, in Proceedings of the FEL2017, Santa Fe, NM, USA (2017). doi:10.18429/JACoW-FEL2017-MOP055
[5] S. Chen, H.X. Deng, X.X. Fu et al., Design of the Beam Distribution System of SHINE, in Proceedings of the IPAC2021, Campinas, SP, Brazil(2021). doi:10.18429/JACoW-IPAC2021-TUPAB081
[6] N.S. Huang, Z.P. Liu, B.J. Deng et al., The MING proposal at SHINE: Megahertz cavity enhanced X-ray generation. Nucl. Sci. Tech. 34, 6 (2023). doi:10.1007/s41365-022-01151-6
[7] P.P.M. Jansweijer, H.Z. Peek, E. de Wolf, White Rabbit: Sub-nanosecond timing over Ethernet. NUCL INSTRUM METH A. 725, 187-190 (2013). doi:10.1016/j.nima.2012.12.096
[8] P. Moreira, P. Alvarez, J. Serrano et al., Digital dual mixer time difference for sub-nanosecond time synchronization in Ethernet. In Proceedings of IEEE IFCS2010, Newport Beach, CA, USA (2010). doi:10.1109/FREQ.2010.5556289
[9] M. Lipiński, T. Włostowski, J. Serrano et al., White rabbit: a PTP application for robust sub-nanosecond synchronization. In Proceedings of ISPCS2011, Munich, Germany (2011). doi:10.1109/ISPCS.2011.6070148
[10] G. Goujon, A. Broquet, N. Janvier, Refurbishment of the ESRF Accelerator Synchronization System Using White Rabbit. In Proceedings of ICALEPCS2017, Barcelona, Spain (2017). doi:10.18429/JACoW-ICALEPCS2017-TUCPL01
[11] M. Kreider, R. Bär, D. Beck et al., Two Years of FAIR General Machine Timing - Experiences and Improvements. In Proceedings of ICALEPCS2017, Barcelona, Spain (2017). doi:10.18429/JACoW-ICALEPCS2017-TUPHA092
[12] Y.Q. Liang, X.J. Wang, J.Q. Li et al., IEEE 1588-Based Timing and Triggering Prototype for Distributed Power Supplies in HIAF. IEEE T INSTRUM MEAS. 71, 1-9 (2022). doi:10.1109/TIM.2022.3194857
[13] P. Loschmidt, G. Gaderer, N. Simanic et al., White rabbit - sensor/actuator protocol for the CERN LHC particle accelerator. In Proceedings of IEEE SENSORS2009, Christchurch, New Zealand (2009). doi:10.1109/ICSENS.2009.5398529
[14] C. Li, S.B. Liu, L.F. Shang et al., The clock system for LHAASO WCDA based on reduced White Rabbit. Nucl. Sci. Tech. 23, 4 (2012). doi:10.13538/j.1001-8042/nst.23.230-236
[15] D. Calvo, D. Real, F. Carrió, Sub-nanosecond synchronization node for high-energy astrophysics: The KM3NeT White Rabbit Node. NUCL INSTRUM METH A. 958, 162777 (2020). doi:10.1016/j.nima.2019.162777
[16] M. Jiménez-López, F. Torres-González, J.L. Gutiérrez-Rivas et al., A Fully Programmable White-Rabbit Node for the SKA Telescope PPS Distribution System. IEEE T INSTRUM MEAS. 68,632-641 (2019). doi:10.1109/TIM.2018.2851658
[17] T. Włostowski, J. Serrano, G. Daniluk et al., Trigger and RF Distribution Using White Rabbit. In Proceedings of ICALEPCS2015, Melbourne, Australia (2015). doi:10.18429/JACoW-ICALEPCS2015-WEC3O01
[18] D. Lampridis, T. Gingold, M. Malczak et al., The Distributed Oscilloscope: A Large-Scale Fully Synchronised Data Acquisition System Over White Rabbit. In Proceedings of ICALEPCS2019, New York, NY, USA (2019). doi:10.18429/JACoW-ICALEPCS2019-TUBPR01
[19] C.L. Yu, H. Zhao, S.M. Hu et al., Design of timing system based on high accurate pulse generator. Nucl. Sci. Tech. 42, 40102-040102 (2019). doi:10.11889/j.0253-3219.2019.hjs.42.040102(in Chinese)
[20] P.X. Yu, Y.B. Yan, Timing System of Shanghai Soft X-ray Free Electron Laser User Facility Beamline and Experimental station. Nuclear Electronics and Detection Technology. 43, 300 (2023) (in Chinese)
[21] D. N. Bochkarev, I. V. Ryabov, I. V. Strelnikov et al., Direct Digital Synthesizers of Frequency and Phase-Modulated Signals. In Proceedings of SYNCHROINFO2019, Russia, (2019). doi:10.1109/SYNCHROINFO.2019.8814244
[22] P. Moreira, J. Serrano, P. Alvarez et al., Distributed DDS in a White Rabbit Network: An IEEE 1588 application. In Proceedings of ISPCS2012, San Francisco, CA, USA (2012). doi:10.1109/ISPCS.2012.6336611
[23] P.X. Yu, Y.B. Yan, G.H. Gong et al., Prototype of White Rabbit Based Beam-Synchronous Timing System for SHINE. In Proceedings of ICALEPCS2021, Shanghai, China (2021). doi:10.18429/JACoW-ICALEPCS2021-THBR03
[24] Y.B. Yan, P.X. Yu, G.H. Chen et al., White Rabbit Based Beam-Synchronous Timing Systems for SHINE. In Proceedings of IPAC2022, Bangkok, Thailand (2022). doi:10.18429/JACoW-IPAC2022-THIYGD1
[25] J.G. Wang, B. Liu, B.W. Wu, Sub-20 fs Synchronization Between Mode-Locked Laser and Radio Frequency Signal. In Proceedings of IBIC2023, Saskatoon, Canada (2023). doi:10.18429/JACoW-IBIC2023-TU2C03
[26] W. Pan, Q. Du, G. Gong, Development of a White Rabbit interface for synchronous data acquisition and timing control. In Proceedings of IEEE RT2012, Berkeley, CA, USA (2012). doi:10.1109/RTC.2012.6418218
[27] J. Song, Q. An, S.B Liu, A high-resolution time-to-digital converter implemented in field-programmable-gate-arrays. IEEE Trans Nucl Sci. 53, 1 (2006). doi:10.1109/TNS.2006.869820
[28] Y. Wang, J. Kuang, C. Liu et al., A 3.9-ps RMS Precision Time-to-Digital Converter Using Ones-Counter Encoding Scheme in a Kintex-7 FPGA. IEEE Trans Nucl Sci. 64, 10 (2017). doi:10.1109/TNS.2017.2746626
[29] P. Dudek, S. Szczepanski, J. V. Hatfield, A high-resolution CMOS time-to-digital converter utilizing a Vernier delay line. IEEE J. Solid-State Circuits. 35, 2 (2000). doi:10.1109/4.823449
[30] X. Y. Chen, L. Tang, X. Shen, A High-Resolution Two-Step Time-to-Digital Conversion in 40 nm CMOS. In Proceedings of ICICM2021, Nanjing, China (2021). doi:10.1109/ICICM54364.2021.9660337
[31] J.D. Case, M. Fedor, M.L. Schoffstall et al., Simple Network Management Protocol (SNMP). RFC Editor RFC1157. (1990). doi:10.17487/RFC1157
[32] M.J. Park, H. Jang, S. Lee et al., EPICS Integration of Simple Network Management Protocol for RISP. In Proceedings of IPAC2016, Busan, Korea (2016). doi:10.18429/JACoW-IPAC2016-THPOY010
[33] Q.R. Mi, G.H. Chen, H.F. Miao et al., Application of Open Source Virtual Platform in Large Accelerator Control System. Nuclear Electronics and Detection Technology. 3, 1-7 (2024) doi:10.20173/j.cnki.ned.20240424.002(in Chinese)
[34] H. Peek, P. Jansweijer, White Rabbit Absolute Calibration. In Proceedings of ISPCS2018, Geneva, Switzerland (2018). doi:10.1109/ISPCS.2018.8543067
[35] S. Bregni, Clock stability characterization and measurement in telecommunications. IEEE T INSTRUM MEAS. 46, 6 (1997). doi:10.1109/19.668274
[36] H. Li, G. Gong, W. Pan et al., Temperature Effect on White Rabbit Timing Link. IEEE Trans Nucl Sci. 26, 3 (2015). doi:10.1109/TNS.2015.2425659