EMP BWEC

# SADC Software Package This repository contains all necessary software to operate an arbitrary number of SADCs on a single computer via SFP network interfaces. The fundamental setup is one PCIe network card with two SFP interfaces for one SADC. Consequently, to operate n SADCs, n network cards are needed.

## Requirements - Python3 - PyROOT

## System Specifications The following setup is running in Mainz and is capable to readout three SADCs simultaneously. ### SADC - 3 x SADC V 3.5 - Mainz Firmware: <https://gitlab.rlp.net/emp/sadc_v_3_5> - SFP-Transceiver: OPTOWAY SPM-8100WG

### Computer - Dell Precision 5820 Tower

1. Intel Xeon Processor W-2245 (8C 3.9GHz 4.7GHz Turbo HT 16.5MB, 155W DDR4-2933)
2. 64GB 4x16GB DDR4 2933 RDIMMECC
3. 2.5“ 256GB SATA Class 20 Solid State Drive
4. 3,5-Zoll-SATA-Festplattenlaufwerk, 2 TB, 7.200 1/min
5. NVIDIA Quadro P400, 2GB, 3 mDP to DP adapter (5820T)

- SFP Network Cards

1. 3 x StarTech.com: PCIe fiber network card - 2-port open SFP - 10G (PART # PEX20000SFPI)
2. SFP-Transceiver: Delock 86186 SFP-Transceiver-Modul 1 GBit/s 550 m Modultyp SX

- Operating System: Ubuntu 20.04 - Cable: LINDY 46341 Glasfaser LWL, Multimode OM4 2.00 m

## The “config” File The fundamental config file is located in sadc_software/network_configuration (<https://gitlab.rlp.net/emp/sadc_software/-/blob/master/network_configuration/config>) and is needed for the network configuration and as input for almost all software. Every network card, which serves one SADC has two interfaces (for example ens1f0 and ens1f1). To define one SADC setup, the config file needs two lines with the following information:

Note: Every interface needs its own subnetwork (check the IPs)!

### Example: Mainz config file for three SADCs: ``` #Interface IP_Interface IP_FPGA #Netmask #Default_Port #Shared_Port #FPGA_ID #Slot #SADC1 ens1f0 192.168.2.4 192.168.2.10 255.255.255.0 50010 50020 0 1 ens1f1 192.168.3.5 192.168.3.11 255.255.255.0 50011 50020 1 1 #SADC2 ens2f0 192.168.4.4 192.168.4.10 255.255.255.0 50010 50021 2 2 ens2f1 192.168.5.5 192.168.5.11 255.255.255.0 50011 50021 3 2 #SADC2 ens4f0 192.168.6.4 192.168.6.10 255.255.255.0 50010 50022 4 3 ens4f1 192.168.7.5 192.168.7.11 255.255.255.0 50011 50022 5 3 ``` ## Network Setup With the help of the config file, the network can be configured easily. However, first deactivate any network configuration programms/GUIs which are offered by nowadays Linux operating systems. The network configuration has to be done via /etc/network/interfaces.

### Generate the “interface” File By executing ``` you@your_pc:~/sadc_software/network_configuration$ python3 generate_interface_file.py

``` (<https://gitlab.rlp.net/emp/sadc_software/-/blob/master/network_configuration/generate_interface_file.py>) the interface configuration file is generated. Make sure, that the name of your standard interface is set correctly (copper_interface). For instance, the output for the Mainz setup is shown in the following: ``` #This file is autogenerated and has to be located in /etc/network/ source-directory /etc/network/interfaces.d

auto lo iface lo inet loopback

auto eno1 iface eno1 inet dhcp

iface ens1f0 inet static address 192.168.0.4 netmask 255.255.255.0 broadcast 192.168.0.255

iface ens1f1 inet static address 192.168.1.5 netmask 255.255.255.0 broadcast 192.168.1.255

iface ens2f0 inet static address 192.168.2.4 netmask 255.255.255.0 broadcast 192.168.2.255

iface ens2f1 inet static address 192.168.3.5 netmask 255.255.255.0 broadcast 192.168.3.255

iface ens4f0 inet static address 192.168.4.4 netmask 255.255.255.0 broadcast 192.168.4.255

iface ens4f1 inet static address 192.168.5.5 netmask 255.255.255.0 broadcast 192.168.5.255

``` After checking the output, move the file to /etc/network/.

### Bring the Interfaces Up Interfaces up: ``` you@your_pc:~/sadc_software/network_configuration$ python3 ifup.py

``` Interfaces down (if needed): ``` you@your_pc:~/sadc_software/network_configuration$ python3 ifdown.py

``` Check the configuration via:

``` ifconfig ``` ### Add IP Tables To merge the data streams from the different interfaces, it is necessary to link the default ports (see section The “config” File) to shared ports by executing ``` you@your_pc:~/sadc_software/network_configuration$ python3 add_iptable_rules.py

``` (<https://gitlab.rlp.net/emp/sadc_software/-/blob/master/network_configuration/add_iptable_rules.py>).

If needed, you can remove the tables via: ``` you@your_pc:~/sadc_software/network_configuration$ python3 remove_iptable_rules.py

``` (<https://gitlab.rlp.net/emp/sadc_software/-/blob/master/network_configuration/remove_iptable_rules.py>).

Check the current iptable setup on your system via: ``` sudo iptables -nL -v –line-numbers -t nat ``` After adding the iptables, the output should look like this: ``` Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes) num pkts bytes target prot opt in out source destination 1 0 0 REDIRECT udp – ens1f0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:50010 redir ports 50020 2 0 0 REDIRECT udp – ens1f1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:50011 redir ports 50020 3 0 0 REDIRECT udp – ens2f0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:50010 redir ports 50021 4 0 0 REDIRECT udp – ens2f1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:50011 redir ports 50021 5 0 0 REDIRECT udp – ens4f0 * 0.0.0.0/0 0.0.0.0/0 udp dpt:50010 redir ports 50022 6 0 0 REDIRECT udp – ens4f1 * 0.0.0.0/0 0.0.0.0/0 udp dpt:50011 redir ports 50022

… ```

``` Note: The iptables need to be added after every reboot of your computer! ``` ## Communication with the SADCs After the successful configuration of the network we can start to interact with the SADCs ### The Channel Set File

Since the input channel order and even the their polarity can differ from setup to setup, a so-called channel_set_file can be made and be located under: ``` you@your_pc:~/sadc_software/channel_set_files$ ``` The mapping has the aim to order the channels in HG, LG pairs:

For instance the first block in the example (0,1,2,3) defines crystal 0 with APD 0 (HG=0,LG=1) and APD 1 (HG=0,LG=1). Whereby the polarity of the different amplifications (HG,LG) is opposed (0,1).

``` Note: It is possible to not define a channel set file (=None). However functions, which address only HG or LG channels are point less though. ```

### Start the SADC Server Open the file:

``` you@your_pc:~/sadc_software/sadc_communication/sadc_control.py ``` (<https://gitlab.rlp.net/emp/sadc_software/-/blob/master/sadc_communication/sadc_control.py>)

If existing, you can choose your channel set file: ``` CHANNEL_CONFIG_FILE = '../channel_set_files/channel_set_proto_16_2.txt' ``` If you don't have a channel set file, write: ``` CHANNEL_CONFIG_FILE = None ``` Furthermore both the HOST and the PORT of the server are defined in the header as well: ``` IP_ADDRESS = '10.106.11.24' # could also be 'localhost' or '127.0.0.1' SADC_CTRL_PORT = 7000 ``` Afterwards execute: ``` you@your_pc:~/sadc_software/sadc_communication$ python3 sadc_control.py ```

After the the initialisation process, the terminal should output SADCX successfully initialised (X = 1,2,…) for every SADC, which is is plugged. For instance, the output for the Mainz setup ends with: ``` SADC1 successfully initialised SADC2 successfully initialised SADC3 successfully initialised MAIN> main line MAIN> module name: main MAIN> parent process: 5425 MAIN> process id: 29878 QUEUE_HANDLER> request_queue_handler QUEUE_HANDLER> module name: main QUEUE_HANDLER> parent process: 29878 QUEUE_HANDLER> process id: 29910 MAIN> socket open, listening on port 7000 ``` ### Troubleshooting at the Initialising Process Sometimes the initialisation process fails for one ore more SADCs. For instance: ``` Socket timeout! SADC1 successfully initialised SADC2 successfully initialised SADC3 initialisation failed! ``` Often, it helps to bring the interfaces down and an up again (Paragraph: Network Setup). If a single SADC has failed to initialise, you can try to bring its dedicated interfaces down and up. In the particular example I would try: ``` sudo ifdown ens4f0 sudo ifup ens4f0 ``` then ``` you@your_pc:~/sadc_software/sadc_communication$ python3 sadc_control.py ``` if this still does not work I would do the same with the other interface (ens4f1).

### Communicate with the SADC Server

#### Telnet A simple way to communicate with the SADC server is to use telnet: ``` telnet <HOST> <PORT> ``` whereby both HOST and PORT are defined in the header of the SADC server script: ``` you@your_pc:~/sadc_software/sadc_communication/sadc_control.py ``` (<https://gitlab.rlp.net/emp/sadc_software/-/blob/master/sadc_communication/sadc_control.py>)

For the Mainz setup, it would be as following: ``` telnet 10.106.11.24 7000 ``` There are three plus n (number of initialised SADCs) identifier commands: ``` HELP ALL WHO SADC1 SADC2 SADC3 ``` whereby HELP returns all possible commands you can send to the SADC server, WHO returns all initialised SADCS, SADCX (X=1,2,…) addresses individual SADCs and ALL addresses all SADCs at the same time. For instance, at the Mainz setup the command WHO gives: ``` WHO SADC1 SADC2 SADC3 ``` and the command HELP gives: ``` HELP ############################# POSSIBLE COMMANDS ############################# #SADC1 SADC1_SET_POLARITY CHANNEL [0..63] VALUE [0 = neg. or 1 = pos.] BROADCAST [0 = single channel or 1 = all channels] SADC1_SET_POLARITY_CONFIG_FILE NO ARGUMENT, Sets polarity according the configuration file if it exists SADC1_SET_THRESHOLD CHANNEL [0..63] VALUE [0..16383] BROADCAST [0 = single channel or 1 = all channels] SADC1_SET_THRESHOLD_HG VALUE [0..16383], This function makes only sense if a configuration file exists SADC1_SET_THRESHOLD_LG VALUE [0..16383], This function makes only sense if a configuration file exists SADC1_SET_TIME_UNDER_THRESHOLD CHANNEL [0..63] VALUE [0..16383] (0..30 reasonable) BROADCAST [0 = single channel or 1 = all channels] SADC1_SET_TIME_UNDER_THRESHOLD_HG VALUE [0..16383] (~20 reasonable), This function makes only sense if a configuration file exists SADC1_SET_TIME_UNDER_THRESHOLD_LG VALUE [0..16383] (~15 reasonable), This function makes only sense if a configuration file exists SADC1_SET_TRACE_LENGTH VALUE [0..729], Number of samples per readout channel SADC1_SET_NUMBER_OF_SAMPLES_BEFORE_TRIGGER VALUE [0..127], Number of samples before trigger SADC1_SET_NUMBER_OF_HITS_RATE VALUE [0..65535], Number of hits recorded for averageing SADC1_SET_SYNC_STATUS FPGA1 [0 = unsync. or 1 = sync] FPGA2 [0 = unsync. or 1 = sync], Synchronisation of sample sender with software trigger SADC1_GET_SAMPLES NO ARGUMENT, Sample request SADC1_GET_RATES NO_ARGUMENT, Rate request SADC1_TAKE_DATA PATH NUMBER_OF_HITS [arbitary number] SADC1_PUSH NO_ARGUMENT, SADC push mode (data frames are transmitted) SADC1_RESET NO_ARGUMENT, SADC reset SADC1_SET_TIMEOUT TIME [s], Sets the timeout in sec. for the data receiver (default: 1 s) #SADC2 SADC2_SET_POLARITY CHANNEL [0..63] VALUE [0 = neg. or 1 = pos.] BROADCAST [0 = single channel or 1 = all channels] SADC2_SET_POLARITY_CONFIG_FILE NO ARGUMENT, Sets polarity according the configuration file if it exists SADC2_SET_THRESHOLD CHANNEL [0..63] VALUE [0..16383] BROADCAST [0 = single channel or 1 = all channels] SADC2_SET_THRESHOLD_HG VALUE [0..16383], This function makes only sense if a configuration file exists SADC2_SET_THRESHOLD_LG VALUE [0..16383], This function makes only sense if a configuration file exists SADC2_SET_TIME_UNDER_THRESHOLD CHANNEL [0..63] VALUE [0..16383] (0..30 reasonable) BROADCAST [0 = single channel or 1 = all channels] SADC2_SET_TIME_UNDER_THRESHOLD_HG VALUE [0..16383] (~20 reasonable), This function makes only sense if a configuration file exists SADC2_SET_TIME_UNDER_THRESHOLD_LG VALUE [0..16383] (~15 reasonable), This function makes only sense if a configuration file exists SADC2_SET_TRACE_LENGTH VALUE [0..729], Number of samples per readout channel SADC2_SET_NUMBER_OF_SAMPLES_BEFORE_TRIGGER VALUE [0..127], Number of samples before trigger SADC2_SET_NUMBER_OF_HITS_RATE VALUE [0..65535], Number of hits recorded for averageing SADC2_SET_SYNC_STATUS FPGA1 [0 = unsync. or 1 = sync] FPGA2 [0 = unsync. or 1 = sync], Synchronisation of sample sender with software trigger SADC2_GET_SAMPLES NO ARGUMENT, Sample request SADC2_GET_RATES NO_ARGUMENT, Rate request SADC2_TAKE_DATA PATH NUMBER_OF_HITS [arbitary number] SADC2_PUSH NO_ARGUMENT, SADC push mode (data frames are transmitted) SADC2_RESET NO_ARGUMENT, SADC reset SADC2_SET_TIMEOUT TIME [s], Sets the timeout in sec. for the data receiver (default: 1 s) #SADC3 SADC3_SET_POLARITY CHANNEL [0..63] VALUE [0 = neg. or 1 = pos.] BROADCAST [0 = single channel or 1 = all channels] SADC3_SET_POLARITY_CONFIG_FILE NO ARGUMENT, Sets polarity according the configuration file if it exists SADC3_SET_THRESHOLD CHANNEL [0..63] VALUE [0..16383] BROADCAST [0 = single channel or 1 = all channels] SADC3_SET_THRESHOLD_HG VALUE [0..16383], This function makes only sense if a configuration file exists SADC3_SET_THRESHOLD_LG VALUE [0..16383], This function makes only sense if a configuration file exists SADC3_SET_TIME_UNDER_THRESHOLD CHANNEL [0..63] VALUE [0..16383] (0..30 reasonable) BROADCAST [0 = single channel or 1 = all channels] SADC3_SET_TIME_UNDER_THRESHOLD_HG VALUE [0..16383] (~20 reasonable), This function makes only sense if a configuration file exists SADC3_SET_TIME_UNDER_THRESHOLD_LG VALUE [0..16383] (~15 reasonable), This function makes only sense if a configuration file exists SADC3_SET_TRACE_LENGTH VALUE [0..729], Number of samples per readout channel SADC3_SET_NUMBER_OF_SAMPLES_BEFORE_TRIGGER VALUE [0..127], Number of samples before trigger SADC3_SET_NUMBER_OF_HITS_RATE VALUE [0..65535], Number of hits recorded for averageing SADC3_SET_SYNC_STATUS FPGA1 [0 = unsync. or 1 = sync] FPGA2 [0 = unsync. or 1 = sync], Synchronisation of sample sender with software trigger SADC3_GET_SAMPLES NO ARGUMENT, Sample request SADC3_GET_RATES NO_ARGUMENT, Rate request SADC3_TAKE_DATA PATH NUMBER_OF_HITS [arbitary number] SADC3_PUSH NO_ARGUMENT, SADC push mode (data frames are transmitted) SADC3_RESET NO_ARGUMENT, SADC reset SADC3_SET_TIMEOUT TIME [s], Sets the timeout in sec. for the data receiver (default: 1 s) #Broadcast Commands ALL_SET_POLARITY CHANNEL [0..63] VALUE [0 = neg. or 1 = pos.] BROADCAST [0 = single channel or 1 = all channels] ALL_SET_POLARITY_CONFIG_FILE NO ARGUMENT, Sets polarity according the configuration file if it exists ALL_SET_THRESHOLD CHANNEL [0..63] VALUE [0..16383] BROADCAST [0 = single channel or 1 = all channels] ALL_SET_THRESHOLD_HG VALUE [0..16383], This function makes only sense if a configuration file exists ALL_SET_THRESHOLD_LG VALUE [0..16383], This function makes only sense if a configuration file exists ALL_SET_TIME_UNDER_THRESHOLD CHANNEL [0..63] VALUE [0..16383] (0..30 reasonable) BROADCAST [0 = single channel or 1 = all channels] ALL_SET_TIME_UNDER_THRESHOLD_HG VALUE [0..16383] (~20 reasonable), This function makes only sense if a configuration file exists ALL_SET_TIME_UNDER_THRESHOLD_LG VALUE [0..16383] (~15 reasonable), This function makes only sense if a configuration file exists ALL_SET_TRACE_LENGTH VALUE [0..729], Number of samples per readout channel ALL_SET_NUMBER_OF_SAMPLES_BEFORE_TRIGGER VALUE [0..127], Number of samples before trigger ALL_SET_NUMBER_OF_HITS_RATE VALUE [0..65535], Number of hits recorded for averageing ALL_SET_SYNC_STATUS FPGA1 [0 = unsync. or 1 = sync] FPGA2 [0 = unsync. or 1 = sync], Synchronisation of sample sender with software trigger ALL_GET_SAMPLES NO ARGUMENT, Sample request ALL_GET_RATES NO_ARGUMENT, Rate request ALL_TAKE_DATA PATH NUMBER_OF_HITS [arbitary number] ALL_PUSH NO_ARGUMENT, SADC push mode (data frames are transmitted) ALL_RESET NO_ARGUMENT, SADC reset ALL_SET_TIMEOUT TIME [s], Sets the timeout in sec. for the data receiver (default: 1 s) WHO NO_ARGUMENT, Returns all accessible SADCs HELP NO_ARGUMENT, List all possible commands ``` ## Setup Skripts

Depending on your current setup, you can define a setup script. The setup script sets parameters such as thresholds, trace lengths, sync status and so on. The setup scripts are located in: ``` you@your_pc:~/sadc_software/setup_scripts/ ``` For instance the setup script for the hv board tests in Mainz (<https://gitlab.rlp.net/emp/sadc_software/-/blob/master/setup_scripts/hv_board_test.py>) reads: ``` import sys

# Communication with the SADC server sys.path.insert(1, '../sadc_communication') from sadc_client_interface import CLIENT_INTERFACE SADC_CONTROLE_HOST = '10.106.11.24' SADC_CONTROLE_PORT = 7000

ci = CLIENT_INTERFACE(SADC_CONTROLE_HOST,SADC_CONTROLE_PORT)

#SADC ID sadc_id = 3 # (at which slot (SADC) is the setup plugged in? (1,2 or 3)) bc = False # Broadcast=False if only one SADC is addressed.

#Thresholds

## APFEL high gain ci.set_threshold_hg(sadc_id,bc,3500) ci.set_time_under_threshold_hg(sadc_id,bc,20)

## APFEL low gain ci.set_threshold_lg(sadc_id,bc,300) ci.set_time_under_threshold_lg(sadc_id,bc,15)

#Number of Samples ci.set_trace_length(sadc_id,bc,729)

#Nuber of Samples before Trigger ci.set_number_of_samples_before_trigger(sadc_id,bc,127)

#Number of Hits for RA Avarage ci.set_number_of_hits_rate(sadc_id,bc,1000) # Number of hits for averaging

#Sync ci.set_sync_status(sadc_id,bc,1,1) # Sync status 0=unsync, 1=sync

```

# Applications

## The Client Interface To simplify the development of applications the so-called client interface can be used. The class description is located in: ``` you@your_pc:~/sadc_software/sadc_communication/sadc_client_interface.py ``` <https://gitlab.rlp.net/emp/sadc_software/-/blob/master/sadc_communication/sadc_client_interface.py>

You can implement it in any application via: ``` sys.path.insert(1, '../sadc_communication') from sadc_client_interface import CLIENT_INTERFACE

ci = CLIENT_INTERFACE(SADC_CONTROLE_HOST,SADC_CONTROLE_PORT) ```

## Example Applications

### Simple Sample Server

``` you@your_pc:~/sadc_software/apps/Simple_Sample_Server.py ``` <https://gitlab.rlp.net/emp/sadc_software/-/blob/master/apps/Simple_Sample_Server.py>

Execute: ``` bokeh serve Simple_Sample_Server.py –args SADC_CONTROLE_HOST [ip or hostname] SADC_CONTROLE_PORT [value] SADC_ID [1,2,3] ``` The standard address for the trace monitor is: ``` http://localhost:5006/Simple_Sample_Server ``` Hint: If you want to have trace monitors for different SADCs, just define other ports: ``` bokeh serve Simple_Sample_Server.py –port 5001 –args 10.106.11.24 7000 1 bokeh serve Simple_Sample_Server.py –port 5002 –args 10.106.11.24 7000 2 bokeh serve Simple_Sample_Server.py –port 5003 –args 10.106.11.24 7000 3 ``` The web applications can then be found under: ``` http://localhost:5001/Simple_Sample_Server #SADC1 http://localhost:5002/Simple_Sample_Server #SADC2 http://localhost:5003/Simple_Sample_Server #SADC3 ``` ### Simple Rate Server

``` you@your_pc:~/sadc_software/apps/Simple_Rate_Server.py ``` <https://gitlab.rlp.net/emp/sadc_software/-/blob/master/apps/Simple_Rate_Server.py>

Execute: ``` python3 Simple_Rate_Server.py SADC_CONTROLE_HOST [ip or hostname] SADC_CONTROLE_PORT [value] SADC_ID [1,2,3] CUST_PORT ``` The standard address for the rate monitor is (CUST_PORT = empty): ``` http://localhost:8050 ``` Hint: If you want to have rate monitors for different SADCs, just define other ports: ``` python3 Simple_Rate_Server.py 10.106.11.24 7000 1 8051 python3 Simple_Rate_Server.py 10.106.11.24 7000 2 8052 python3 Simple_Rate_Server.py 10.106.11.24 7000 3 8053 ``` The web applications can then be found under: ``` http://localhost:8051 #SADC1 http://localhost:8052 #SADC2 http://localhost:8053 #SADC3 ```

### Take Data Example

``` you@your_pc:~/sadc_software/apps/Take_Data_Example.py ``` <https://gitlab.rlp.net/emp/sadc_software/-/blob/master/apps/Take_Data_Example.py>

The example is self-descriptive. However I want to point out an important feature. It is only possible to run one data taking process at a time. A second data taking request will be rejected by the SADC server. It is possible to ask the SADC server, if data taking is possible (status = IDLE) or if it is locked (status = LOCKED).

``` status = ci.get_daq_status(sadc_id,bc) # Ask for DAQ status (IDLE: ready for data taking, LOCKED: data taking is still running)

if(status=='IDLE'):

  print(ci.take_data(sadc_id,bc,path,number_of_hits))
  

```

### Remarks

In order to communicate with/readout several SADCs at the same time you can always use the “ALL” identifier, which translates at the client interface with broadcast (bc) = True.

For instance: ``` ci = CLIENT_INTERFACE(SADC_CONTROLE_HOST,SADC_CONTROLE_PORT)

ci.soft_trigger_sample(SACD_ID,False) # Only SADC_ID is addressed

ci.soft_trigger_sample(SACD_ID,True) # All SADCs are addressed

```

Note: The return of the individual SADCs is summarised in a list of all returns: No broadcast → RETURN, broadcast → [RETURN, RETURN, RETURN]