@@ -64,16 +64,17 @@ Generally speaking, the color codes are:
...
@@ -64,16 +64,17 @@ Generally speaking, the color codes are:
- Red: Something is wrong,
- Red: Something is wrong,
- Grey: The service is disabled.
- Grey: The service is disabled.
In the __FAQ__ you can learn how to start __DAQC__. So let's assume you have started it up. __DAQC__ provides a pretty web frontend on port `8080`. This means that you can open a browser on any computer on the CERN network (including __Bumblebee__ ) and type `polarbear:8080`.
In the __FAQ__ you can learn how to start __DAQC__. So let's assume you have started it up. __DAQC__ provides a pretty web frontend on port `8080`. This means that you can open a browser on any computer on the network (including __Bumblebee__ ) and type `bumblebee:8080`.
Notice, that even though you can access it from anywhere on the CERN network, you need the credentials to be able to change anything.
Notice, that even though you can access it from anywhere on the network, you need the credentials to be able to change anything.
The web frontend will show you something like the following:
The web frontend will show you something like the following:
1. __VME computers*__ (__RIO2__, __RIO7__): This box shows you whether the VME computers are online, i.e. if they can be reached from __Bumblebee__.
2. __Readout*__: This box shows the status of the readout, i.e. the event builder on __Bumblebee__ and the two `drasi` instances running on the two VME-computers. The `2/2` means that both nodes are running. The event builder broadcasts the data such that we can attach onto it and save it to a file.
1. __VME computer*__ (__MOT2__): This box shows you whether the VME computer is online, i.e. if they can be reached from __Bumblebee__.
2. __Readout*__: This box shows the status of the readout, i.e. the event builder on __Bumblebee__ and the two `drasi` instances running on the VME-computer. The `1/1` means that the vme node is running. The event builder broadcasts the data such that we can attach onto it and save it to a file.
3. __Relay__: The relay is a `ucesb` instance that attaches to the event builder's data stream and fans it out to several data streams. This also validates the data, i.e. if it crashes, the data may be corrupt. The relay provides data to, among others, the online analysis. So if the relay dies, so does the online analysis. The relay is *NOT* responsible for the data taking or recording, and is thus non-essential.
3. __Relay__: The relay is a `ucesb` instance that attaches to the event builder's data stream and fans it out to several data streams. This also validates the data, i.e. if it crashes, the data may be corrupt. The relay provides data to, among others, the online analysis. So if the relay dies, so does the online analysis. The relay is *NOT* responsible for the data taking or recording, and is thus non-essential.
4. __Go4__: The online analysis. We run the online analysis as a server. That means that we do not NEED a GUI. Of course, to see anything, we need a GUI. However, by running the analysis as a server, we can attach multiple GUI instances. Moreover, if the GUI crashes, the online analysis process do not necessarily crash. In this way, we can also access Go4's web version in a browser. Open up a browser and type `polarbear:5000`, and you will see the online analysis (if it is running, of course).
4. __Go4__: The online analysis. We run the online analysis as a server. That means that we do not NEED a GUI. Of course, to see anything, we need a GUI. However, by running the analysis as a server, we can attach multiple GUI instances. Moreover, if the GUI crashes, the online analysis process do not necessarily crash. In this way, we can also access Go4's web version in a browser. Open up a browser and type `bumblebee:5000`, and you will see the online analysis (if it is running, of course).
Notice that __DAQC__ only know about the server, and not about any of the GUI instances. So if they die, simply reopen them.
Notice that __DAQC__ only know about the server, and not about any of the GUI instances. So if they die, simply reopen them.
5. __Sync__: We can automatically synchronize the recorded data to remote hosts, for instance a server at a local institute. You can see in the __FAQ__ how to add a new destination.
5. __Sync__: We can automatically synchronize the recorded data to remote hosts, for instance a server at a local institute. You can see in the __FAQ__ how to add a new destination.
6. __Mesytec__: We have a bunch of Mesytec amplifiers and their settings are usually quite important for the experiment. Before the experiment starts, one have usually settled on some set of parameters. These parameters can be saved using the Mesytec Control. In order to be sure that these settings haven't changed during the experiment (power failure, manual tinkering, etc..) __DAQC__ can compare the current settings of the modules with some saved reference settings (typically the ones you would also load). If it turns red, it means that there are inconsistencies between these two sets of settings. Refer to the __FAQ__ section to see how you can manage this feature.
6. __Mesytec__: We have a bunch of Mesytec amplifiers and their settings are usually quite important for the experiment. Before the experiment starts, one have usually settled on some set of parameters. These parameters can be saved using the Mesytec Control. In order to be sure that these settings haven't changed during the experiment (power failure, manual tinkering, etc..) __DAQC__ can compare the current settings of the modules with some saved reference settings (typically the ones you would also load). If it turns red, it means that there are inconsistencies between these two sets of settings. Refer to the __FAQ__ section to see how you can manage this feature.
...
@@ -83,22 +84,23 @@ The web frontend will show you something like the following:
...
@@ -83,22 +84,23 @@ The web frontend will show you something like the following:
__Note I__: When you press start, the file taking starts immediately - so take your time time to fill out the run sheet. When you press Stop the file taking also stops immediately - so take your time to fill out the run sheet.
__Note I__: When you press start, the file taking starts immediately - so take your time time to fill out the run sheet. When you press Stop the file taking also stops immediately - so take your time to fill out the run sheet.
__Note II__: The data acquisition continues to run after a file is closed. Only file recording is stopped by pressing "Stop".
__Note II__: The data acquisition continues to run after a file is closed. Only file recording is stopped by pressing "Stop".
__Note III__: When you are done filling out the run sheet, please make sure that the blue __START__ button is visible again (behind the run sheet window). This will usually take about 1-2 seconds.
When taking file, instead of saving one big fat file for each run, we chop it up into smaller pieces. The size of these pieces can be chosen in the right bottom corner. This results in a data structure like this
When taking file, instead of saving one big fat file for each run, we chop it up into smaller pieces. The size of these pieces can be chosen in the right bottom corner. This results in a data structure like this
To see the online analysis, you can open the dedicated GUI program GO4 on __Bumblebee__, by just typing `go4` in a terminal. If you want to watch the online analysis from a remote computer on the CERN network, you can open a browser and type `polarbear:5000`. This will open a web version of the online analysis.
To see the online analysis, you can open the dedicated GUI program GO4 on __Bumblebee__, by just typing `go4` in a terminal. If you want to watch the online analysis from a remote computer on the CERN network, you can open a browser and type `bumblebee:5000`. This will open a web version of the online analysis.
To see the different histograms, double-click `localhost:5000` in the left side, and then double-click the `Histograms` tab.
To see the different histograms, double-click `localhost:5000` in the left side, and then double-click the `Histograms` tab.
...
@@ -106,25 +108,23 @@ Make sure that the three green arrows in the top left (second toolbar) is presse
...
@@ -106,25 +108,23 @@ Make sure that the three green arrows in the top left (second toolbar) is presse
If nothing happens, follow the flowchart in the troubleshooting section.
If nothing happens, follow the flowchart in the troubleshooting section.
Go4 in accessible only inside the CERN network. That means anyone inside the CERN network can access the Go4 and do start/stop monitoring as well as clearing the histograms.
Go4 in accessible only inside the network. That means anyone inside the network can access the Go4 and do start/stop monitoring as well as clearing the histograms.
### Editing the online analysis
### Editing the online analysis
The online analysis is a custom made integration of the GSI project [Go4](https://www.gsi.de/en/work/research/experiment_electronics/data_processing/data_analysis/the_go4_home_page.htm?C=0), [ausalib](https://git.kern.phys.au.dk/ausa/ausalib/wikis/home) and [ucesb](http://fy.chalmers.se/~f96hajo/ucesb/). We call the project [go4cesb](https://git.kern.phys.au.dk/ausa/go4cesb/wikis/home).
The online analysis is a custom made integration of the GSI project [Go4](https://www.gsi.de/en/work/research/experiment_electronics/data_processing/data_analysis/the_go4_home_page.htm?C=0), [ausalib](https://git.kern.phys.au.dk/ausa/ausalib/wikis/home) and [ucesb](http://fy.chalmers.se/~f96hajo/ucesb/). We call the project [go4cesb](https://git.kern.phys.au.dk/ausa/go4cesb/wikis/home).
This project enables us to relatively easy write custom online analysis as well as using some prewritten analysis. See [this guide](https://git.kern.phys.au.dk/ausa/go4cesb/wikis/custom-analysis) for writing custom online analysis. Notice that `parameters.json` and `AusaUserAnalysis.h/cxx` are symbolic linked to the files in `~/<experiment number>/go4/`.
This project enables us to relatively easy write custom online analysis as well as using some prewritten analysis. See [this guide](https://gitlab.au.dk/ausa/go4cesb/wikis/custom-analysis) for writing custom online analysis. Notice that `parameters.json` and `AusaUserAnalysis.h/cxx` are symbolic linked to the files in `~/<experiment number>/go4/`.
The prewritten code makes it easy to include calibrations. Simply create the calibrations according to the `ausalib`[setup format](https://git.kern.phys.au.dk/ausa/ausalib/wikis/Setup). You will find the setup (and [`matcher.json`](https://git.kern.phys.au.dk/ausa/ausalib/wikis/JSON-Matcher) in `~/<experiment name>/` folder.
The prewritten code makes it easy to include calibrations. Simply create the calibrations according to the `ausalib`[setup format](https://gitlab.au.dk/ausa/ausalib/wikis/Setup). You will find the setup (and [`matcher.json`](https://gitlab.au.dk/ausa/ausalib/wikis/JSON-Matcher) in `~/<experiment name>/` folder.
## Grafana
## Grafana
[Grafana](https://grafana.com/) is a very nice graphing tool we use for time series data. We can send all kinds of information up with and display it very easy. Currently we send up information such as the VULOM scalers, the `drasi` stats and some more.
[Grafana](https://grafana.com/) is a very nice graphing tool we use for time series data. We can send all kinds of information up with and display it very easy. Currently we send up information such as the VULOM scalers, the `drasi` stats and some more.
Grafana is hosted as a CERN webservice and can be accessed at
Grafana is hosted as a webservice and can be accessed at
https://sec-monitor.web.cern.ch/
__NOTICE__: To login we use CERN's login system. The user is required to be subscribed to the `isolde` e-group, which can be done [here](https://e-groups.cern.ch/).
__MISSING LINK__
There are a few dashboards of interest. For the general setup performance, one can look at the `Triggers` and `Dead time` which shows the rate and dead time for each trigger. We will make purpose built dashboard(s) for each experiment, once we know which triggers goes where etc.
There are a few dashboards of interest. For the general setup performance, one can look at the `Triggers` and `Dead time` which shows the rate and dead time for each trigger. We will make purpose built dashboard(s) for each experiment, once we know which triggers goes where etc.
...
@@ -134,52 +134,36 @@ monitor
...
@@ -134,52 +134,36 @@ monitor
```
```
## Startup guide
## Startup guide
This section will give you the step-by-step guide to how to start up the acqusition from scratch. It is usually only required if one of the VME computers lost power or was rebooted. Otherwise a simple restart of __DAQC__ should be enough.
This section will give you the step-by-step guide to how to start up the acqusition from scratch. It is usually only required if the VME computer lost power or was rebooted. Otherwise a simple restart of __DAQC__ should be enough.
1. __Turn on the VME-crates and the NIM-crates__: Make sure that all crates have power and are running. Wait a minute or two, to make sure both VME computers are online.
1. __Turn on the VME-crate and the NIM-crates__: Make sure that all crates have power and are running. Wait a minute or two, to make sure both VME computers are online by ssh'ing into mot2.
2. __Reconfigure the trigger-module__: We need to reconfigure the `VULOM`. We can do this with the `rewire.sh` script on each VME computer. Do the following:
2. __Start up DAQC__: Next we start up the acquisition:
```bash
ssh rio2
cd daq/f_user
./rewire.sh
ssh rio7
cd daq/f_user
./rewire.sh
```
3. __Start the mux-server__:
This is required for __DAQC__ to be able to change the triggers.
```bash
ssh rio7
su
service mux-srv start
```
5. __Start up DAQC__: Next we start up the acquisition:
```bash
```bash
screen -R daqc
daqc
daqc
```
```
__DAQC__ will automatically reload the vulom trigger configuration, start the mux server and start drasi.
Notice that `daqc` is an alias. You can read some more details about what this actually does in the __FAQ__. We do, however, not need to worry about that now.
Notice that `daqc` is an alias. You can read some more details about what this actually does in the __FAQ__. We do, however, not need to worry about that now.
It will take some time to start up. The sequence will end with something like
It will take some time to start up. The sequence will end with something like
```
```
2018-06-14 11:02:37,833 - [rio7] Waiting for drasi 12
2018-06-14 11:02:37,833 - [mot2] Waiting for drasi 12
2018-06-14 11:02:38,835 - [rio7] Waiting for drasi 13
2018-06-14 11:02:38,835 - [mot2] Waiting for drasi 13
2018-06-14 11:02:39,837 - [rio7] Waiting for drasi 14
2018-06-14 11:02:39,837 - [mot2] Waiting for drasi 14
2018-06-14 11:02:40,839 - [rio7] Waiting for drasi 15
2018-06-14 11:02:40,839 - [mot2] Waiting for drasi 15
2018-06-14 11:02:41,841 - [rio7] Waiting for drasi 16
2018-06-14 11:02:41,841 - [mot2] Waiting for drasi 16
2018-06-14 11:02:42,843 - [rio7] Waiting for drasi 17
2018-06-14 11:02:42,843 - [mot2] Waiting for drasi 17
2018-06-14 11:02:43,845 - [rio7] Started Drasi
2018-06-14 11:02:43,845 - [mot2] Started Drasi
2018-06-14 11:02:43,846 - drasi_rio7 revived
2018-06-14 11:02:43,846 - drasi_mot2 revived
```
```
6. __Check that the data aqusition is running__: We can now go to the __DAQC__ frontend to monitor everything. Go to a browser and enter `polarbear:8080`. You should now see __DAQC__. Everything should be green. That means we are happy.
6. __Check that the data acqusition is running__: We can now go to the __DAQC__ frontend to monitor everything. Go to a browser and enter `bumblebee:8080`. You should now see __DAQC__. Everything should be green. That means we are happy.
If everything does not turn green within a minute or so, try to restart __DAQC__. This may, in particular, be required if only the crates died, but __DAQC__ kept running on __Bumblebee__.
If everything does not turn green within a minute or so, try to restart __DAQC__. This may, in particular, be required if only the crates died, but __DAQC__ kept running on __Bumblebee__.
7. __Check that the shaper settings are loaded__: If the NIM-crates' power was off, we should also load the settings for these. This assumes you have saved the settings in `settings.json`. Consult an elog or someone who has been setting up the shapers, which file you should be using. Open a terminal and issue
7. __Check that the shaper settings are loaded__: If the NIM-crates' power was off, we should also load the settings for these. This assumes you have saved the settings in `settings.json`. Consult an elog or someone who has been setting up the shapers, which file you should be using. Open a terminal and issue
```bash
```bash
mesyctrl
mesytec
```
```
Check that we see all modules. Now use `l` to load the settings file.
Check that we see all modules. Now use `l` to load the settings file.
8. __Push data to Grafana__: Open a terminal and issue (__DAQC__ must be running)
8. __Push data to Grafana__: Open a terminal and issue (__DAQC__ must be running)
...
@@ -192,6 +176,7 @@ This section will give you the step-by-step guide to how to start up the acqusit
...
@@ -192,6 +176,7 @@ This section will give you the step-by-step guide to how to start up the acqusit
- __Start DAQC__:
- __Start DAQC__:
Simply open a terminal and issue:
Simply open a terminal and issue:
```bash
```bash
screen -R daqc
daqc
daqc
```
```
You are now inside a screen session called `daqc`, and __DAQC__ is starting up. If __DAQC__ was already running, we simply entered the screen session called `daqc`.
You are now inside a screen session called `daqc`, and __DAQC__ is starting up. If __DAQC__ was already running, we simply entered the screen session called `daqc`.
...
@@ -211,33 +196,30 @@ This section will give you the step-by-step guide to how to start up the acqusit
...
@@ -211,33 +196,30 @@ This section will give you the step-by-step guide to how to start up the acqusit
```bash
```bash
daqc
daqc
```
```
If the readout does not restart properly (it takes >20 sec and just keep going, try to stop __DAQC__ and issue `die_drasi`, and then start __DAQC__
If the readout does not restart properly (it takes >20 sec and just keep going, try to stop __DAQC__ and issue `pkill drasi`, and then start __DAQC__
- __See DAQC__:
- __See DAQC__:
Go to a browser and open `polarbear:8080`
Go to a browser and open `bumblebee:8080`
- __Start a file__:
- __Start a file__:
Go to the browser with __DAQC__ (`polarbear:8080` in the URL). Hit the big blue button saying "Start". __REMEMBER__ to fill in the run sheet information!!!
Go to the browser with __DAQC__ (`bumblebee:8080` in the URL). Hit the big blue button saying "Start". __REMEMBER__ to fill in the run sheet information!!!
- __Stop a file__:
- __Stop a file__:
Go to the browser with __DAQC__ (`polarbear:8080` in the URL). Hit the big blue button saying "Stop". __REMEMBER__ to fill in the run sheet information!!!
Go to the browser with __DAQC__ (`bumblebee:8080` in the URL). Hit the big blue button saying "Stop". __REMEMBER__ to fill in the run sheet information!!!
- __Find my data__:
- __Find my data__:
The raw data is located in `~/<experiment name>/data/raw`. You can unpack it into a simple ROOT format with the `unpack.sh` script in the `data` directory. It will place the unpacked files in the `unpacked` directory
The raw data is located in `~/<experiment name>/data/raw`. You can unpack it into a simple ROOT format with the `unpack.sh` script in the `data` directory.
```bash
./unpackMBS.sh raw/15_*
```
- __Start Go4__:
- __Start Go4__:
To start __Go4__, open any terminal and issue
To start __Go4__, open any terminal and issue
```bash
```bash
go4
go4
```
```
Enter the `polearbear:5000` tab on the left, and then the `Histograms` folder.
Enter the `bumblebee:5000` tab on the left, and then the `Histograms` folder.
- __See the VULOM scalers__:
- __See the VULOM scalers__:
You can get the VULOM scalers from both the slave and the master. However, the master node is most interesting. To see the scalers on the master:
You can get the VULOM scalers from both the slave and the master. However, the master node is most interesting. To see the scalers on the master:
```bash
```bash
ssh rio7
ssh mot2
scalers
scalers
```
```
...
@@ -264,7 +246,7 @@ This section will give you the step-by-step guide to how to start up the acqusit
...
@@ -264,7 +246,7 @@ This section will give you the step-by-step guide to how to start up the acqusit
- __Start the Mesytec control__:
- __Start the Mesytec control__:
Find a terminal and issue
Find a terminal and issue
```bash
```bash
mesyctrl
mesytec
```
```
...
@@ -289,19 +271,10 @@ This section will give you the step-by-step guide to how to start up the acqusit
...
@@ -289,19 +271,10 @@ This section will give you the step-by-step guide to how to start up the acqusit
monitor
monitor
```
```
## Tools
#### Channels
If __DAQC__ is running, you can run
```bash
channels
```
in any terminal to get an overview of the rates is all ADC's and TDC's. It looks something like this:
Since the DAQ consists of so many moving parts, a lot can go wrong. However, a lot of these moving parts are non-essential, i.e. data can still be taken. The readout itself is very, very stable, i.e the two __RIO__'s and the event builder on __Bumblebee__. We have yet to see a proper crash of these.
Since the DAQ consists of so many moving parts, a lot can go wrong. However, a lot of these moving parts are non-essential, i.e. data can still be taken. The readout itself is very, very stable, i.e the two __RIO__'s and the event builder on __Bumblebee__. We have yet to see a proper crash of these.
What is much more likely to cause problems, and probably will during every experiment is __Go4__. Therefore, this should be the first suspicion.
What is much more likely to cause problems, and probably will during every experiment is __Go4__. Therefore, this should be the first suspicion.
The following flowchart will hopefully catch most situations. If you are stuck in a loop (an nothing starts working) do call an expert!
The following flowchart will hopefully catch most situations. If you are stuck in a loop (an nothing starts working) do call an expert!
