IDCP Parameters (StructuredData)¶
IDCP Parameters are stored and managed with StructuredData. This is a concept of storing the data in a single file in YAML format. StructuredData provides the programs SDview and SDshell. SDview is a GUI based program which is used to display and query the data. SDshell is a powerful text based shell which is used to query or modify the data. Since the data stored in a text file it can also be displayed or modified with a simple text editor.
The IDCP Parameters are stored, among other locations, here:
/opt/OPI/idcp/id_db.SDCyml
The file can be found there on these hosts:
elbe.acc.bessy.de
stretch.acc.bessy.de
nfs.blc.bessy.de
nfs.ctl.bessy.de
cree.blc.bessy.de
crow.blc.bessy.de
hscon1l.blc.bessy.de
hscon2l.blc.bessy.de
shoshoni.blc.bessy.de
The data is also provided by an XML-RPC server on gwc2c.acc.bessy.de port 7643, the “iddb” utility, which is described below, uses this server.
The software repository¶
The data file is created from file id_db.SDCyml which is part of a mercurial repository. The repository can be fetched with this command the following command. The data file itself can then be found in directory “data”. Note that you usually don’t need to do this, See further below for examples of a simple query of the data:
hg clone http://repo.acc.bessy.de/hg/id_db
Using iddb to query the data¶
Installation¶
iddb is installed on our development hosts,
If you have a linux system without iddb, you can install it like described below:
Installation of bii_scripts¶
iddb is part of bii_scripts. You can install the complete project, here is a link to the repository:
Direct installation of iddb¶
You can download iddb directly here:
iddb.
These extra steps are necessary to make it usable:
export IDDB_SOURCE=server:gwc2c.acc.bessy.de:7643
chmod u+x iddb
You should copy iddb to a directory that is defined in your PATH variable, e.g. $HOME/bin.
Usage¶
iddb does a query of the XML-RPC server on host gwc2c. It has many useful
options, you get help with iddb -h.
If you call iddb without any parameters you get an overview of all insertion devices:
$ iddb
name devicename key prefix application group status
--------------------------------------------------------------
BAM-WLS W7IT1R 0 WLS installed
HMI-WLS W7IT2R 0 WLS test
PSF-WLS W7IT7R 0 WLS installed
U125/1 U125IL2RP 90 idcp90 idcp MLS installed
U125/2 U125ID2R 3 idcp3 idcp BII installed
U139 U139ID6R 110 idcp110 idcp BII installed
U15 U15IV 80 idcp80 idcp DESY-U15 test
U17 U17IT6R 120 idcp120 idcp BII installed
U41 U41IT3R 6 idcp6 idcp BII installed
U49/1 U49ID4R 7 idcp7 idcp BII installed
U49/2 U49ID3R 5 idcp5 idcp BII installed
UE112 UE112ID7R 13 idcp13 idcp BII installed
UE46 UE46IT5R 10 idcp10 idcp BII installed
UE48 UE48IT6R 12 idcp12 idcp BII installed
UE49 UE49IT4R 8 idcp8 idcp BII installed
UE52 UE52ID5R 9 idcp9 idcp BII installed
UE56/1 UE56ID6R 11 idcp11 idcp BII installed
UE56/2 UE56ID8R 15 idcp15 idcp BII installed
UE56S UE56IS 95 idcp95 idcp BII installed
Ubonsai U1IV 99 idcp99 idcp BII simulated
With “find” you can query data for a given StructuredData path, here is an example:
$ iddb -i ue112 find 'global.*'
id-data.UE112.global.application : idcp
id-data.UE112.global.build_enabled : True
id-data.UE112.global.description : UE112 ,UE112ID7R,idcp13
id-data.UE112.global.device_status : installed
id-data.UE112.global.instance_no : 0
id-data.UE112.global.primary_key : 39
id-data.UE112.global.rsync_dist_group: BII
id-data.UE112.global.undulator : UE112
With “rxfind” alone you get all the data for an undlator:
$ iddb -i ue112 rxfind
id-data.UE112.accp-can-protocol.facility : bessy
id-data.UE112.accp-can-protocol.node.inhibit : 500
id-data.UE112.accp-can-protocol.node.nid : 13
id-data.UE112.accp-can-protocol.node.timeout : 1500
id-data.UE112.accp-can-protocol.vars.command.fields.cmd : 0
id-data.UE112.accp-can-protocol.vars.command.index : 0
id-data.UE112.accp-can-protocol.vars.command.server : self
id-data.UE112.accp-can-protocol.vars.command.type : RW,MULTIPLEX,CHAR,UNSIGNED
id-data.UE112.accp-can-protocol.vars.position.fields.aparshift: 3
(many more lines follow).
You can use this with a regular expression:
$ iddb --id ue112 rxfind '.*config.*max'
id-data.UE112.config.h_max_diff : 0.1
id-data.UE112.config.h_max_pos : 56.0
id-data.UE112.config.h_max_velocity : 0.42
id-data.UE112.config.max_par_is_time : True
id-data.UE112.config.v_max_diff : 0.1
id-data.UE112.config.v_max_pos : 185
id-data.UE112.config.v_max_velocity : 0.42
With the “dump” command you get the complete data for an insertion device as a python structure (only the first line is shown here for the UE112 undulator):
iddb dump ue112 | less
{'UE112': {'accp-can-protocol': {'facility': 'bessy',
...
Using an editor to view the data¶
On some hosts, the StructuredData data file is installed on the local host at a fixed location. On these hosts you can directly open the data file with you favorite editor, for example:
gedit /opt/OPI/idcp/id_db.SDCyml
The file format is YAML, if your editor supports syntax highlighting for this you use this option. When you want to find some parameters, with a simple editor, you are on your own. The tools described below offer more support for querying the data.
Using SDview to query the data¶
If you have StructuredData
installed, you can use SDview to query the data.
The program is described here: SDview.
Starting SDview¶
Log on host elbe.acc.bessy.de for example, then enter:
SDview -f /opt/OPI/idcp/id_db.SDCyml
This starts SDview and loads the file “idcp_db.SDCyml”.
SDview basics¶
Here is the initial window of SDview
Organization of the data¶
For a detailed explanation on the organization of the data please look here: OrganizationAnchor.
A first overview¶
After clicking on the button “expand” and enlarging the window a bit you get this picture:
You see all undulators as sub-nodes of node “id-data”.
Showing all parameter groups of a single undulator¶
If we now click for example on the small “+” near “U125/2” we get this picture:
Here you see all parameter groups of that undulator.
Showing a single parameter group of an undulator¶
If we now click on the small “+” near “names” within the parameter group of the U125/2 undulator we get this picture:
Here we see all parameters of the group “names” of the undulator “U125/2”. On the right side of the small white icons there is the parameter name followed by a colon “:” and the parameter value. Note that for string values (e.g. “devicename”) the value is enclosed in single quotes. Numbers like the value of “key” have no quotes.
Showing a single parameter group of an undulator¶
Here we show only some of the search capabilities of SDview. For a detailed description of all search options look here:
Searching for a given parameter name¶
We now want to find all parameters named “devicename” of all undulators. First we click on “collapse all” to collapse the tree. We select “ipattern” as a type and check “report only leaves” and “show paths and values” at “result window:”. We now enter “devicename” in the entry field after “pattern” and press <Return>.
This is how the program now looks like:
All matching parts in the tree are now marked red, although not all are currently visible. If you grab the scroll bar with your mouse and scroll you can see the other matches. You can also press the “up” or “down” buttons to move between matches. Note that if you click with the mouse into the tree, the selection is cleared. If have have done this by accident, simply click again on “search”.
Since we selected “show paths and values” an additional window has appeared which shows the search results in a text window. It looks like this:
Searching for a value¶
In this example we look for a given number. Note that we can also search values by ipatterns and regular expressions. Here however, we simply look for the number “180”. We select “find value” at “type:” and enter 180 after “pattern:”. Since there are many parameters in the parameter group “config” the first found value is visible first, we have to scroll to see it. The main window now looks like this:
Since we selected “show paths and values” an additional window has appeared which shows the search results in a text window. It looks like this:
Using SDpyshell to query the data¶
If you have StructuredData
installed, you can use SDpyshell to query the data.
You should be familiar with the basics of the SDpyshell, it is described in the chapter “Basic Syntax” here:
Starting SDpyshell¶
Log on to one of the development hosts and enter:
SDpyshell -f /opt/OPI/idcp/id_db.SDCyml
This starts the SDshell and loads the file “idcp_db.SDCyml”.
SDpyshell basics¶
SDpyshell has a history. You can retrieve previously entered commands by pressing the “up-arrow” key on your keyboard. You can use the “left-arrow” key in order to modify the command.
You leave SDpyshell by pressing Control-D or entering “quit”.
If you enter “help()” SDpyshell shows you a list of all commands and help topics. If you enter “help(“command”) (note the double quotes) SDpyshell shows a help for the given command. Here is an example:
>>> help("help")
help
****
help()
^^^^^^
This command for interactive use displays help for a given help topic or
command. If no arguments are given it displays all help topics.
The command takes the following parameters:
- item: This specifies the help topic which must be a string. This
parameter is optional. If it is not provided or None the command
displays a list of all help topics.
- level: This specifies the help level. This parameter is mandatory if a
help topic with the same name occurs at more than one place in the list of
help topics. It is then used to specify exactly which help topic is
requested. If you request a topic that is more than once in the list, the
help command shows you all topics together with their level parameter.
txt.help()
^^^^^^^^^^
This command returns the text that `help()`_ prints to the console as a string.
For an explanation of parameters look at the description of `help()`_.
fun.help()
^^^^^^^^^^
This command is identical to `txt.help()`_.
Organization of the data¶
IDCP parameters are organized in a hierarchical structure much like files in a directory tree. Similar to a filesystem each parameter can be identified by a path. Each value has a unique path.
The two keys at the top of the structure are “id-data” and “id-metadata”.
Below “id-data” are the parameters for each insertion device, below “id-metadata” are descriptions of the parameters. The keys below “id-data” are the human readable undulator names.
Here is an example how to list all undulators with the SDshell:
>>> paths("id-data.*")
- id-data.U125/1
- id-data.U125/2
- id-data.U139
- id-data.U2
- id-data.U3
- id-data.U4
- id-data.U41
- id-data.U48
- id-data.U49/1
- id-data.U49/2
- id-data.UE112
- id-data.UE46
- id-data.UE49
- id-data.UE52
- id-data.UE56/1
- id-data.UE56/2
- id-data.UE56R
- id-data.Ubonsai
For each undulator parameters are organized in groups. Here is a command that shows the parameter groups for the U125/1 undulator:
>>> paths("id-data.U125/1.*")
- id-data.U125/1.accp-can-protocol
- id-data.U125/1.config
- id-data.U125/1.correction
- id-data.U125/1.feedback
- id-data.U125/1.global
- id-data.U125/1.interface
- id-data.U125/1.measurement
- id-data.U125/1.names
- id-data.U125/1.network
- id-data.U125/1.operation
- id-data.U125/1.physical
- id-data.U125/1.referencing
This example shows all parameters of the group “network” for the U125/1 undulator:
>>> paths("id-data.U125/1.network.*")
- id-data.U125/1.network.bootserver
- id-data.U125/1.network.gateway
- id-data.U125/1.network.ioc
- id-data.U125/1.network.login
- id-data.U125/1.network.mount_filesystem
- id-data.U125/1.network.ntpserver
- id-data.U125/1.network.sec_lswitch_hosts
Simple queries¶
You can get all parameters of a group with the command “get”:
>>> get("id-data.U125/1.network", "yaml")
bootserver: nfs.mlscs.bessy.de
gateway: charm.mlscs.bessy.de
ioc: eis4gp.mlscs.bessy.de
login: epics
mount_filesystem: 0
ntpserver: 192.168.48.1
sec_lswitch_hosts: ''
If you use “find”, the complete parameter path is shown for each parameter:
>>> find("id-data.U125/1.network.**")
id-data.U125/1.network.bootserver : nfs.mlscs.bessy.de
id-data.U125/1.network.gateway : charm.mlscs.bessy.de
id-data.U125/1.network.ioc : eis4gp.mlscs.bessy.de
id-data.U125/1.network.login : epics
id-data.U125/1.network.mount_filesystem : 0
id-data.U125/1.network.ntpserver : 192.168.48.1
id-data.U125/1.network.sec_lswitch_hosts: ''
You can of course query the value of a single parameter, too:
>>> get("id-data.U125/1.network.ioc")
eis4gp.mlscs.bessy.de
Parameter metadata¶
Extra information for parameters such as a description is stored below “id-metadata”. There are two sub keys here, “id-path-info” and “parameter-info”:
>>> paths("id-metadata.*")
- id-metadata.id-path-info
- id-metadata.parameter-info
Parameter metadata: parameter-info¶
Below “parameter-info” descriptions are stored for most parameters. Here you do have not to know the parameter group name. This is an example for the parameter network.ioc:
>>> find("id-metadata.parameter-info.ioc.**")
id-metadata.parameter-info.ioc.description: the IOC's name
id-metadata.parameter-info.ioc.group : network
Parameter metadata: id-path-info¶
Below “id-path-info” descriptions are stored for parameter-paths. This is needed for parameters that are organized in a deeper structure than “group-name.parameter-name”.
Here is an example for one of the parameters of the ACCP protocol, which defines the CAN bus communication with the BESSY controlsystem. First we query the value of the parameter “vars.command.type” of the group “accp-can-protocol” of the U125/2 undulator:
>>> get("id-data.U125/2.accp-can-protocol.vars.command.type")
RW,MULTIPLEX,CHAR,UNSIGNED
Here we read the description of this parameter. Note that the parameter path is a simple key here, so we have to prepend the dots “.” in the path with a backslash:
>>> get("id-metadata.id-path-info.accp-can-protocol\.vars\.command\.type.**")
the type of the CAN variable
Finding a parameter if the exact name is not known¶
Suppose we want to find a parameter that contains the string “max” for the U125/1 undulator. In order to perform this search we use the command “ifind”. It takes a string as parameter which consists of words separated by spaces. It finds all paths were all the words can be found in any order. We have to search all parameter paths, so we must match “id-data”. We want the U125/1 undulator, so we match “U125/1” and we want everything where “max” is in the path. We do the search like this:
>>> ifind("id-data U125/1 max")
id-data.U125/1.config.v_max_diff : 0.1
id-data.U125/1.config.v_max_pos : 185
id-data.U125/1.config.v_max_velocity : 3500
id-data.U125/1.correction.ps_maxcurr_0: 3.0
id-data.U125/1.correction.ps_maxcurr_1: 3.0
id-data.U125/1.correction.ps_maxcurr_2: 3.0
id-data.U125/1.correction.ps_maxcurr_3: 3.0
id-data.U125/1.correction.ps_maxcurr_4: 3.0
id-data.U125/1.correction.ps_maxcurr_5: 3.0
There is also the command rxfind which can search all paths with a perl compatible regular expression.
Comparing a parameter for all insertion devices¶
Suppose we want a list of all device names for all undulators. We can use ifind again:
>>> ifind("devicename")
id-data.U125/1.names.devicename : U125IL2RP
id-data.U125/2.names.devicename : U125ID2R
id-data.U139.names.devicename : U139ID6R
id-data.U2.names.devicename : U2IV
id-data.U3.names.devicename : U3IVP
id-data.U4.names.devicename : U4IV
id-data.U41.names.devicename : U41IT6R
id-data.U48.names.devicename : U48IV
id-data.U49/1.names.devicename : U49ID4R
id-data.U49/2.names.devicename : U49ID3R
id-data.UE112.names.devicename : UE112ID7R
id-data.UE46.names.devicename : UE46IT5R
id-data.UE49.names.devicename : UE49IT4R
id-data.UE52.names.devicename : UE52ID5R
id-data.UE56/1.names.devicename : UE56ID6R
id-data.UE56/2.names.devicename : UE56ID8R
id-data.UE56R.names.devicename : UE56IV
id-data.Ubonsai.names.devicename: U1IV
Querying a group of parameters¶
We want to see all parameters whose paths start with “id-data.[undulator].names”. We specify this with a pattern. We use a wildcard at the end of the path “**” that indicates that we want all paths that start with the given pattern. We do the query like this:
>>> find("id-data.U125/1.names.**")
id-data.U125/1.names.devicename: U125IL2RP
id-data.U125/1.names.id : U125-1:Mls
id-data.U125/1.names.key : 90
id-data.U125/1.names.name : U125/1
id-data.U125/1.names.prefix : idcp90
Find parameters that have a certain value¶
Suppose we want to find all parameters that have the value “installed”. This is done with the command findval:
>>> findval("installed")
id-data.U125/2.global.device_status: installed
id-data.U139.global.device_status : installed
id-data.U41.global.device_status : installed
id-data.U49/1.global.device_status : installed
id-data.U49/2.global.device_status : installed
id-data.UE112.global.device_status : installed
id-data.UE46.global.device_status : installed
id-data.UE49.global.device_status : installed
id-data.UE52.global.device_status : installed
id-data.UE56/1.global.device_status: installed
id-data.UE56/2.global.device_status: installed
A complex query¶
Here is an example of a more complex query. You are not supposed to understand all details here, for more information look at the documentation of StructuredData
We want to have all device names of all insertion devices where the parameter “device_status” has the value “installed”:
>>> findval("installed", pattern="id-data.*.global.device_status")
id-data.U125/2.global.device_status: installed
id-data.U139.global.device_status : installed
id-data.U41.global.device_status : installed
id-data.U49/1.global.device_status : installed
id-data.U49/2.global.device_status : installed
id-data.UE112.global.device_status : installed
id-data.UE46.global.device_status : installed
id-data.UE49.global.device_status : installed
id-data.UE52.global.device_status : installed
id-data.UE56/1.global.device_status: installed
id-data.UE56/2.global.device_status: installed
If we call “fun.findval” instead of “findval” the results are not printed to the screen but returned. We now use “substpath” to modify the returned paths:
>>> substpath(fun.findval("installed", pattern="id-data.*.global.device_status") ,"*.*.names.devicename")
['id-data.U49/1.names.devicename', 'id-data.U49/2.names.devicename', 'id-data.UE49.names.devicename', 'id-data.UE52.names.devicename', 'id-data.U41.names.devicename', 'id-data.U125/2.names.devicename', 'id-data.U139.names.devicename', 'id-data.UE112.names.devicename', 'id-data.UE56/1.names.devicename', 'id-data.UE46.names.devicename', 'id-data.UE56/2.names.devicename']
If we call “fun.substpath” instead “substpath” the paths are returned and can be stored in a new variable “p”:
>>> p= fun.substpath (fun.findval("installed", pattern="id-data.*.global.device_status"), "*.*.names.devicename")
We now call “get” with this list of paths in order to retrieve the values, the undulator device names in this case. We use use the better readable “yaml” format for the output.
The first parameter “:**” matches any path, this is currently needed since the first parameter of “get” must never be omitted:
>>> get("**", paths=p, formatspec="yaml")
- U125ID2R
- U139ID6R
- U41IT6R
- U49ID4R
- U49ID3R
- UE112ID7R
- UE46IT5R
- UE49IT4R
- UE52ID5R
- UE56ID6R
- UE56ID8R
Programming¶
Since SDpyshell is based on the python programming language, you can define functions to make things easier. The following function for example returns the paths for all insertion devices where a certain parameter has a certain value:
>>> def idfind(par, val):
. p= fun.addpaths("id-data.*", par)
. found= fun.findval(val, pattern=p)
. return fun.substpath(found, "*.*")
.
We can now use this function to get paths for insertion devices that have a shift drive:
>>> idfind("physical.has_hdrive", 1)
['id-data.UE49', 'id-data.UE52', 'id-data.UE56R', 'id-data.U2', 'id-data.UE112', 'id-data.UE56/1', 'id-data.UE46', 'id-data.UE56/2']
We can also print the results in YAML format:
>>> format(idfind("physical.has_hdrive", 1), "yaml")
- id-data.UE49
- id-data.UE52
- id-data.UE56R
- id-data.U2
- id-data.UE112
- id-data.UE56/1
- id-data.UE46
- id-data.UE56/2
Query the data with xmlrpc¶
There is a xmlrpc server for the insertion device data running on host gwc2c.acc.bessy.de, port 7643.
Below we show a short example how to query the data from the interactive python shell. Note that you should only call functions from the text and functional layer. So function names should start with “txt.” or “fun.”. Unused parameters must be empty strings, the container name should always be “id_db”:
idadm@elbe: ~/cron > python3
Python 3.2.3 (default, Mar 25 2017, 08:49:29)
[GCC 4.7.2] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import pprint
>>> import xmlrpc.client
>>> s = xmlrpc.client.ServerProxy('http://gwc2c.acc.bessy.de:7643')
>>> pprint.pprint(s.fun.paths("id-data.*","","id_db"))
['id-data.BAM-WLS',
'id-data.HMI-WLS',
'id-data.PSF-WLS',
'id-data.U125/1',
'id-data.U125/2',
'id-data.U139',
'id-data.U15',
'id-data.U17',
'id-data.U41',
'id-data.U49/1',
'id-data.U49/2',
'id-data.UE112',
'id-data.UE46',
'id-data.UE48',
'id-data.UE49',
'id-data.UE52',
'id-data.UE56/1',
'id-data.UE56/2',
'id-data.UE56IV',
'id-data.Ubonsai']
>>> s.fun.get("id-data.U125/1.network.ioc",True,"","id_db")
'eis4gp.mlscs.bessy.de'
As convenience functions for the BII-Controls application, there are some additional functions in module “id” defined:
>>> pprint.pprint(s.id.p_ids(["installed"]))
['id-data.U49/1',
'id-data.U49/2',
'id-data.UE49',
'id-data.UE52',
'id-data.UE46',
'id-data.PSF-WLS',
'id-data.U41',
'id-data.U139',
'id-data.UE112',
'id-data.BAM-WLS',
'id-data.UE56/1',
'id-data.U125/2',
'id-data.UE56/2',
'id-data.U125/1']
>>> pprint.pprint(s.id.cioc_data()['UE48IT6R'])
{'device_status': 'test',
'name': 'UE48',
'nid': 12,
'vars': {'command': {'cid': 0,
'fields': {'cmd': {'description': 'mux of the ID command',
'mux': 0}},
'server': 'id-ioc',
'type': 'RW,MULTIPLEX,CHAR,UNSIGNED'},
'position': {'fields': {'aparshift': {'adel': 0.01,
'description': 'mux of the antiparallel shift',
'eguf': 218.447,
'egul': -218.453,
'hyst': 0.10000000000000001,
'max': 24.0,
'mdel': 0.01,
'min': -24.0,
'mux': 3,
'prec': 2,
'signal': 'ashift',
'unit': 'mm',
'value_description': 'Antiparallel Shift'},
'gap': {'adel': 0.01,
'description': 'mux of the gap',
'eguf': 218.447,annotated_
'egul': -218.453,
'hyst': 0.10000000000000001,
'max': 185,
'mdel': 0.01,
'min': 16.0,
'mux': 0,
'prec': 2,
'signal': 'gap',
'unit': 'mm',
'value_description': 'Gap'},
'parshift': {'adel': 0.01,
'description': 'mux of the parallel shift',
'eguf': 218.447,
'egul': -218.453,
'hyst': 0.10000000000000001,
'max': 24.0,
'mdel': 0.01,
'min': -24.0,
'mux': 2,
'prec': 2,
'signal': 'pshift',
'unit': 'mm',
'value_description': 'Parallel Shift'},
'reserved': {'adel': 0.01,
'description': 'unused mux',
'eguf': 218.447,
'egul': -218.453,
'hyst': 0.10000000000000001,
'mdel': 0.01,
'mux': 1,
'prec': 2}},
'rsob': 12,
'server': 'cioc',
'type': 'WO,MULTIPLEX,SHORT,SIGNED'},
'status': {'cid': 12,
'fields': {'stat': {'description': 'mux of the ID status',
'mux': 0}},
'server': 'id-ioc',
'type': 'RW,MULTIPLEX,CHAR,UNSIGNED'}}}
Note that cioc_data is intended to be called only once, since it returns a data structure for all installed insertion devices at the Bessy II storage ring.