Parameters / Configuration Files / Parameter Mapping

This part describes the different categories and types of parameters supported by the OPALS framework, how they are specified via the 3 bindings and / or configuration files, and how to efficiently set parameter values of the current module based on the results of preceding module invocations stored in parameter files.

Each parameter is associated with a certain name, type, semantic, category, and optionality.

Parameter Categories

OPALS parameters belong to either of 3 categories:

  • Specific parameters are peculiar to each module, having a module-specific type and semantic. For a list of all specific parameters supported by a certain module, see the respective section referenced in List of OPALS Modules.
  • Common parameters control functionality that is implemented for all modules by the OPALS framework. Hence, they are supported by all modules, always having the same type and semantic. For a list of all common parameters, see Table of OPALS common parameters
  • Global parameters also form a fixed set of parameters supported by all modules. However, the values of global parameters are thought to be constant for a certain OPALS installation, user, or project, and are thus expected to be specified in configuration files only. For a list of all global parameters, see Table of OPALS global parameters

Parameter Types

Every parameter is associated with a certain data type. All parameter types exposed by OPALS modules are either built-in data types defined by the C++ standard (int, double, etc.), container types defined in the C++ Standard Template Library (std::vector, std::list, etc.), or custom data types declared in OPALS header files (e.g. opals::Array, or enumerators like opals::LogLevel::Type). Thus, when using the C++ - interface and dynamic linkage, values of these data types may be instantiated and passed directly. However, when using OPALS modules as executables or Python modules, values that are convertible to the respective internal type need to be passed, where the set of convertible types / values depends on both the data type and the binding in use: parameter values for

  • executables are always passed as strings, while values for
  • Python modules are passed as Python types that are 'similar' to the respective C++ type

Parameter types can be classified into

  • scalars, holding a single value only:
    • numeric scalars
    • (character) strings
    • enumerators
    • custom scalar types other than enumerators, e.g. file paths (opals::Path)
  • containers, which hold one or more scalars of the container's element type

Referencing the installation root directory

Parameters that specify file paths are of type opals::Path. When encountering parameter values of type opals::Path that begin with the (case-sensitive) string $OPALS_ROOT, the OPALS framework internally replaces this sub-string with the installation's root directory. Thus, "$OPALS_ROOT/addons/pal/standardPalette.xml" is translated internally to the absolute path of the standard palette file.

Parameter Optionalities

A parameter's optionality defines if users must, may, or must not set its value:

  • mandatory parameters must be specified;
  • estimable parameters and parameters with a default may be given a custom value;
  • optional parameters may be given a value; if not, respective functionality is skipped;
  • output parameters hold results only. Hence, they must not be set, but only queried;

The optionality of each parameter is given in the respective parameter description.

Specification of Parameters on the Command-Line

OPALS executables may be invoked from the operating system's shell / command prompt by calling the respective file path. The characters following this path are split into a list of character strings (tokens) according to the system's rules. Under Windows, blanks are interpreted as token separators. This separation may be circumvented by enclosure in double quotes (for further information, see the respective page in MSDN).

The list of tokens is then interpreted by the OPALS framework:

  • tokens beginning with one or two hyphens ('-') are interpreted as parameter names (excluding the hyphens)
  • all tokens up to the next parameter name are considered as representing corresponding parameter values. These tokens are then converted to their internal representation. Thus, users may be confronted with 2 difficulties resulting from the syntax in use:
    • (character) strings including blanks that should be converted to a single parameter value must be prevented from being split into different tokens using the operating system's rules (e.g. enclosure in double quotes). Above all, this may be necessary for file paths.
    • tokens that start with a hyphen and represent parameter values: these tokens must be prevented from being recognized as parameter names by prefixing the hyphen with a backslash ('\').

Values extracted from the passed tokens are converted and set in alphabetic order of parameter names, except for the following common parameters, which are processed first, and in the following order:

  1. cfgFile
  2. paramMapping
  3. inParamFiles
  4. screenLogLevel
  5. fileLogLevel
  6. logFile

Example

The following code submitted on a command-line on Windows calls Module Import, sets the minimum importance of log messages to be exported to file to opals::LogLevel::verbose, and imports 2 files:

opalsImport -inFile fileNameWithoutBlanks.las "file name with blanks.las" -fileLogLevel verbose

Specification of Parameters in the Python Shell

In the Python shell, parameter values are set by calling the respective set_<parameter-name> - member function of a module instance. Of course, numeric types expect numeric Python types to be passed - the same holds for character strings. Other C++ types expect the following counterparts from the Python shell:

  • opals::Path expects a Python string
  • C++ enumerators are provided by OPALS modules as custom Python types, e.g. LogLevel.info
  • Other custom scalar types must be passed as strings
  • Container types must be passed as Python lists. If only a single value should be set for a container type, a scalar may be passed instead.

Example

The following Python code snippet instantiates Module Import, sets the minimum importance of log messages to be exported to file to opals::LogLevel::verbose, and imports 2 files:

import opals
from opals import Import
theImp = Import.Import();
theImp.fileLogLevel = opals.Types.LogLevel.verbose
theImp.inFile = [ r"demo\strip18.odm", r"demo\strip19.odm" ]

Note: by default, Python interprets backslashes contained in strings as escape characters. As shown above, this can be circumvented by prefixing the string definition with 'r'.

Specification of Parameters in Configuration Files

Parameter values may not only be specified on the command-line or via the Python / C++ interface, but all bindings of OPALS modules additionally parse configuration files at certain locations and set parameters from these, if present:

  1. $OPALS_ROOT/cfg/opals.cfg
    where $OPALS_ROOT denotes the root directory of the OPALS installation
  2. <home>/opals/opals.cfg
    where <home> denotes the user's home path, e.g. "C:\Documents and Settings\Willi\" under Windows
  3. <cwd>/opals.cfg
    where <cwd> denotes the current working directory
  4. <user-configuration-file>
    only parsed if set on the command-line / via the interface, or in one of the configuration files above, using the common parameter 'cfgFile'

As the same parameter may be present in more than 1 configuration file, a precedence has been defined: parameter values specified in configuration files mentioned later in the list above override those specified in files listed earlier. Kindly note that this also holds true for container types: values specified in different configuration files do not add (while multiple definitions in the same file do, see below). Furthermore, parameter values specified via the command-line / interface override any corresonding values found in (non-user-) configuration files.

Note that user-configuration-files are processed (and respective values are set) immediately. Thus, if a parameter value is specified both in a supplied user-configuration-file and via the interface, the order of respective member function calls is of importance.

Configuration File Syntax

Configuration files are INI-like ASCII text files that are interpreted line by line. Lines containing white-space only are ignored, as all characters including and following a hash mark ('#') are. Parameter values are generally specified using the syntax

parameterName = parameterValue

i.e. the character string up to the first equals sign ('=') (excluding leading and trailing white-space) is interpreted as the name of the parameter that the value(s) represented by the character string following the equals sign shall be assigned to. The string representing the value(s) to be assigned is split into a list of tokens in the same way as is done on the command-line (see Specification of Parameters on the Command-Line). Multiple specifications of container type parameters in the same configuration file add up. Thus, several elements may be assigned to a container type parameter in either of two ways:

containerParameterName = parameterValue1
containerParameterName = parameterValue2
containerParameterName = parameterValue1 parameterValue2

Using this syntax, the value of the parameter associated with the respective name will be set for any module that supports this parameter. However, setting the value can be restricted to a certain module only using the following sytax:

moduleName.parameterName = parameterValue

The same effect can be achieved by placement of the module name in squared brackets on a preceding line:

[moduleName]
# all following specifications refer to module 'moduleName'
parameterName = parameterValue

Example

# For all modules,
# set the minimum importance of log messages to be written to file to opals::LogLevel::info
fileLogLevel = info
# Only for Module 'Import',
# set the minimum importance of log messages to be written to file to opals::LogLevel::verbose
opalsImport.fileLogLevel = verbose

Variable Support in Configuration Files

OPALS supports variable substitution within configuration files. It is possible to refer to other parameters in the value string as shown below (The concept is also known as (extended) interpolation in the configparser class in python). The feature can be applied e.g. within package script configuration files for defining parameter values only once or defining all adaptable parameters in a central section called [script.variables]. Parameters from the current section or the [script.variables] section can be referenced without section prefix. Reference from other sections require the full section prefix. E.g.

# define a special input filter for opalsGrid
filter = "generic[classification==2]"
# use the same input filter for opalsCell
filter = ${opalsGrid.filter}

An example using the central variable section is in the following

# the following radius should be used as neighbourhood definition size for all modules
script.variables.radius = 5
opalsGrid.gridSize = ${radius}
opalsCell.cellSize = ${radius}
opalsNormals.searchRadius = ${radius}

For completeness it is mentioned that the substitution mechanism is working recursively up to a depth of 10. Hence, a variable that refers to an other variable is fully supported.

The substitute implementation searches for the ${...} pattern in all parameter values. Currently, there is no way of disabling the substitution mechanism. Since the distinctive pattern may not occur in OPALS filter syntax and is unlikely to be used within file names, this shortcoming should be of minor concern.

Parameter Mapping

The invocation of each module's central function (run()) returns a list of all final module parameters, possibly including unchanged input-parameters, but more interestingly also (estimated) paths of exported files and computed results not stored to file. The list of final parameters may be

  • stored to a parameter file (in XML format) using the common parameter 'outParamFile' (must be set before run() when using Python / C++)
  • assigned to a variable on the calling side when using a Python or C++ binding

This list of final parameters may then be used to set the parameters of a later module call (probably of another module) by:

  • reading the previously written file using the common parameter 'inParamFiles', or
  • passing the list in memory using the variable from the calling side (Python and C++ only) and the function 'setParams(...)' present in all modules

When passing a list of parameters as input to the current module, the common parameter 'paramMapping' comes into play, as it specifies which parameters in the list shall be assigned to which parameters of a module. The syntax used is somehow similar to the one used in configuration files.

ParamNameToAssign = ParamNameToUse

Takes the last parameter with name 'ParamNameToUse' found in the (ordered) list of parameters, and assigns it to the parameter named 'ParamNameToAssign' of the current module. This mapping of output to input parameters may be restricted to output parameters from certain modules only:

ParamNameToAssign = OutputModule.ParamNameToUse

Assigns the parameter named 'ParamNameToUse' of the last call of only the module named 'OutputModule' to the parameter named 'ParamNameToAssign' of the current module. Likewise, parameters may be mapped to parameters of certain modules only:

InputModule.ParamNameToAssign = ParamNameToUse

Assigns the parameter named 'ParamNameToUse' to the parameter named 'ParamNameToAssign' only, if the current module is named 'InputModule'. These two types of restrictions may be combined to result in a most restrictive mapping:

InputModule.ParamNameToAssign = OutputModule.ParamNameToUse

Assigns the parameter named 'ParamNameToUse' from the last call of 'OutputModule' to the parameter named 'ParamNameToAssign' of the current module, if it is named 'InputModule'.

Multiple mappings to the same parameter must be separated by commas:

ParamNameToAssign = ParamNameToUse, AnotherParamNameToUse

Assigns the last ocurrence of any parameter either named 'ParamNameToUse' or 'AnotherParamNameToUse' to the current module's parameter named 'ParamNameToAssign'.

Mappings to different parameters must be separated by semicolons:

FirstParamNameToAssign = FirstParamNameToUse; SecondParamNameToAssign = SecondParamNameToUse;

Concerning container types to be assigned, parameter mapping works in a greedy manner: while scalar parameters and the fixed-size container opals::Array take exactly as many values from the list as needed (1 or N, resp., beginning at the end of the list), all values found in the list that satisfy the mapping rules will be assigned to vector-type parameters.

As the OPALS framework does not check the types of parameters to be mapped beforehand, it is the user's responsibility to assure that values to be used are convertible to the types of parameters to be assigned.

Note: as the character string specifying the value of the common parameter 'paramMapping' is split into tokens like any other parameter value on the command-line or configuration file, it is necessary to enclose it in double quotes under Windows, if it contains white-space.

Data-Dependent Conditions

In addition to restrictions based on module and parameter names, the parameter values of the current module and the parameter in question may optionally be combined to a logical expression. If for a certain mapping, the restrictions on module and parameter names are fulfilled, then the value from list/file is only assigned to the corresponding value of the current module, if this expression evaluates to true. If given, expressions must follow the mapping, separated by white space, and be formulated in Python syntax. The current module's parameters are exposed through the variable named 'me', having attributes only for the currently set parameters. The parameter under question may be accessed via the variable 'you'. 'you' and the exposed attributes of 'me' are fully typed, why respective methods/member functions may be invoked. Depending on the complexity of the involved parameter types, elaborate expressions may be composed, eventually using functions built into Python. As the parameter mapping string itself must be double quoted, it is advisable to enclose string constants in single quotes.

Example

In a configuration file, set the parameter mapping for all modules in a way that the final parameter 'outFile' of Module Import will be assigned to the parameter 'inFile' of Module Grid :

paramMapping = "opalsGrid.inFile = opalsImport.outFile"

The same effect can be achieved by setting the parameter 'paramMapping' for Module Grid only:

opalsGrid.paramMapping = "inFile = opalsImport.outFile"

The following code submitted to the command-line writes the final parameters of Module Import to a parameter file. The subsequent call to Module Grid uses this file in combination with the parameter mapping defined as above in a (default-) configuration file to set the ODM file path as input:

opalsImport -inFile demo\strip18.las -outParamFile results.xml
opalsGrid -inParamFiles results.xml

The code snippet below shows a parameter mapping being restricted by a data-dependent expression: a transformation of type opals::TrafPars3dAffine produced by opalsGeorefApprox shall be assigned to the parameter 'trafo' of opalsExport (having the same type). Separated by white space, the condition in Python syntax follows the actual mapping, restricting the set of candidate parameters to those whose member 'IdGridMov' is a prefix of the current module's inFile - strictly speaking, the respective file stems are compared:

  • The grid ID of the moving grid which the candidate transformation parameter represented by 'you' refers to, is extracted using opals::TrafPars3dAffine's member function getIdGridMov() .
  • Using the resulting string, an opals::Path is constructed, whose file stem (file name without extension) is returned from opals::Path's member function stem() .
  • The value of parameter 'inFile' of type opals::Path of the current module is assessed via the respective attribute of 'me'.
  • Again, a string results from stem() .
  • As a Python string results, we may directly apply its method startswith, which is used to check whether the string originating from 'you' is a prefix.
    paramMapping = "trafo = opalsGeorefApprox.outTrafPars me.inFile.stem().startswith(Path(you.getIdGridMov()).stem())"

The following Python script passes the list of final parameters of Module Import as input to Module Grid, using the variable 'results' for storage:

import opals
from opals import Import, Grid
theImp = Import.Import()
theImp.inFile = r"demo\strip18.odm"
results = theImp.run()
theGrid = Grid.Grid()
theGrid.mapParams( results )
theGrid.run()

As in this simple example, only one scalar, numeric parameter is passed via the parameter file, of course the following approach is equally practical:

theImp.run()
outFile = theImp.outFile
theGrid = Grid.Grid()
theGrid.inFile = outFile

Parameters Supported by all Modules

OPALS Common Parameter

-nbThreads Type: unsigned int
Remarks: mandatory
Description: number of concurrent threads
The number of threads to spawn in modules with parallel processing support
-screenLogLevel Type: opals::LogLevel
Remarks: optional
Description: verbosity level of screen output
Possible values:  
  none ...... Suppress all logging output.
  error ..... Some failure that cannot be handled. Program execution is aborted.
  warning ... Some weird program state which can still be handled (e.g. 'poor matrix condition', 'poor data distribution')
  info ...... Some progress that may be interesting in everyday-use
  verbose ... Something that may help in understanding normal program behaviour
  debug ..... Anything that may help 'debugging' by means of a release-build
The minimum level of log message importance to print on the screen
-fileLogLevel Type: opals::LogLevel
Remarks: optional
Description: verbosity level of log file output
Possible values:  
  none ...... Suppress all logging output.
  error ..... Some failure that cannot be handled. Program execution is aborted.
  warning ... Some weird program state which can still be handled (e.g. 'poor matrix condition', 'poor data distribution')
  info ...... Some progress that may be interesting in everyday-use
  verbose ... Something that may help in understanding normal program behaviour
  debug ..... Anything that may help 'debugging' by means of a release-build
The minimum level of log message importance to export to the xml-log
-lineBufferedScreenLog Type: bool
Remarks: mandatory
Description: line buffered log output to screen
Flush the screen log stream (stdout) each time a log message has been emitted. Useful for watching progress when stdout has been redirected to a file. Otherwise, it is flushed only when its buffer is full.
-logFile Type: opals::Path
Remarks: optional
Description: log file path
The path of the xml log file. Write permission needed.
-cfgFile Type: opals::Path
Remarks: optional
Description: configuration file
User-defined path to a configuration file
-paramMapping Type: opals::String
Remarks: mandatory
Description: mapping of parameters from file to own parameters
specify the mapping of parameters read from file to those of the current module
-inParamFiles Type: opals::Vector<opals::Path>
Remarks: optional
Description: parameters to import from file
read parameters from XML parameter files
-outParamFile Type: opals::Path
Remarks: optional
Description: final parameter export
Save the final parameters to this file
-scope Type: opals::String
Remarks: optional
Description: scope of execution
Restrict the assignment of parameters in user-cfg-files to scope.moduleName.paramName
-deleteEmptyOutFile

Type: bool
Remarks: optional
Description: delete empty output file
Delete output file after processing if it does not contain a single valid data value

OPALS Global Parameter

-coord_system Type: opals::String
Remarks: mandatory
Description: default coordinate system (EPSG Code or WKT string)
A default coordinate system is assigned for all data sets lacking a specific geo-referencing information.
The coord. system can either be defined as an EPSG Code (EPSG:nnnn) or as a Well Known Text String (WKT:...)
-points_in_memory Type: unsigned int
Remarks: optional
Description: limit number of points kept in memory by the ODM
As long as the ODM has less points in memory than the given limit, an additional tile is loaded in to memory. Once the limit is exceed and a new tile is requested, the ODM swaps unneeded tils onto disk based on a recent used list.
-oformat_grid Type: opals::String
Remarks: mandatory
Description: default grid output format
More Details...
-oformat_vector Type: opals::String
Remarks: mandatory
Description: default vector output format
More Details...
-oformat_tin Type: opals::String
Remarks: mandatory
Description: default TIN output format
More Details...
-oformat_lidar Type: opals::String
Remarks: mandatory
Description: default lidar output format
This format is used as default LIDAR file format in case no format was specified (e.g. opalsEpxort)
-create_option Type: opals::Vector<opals::String>
Remarks: optional
Description: dataset create options
The creation of datasets (grid, vector, tin...) may be controlled by one or multiple create options (e.g., coordinate precision, tile size, compression ...). For grid/raster datasets, the create option strings must correspond to the defintion of the respective GDAL driver. Please not, that only well formatted options are considered.
-postfix_z Type: opals::String
Remarks: mandatory
Description: postfix for grid files representing z
More Details...
-postfix_attribute Type: opals::String
Remarks: mandatory
Description: postfix for grid files repesenting an abitrary attribute
More Details...
-postfix_min Type: opals::String
Remarks: mandatory
Description: postfix for minimum grid files
More Details...
-postfix_max Type: opals::String
Remarks: mandatory
Description: postfix for maximum grid files
More Details...
-postfix_range Type: opals::String
Remarks: mandatory
Description: postfix for differences between maximum and minimum grid files
More Details...
-postfix_nmin Type: opals::String
Remarks: mandatory
Description: postfix for n-minimum grid files
More Details...
-postfix_nmax Type: opals::String
Remarks: mandatory
Description: postfix for n-maximum grid files
More Details...
-postfix_rms Type: opals::String
Remarks: mandatory
Description: postfix for rms grid files
More Details...
-postfix_var Type: opals::String
Remarks: mandatory
Description: postfix for variance grid files
More Details...
-postfix_mean Type: opals::String
Remarks: mandatory
Description: postfix for mean grid files
More Details...
-postfix_median Type: opals::String
Remarks: mandatory
Description: postfix for median grid files
More Details...
-postfix_sum Type: opals::String
Remarks: mandatory
Description: postfix for sum grid files
More Details...
-postfix_minority Type: opals::String
Remarks: mandatory
Description: postfix for minority grid files
More Details...
-postfix_majority Type: opals::String
Remarks: mandatory
Description: postfix for majority grid files
More Details...
-postfix_entropy Type: opals::String
Remarks: mandatory
Description: postfix for entropy grid files
More Details...
-postfix_center Type: opals::String
Remarks: mandatory
Description: postfix for closest-to-cell-center grid files
More Details...
-postfix_pdens Type: opals::String
Remarks: mandatory
Description: postfix for point density grid files
More Details...
-postfix_pcount Type: opals::String
Remarks: mandatory
Description: postfix for point count grid files
More Details...
-postfix_quantile Type: opals::String
Remarks: mandatory
Description: postfix for quantile grid files
More Details...
-postfix_sigmaz Type: opals::String
Remarks: mandatory
Description: postfix for sigmaZ grid files
More Details...
-postfix_sigma0 Type: opals::String
Remarks: mandatory
Description: postfix for sigma0 grid files
More Details...
-postfix_excen Type: opals::String
Remarks: mandatory
Description: postfix for excentricity grid files
More Details...
-postfix_slope Type: opals::String
Remarks: mandatory
Description: postfix for slope grid files
Rasters of this type contain the steepest slope [%]
-postfix_slpDeg Type: opals::String
Remarks: mandatory
Description: postfix for slpDeg grid files
Rasters of this type contain the steepest slope [deg]
-postfix_slpRad Type: opals::String
Remarks: mandatory
Description: postfix for slpRad grid files
Rasters of this type contain the steepest slope [rad]
-postfix_expos Type: opals::String
Remarks: mandatory
Description: postfix for expostion grid files
More Details...
-postfix_nx Type: opals::String
Remarks: mandatory
Description: postfix for normal x grid files
normal-x grids contain the x-component of the surface normal unit vector
-postfix_ny Type: opals::String
Remarks: mandatory
Description: postfix for normal y grid files
normal-y grids contain the y-component of the surface normal unit vector
-postfix_openness Type: opals::String
Remarks: mandatory
Description: postfix for openness grid files
openness grids contain the local viewshed (i.e. maximum opening angle of a cone)
-column_name_alias Type: opals::Vector<opals::ColumnNameAlias>
Remarks: optional
Description: column name alias
More Details...
-postfix_kmin Type: opals::String
Remarks: mandatory
Description: postfix for minimum curvature grid files
kmin grids contain the minimum curvature of a specific surface point
-postfix_kmax Type: opals::String
Remarks: mandatory
Description: postfix for maximum curvature grid files
kmmax grids contain the maximum curvature of a specific surface point
-postfix_kmean Type: opals::String
Remarks: mandatory
Description: postfix for mean curvature grid files
kmean grids contain the mean curvature of a specific surface point: kmin = (kmin+kmax)/2
-postfix_kgauss Type: opals::String
Remarks: mandatory
Description: postfix for gaussian curvature grid files
kgauss grids contain the gaussian curvature of a specific surface point: kgauss = kmin*kmax
-postfix_kminDir Type: opals::String
Remarks: mandatory
Description: postfix for azimuth of minimum curvature grid files
kminDir grids contain the direction (azimuth) of the the minimum curvature in a specific surface point
-postfix_kmaxDir Type: opals::String
Remarks: mandatory
Description: postfix for azimuth of maximum curvature grid files
kmaxDir grids contain the direction (azimuth) of the the maximum curvature in a specific surface point
-postfix_absKmaxDir Type: opals::String
Remarks: mandatory
Description: postfix for azimuth of maximum absolute curvature grid files
absKmaxDir grids contain the direction (azimuth) of the the maximum absolute curvature in a specific surface point
-postfix_precision

Type: opals::String
Remarks: mandatory
Description: forgotten to check-in by gm

Author
wk,le,lwiniwar
Date
09.11.2011