Module Import
See Also
opals::IImport

Aim of module

Loads vector and raster data into an ODM for subsequent use in modules dealing with point cloud and line related data.

General description

Module Import tries to preserve the full information of imported file formats. Therefore, the ODM contains an additional administration level, storing header and layer information for each file. Hence, it is possible to reproduce the original files from an ODM file (even in case of a multiple file import). Currently, Module Import supports 6 different vector file formats. Additionally, any of the supported raster formats may be imported. However, please notice that this is a rather rare use case: raster data imported into an ODM will be treated like any other data in an ODM i.e. the topological information defined implicitly by rasters is lost.

See Also
Module Import

OPALS uses the OPALS Data Manager (ODM) for the administration of ALS point clouds as well as line related geometries (eg. breaklines, formlines, borderlines). All subsequent OPALS modules dealing with primary topographic data are based on a data manager file (*.odm) as data input. Thus, running Module Import is a pre-requisite step for many OPALS application modules like Module Grid, Module Cell, Module Bounds ...

The ODM can be seen as a database system with focus on spatial data administration. Since the data are organized in a spatial order, the ODM provides efficient spatial access. The second key feature of the ODM is the storage of an arbitrary number of attributes (in OPALS they are referred to as additional info) for each geometry entry. Moreover, additional infos (addInfo) can be dynamically added at any time, allowing OPALS modules a transparent propagation of computational results across module boundaries (eg. Module Normals which estimates surface normals in each point).

For more details on the ODM and its features please refer to OPALS Data Manager manual.

In general it is not necessary to specify the file format of the input file since OPALS automatically detects the format based on the file content (and NOT based on the extension). In case the recognition fails and for formats described by a generic opals format definition, the file format can be specified by the iFormat parameter. opalsImport supports the listed format labels as shown below. For GDAL file formats it is either possible to use the generic format label gdal or the specific GDAL driver name as listed here .

Format Description
wnp, bwnp Winput and binary Winput files
xyz, bxyz xyz and binary xyz files
las LAS files
shape ESRI shape file format
sdw Riegl SDW files
fwf IPF internal full waveform ascii format (version: 0.8, 0.9, 1.0, and 1.1)
<opals format definition xml file> Generic import of ASCII and binary file
trj, btrj ASCII and binary trajectory files. For details see Trajectory file
scop SCOP DTM file format (Random Digital Height, RDH)
gdal GDAL readable file format (see Supported grid/raster formats)
<GDAL driver name> Use a specific GDAL driver to read the file. Driver names are listed in Supported grid/raster formats

To import multiple files one can set all input files at once or make multiple Module Import runs in appending mode. To append data to an ODM file define the corresponding ODM file as first input file or define the ODM file as input and output file.

For some LiDAR processing steps it is necessary to know the beam vector for each point. Therefore opalsImport provides the possibility to consider trajectory files during import. It is possible to specify a single trajectory file that is used for all input files or multiple trajectory files where each trajectory file corresponds to an input file. In case the gps time range of the trajectory file doesn't fully cover the gps time range of the corresponding input file an exceptions will be thrown. For situations where the trajectory is given in multiple files which do not match the time range of input files, it is necessary to concatenate all trajectory data into a single trajectory file. Beside the beam vector in the world coordinate system (default option) it is possible to append the range, the scan angle and the beam vector in the sensor coordinate system (SCS) (see parameter storeBeamInfo). The latter two options require the computation of the beam vector in the sensor coordinate system which is the inverse operation of opalsDirectGeoref (a mounting calibration is not supported yet). Please note that the correct transformation can only be applied if the world coordinate system (see global parameter coord_ref_sys) is specified correctly (For details refer to the section World coordinate system). The beam vector in SCS can be used e.g. to improve the mounting calibration for data sets where only the final word coordinate point data and the trajectory is available by creating artificial strip files in the sensor coordinate system.

The link between point data and trajectory is given by the gps time, which limits this import parameter to file formats containing gps time information. The flight position at shot time is interpolated in a piecewise linear manner within the trajectory data. OPALS supports different trajectory file formats including the generic import. The format has to provide the data columns x, y, z, gps time, roll, pitch and yaw. Please refer to section trajectory file for further details.

The ODM tiles point data for efficient handling of huge data sets. Since tiles are always completely loaded into the memory, appropriate tile size estimation is important. This is why a spatial index statistics is reported after import. Too big tiles can lead to memory problems, too small tiles increase the administration overhead. The point density of LiDAR data is usually quite homogeneous making the tile size estimation non-crucial. However, in rare situations it can be necessary to influence the estimation process by setting the tileSize or the tilePointCount parameter. Please refer to section Analysing the index statistics of an ODM for further details.

Finally, it should be mentioned that it is possible to filter data during import in a very flexible way. This can be relevant if only a subset of the data should be passed to a processing chain. For details on filters and how they are defined please refer to the Filter manual.

Parameter description

-inFileinput files
Type: opals::Vector<opals::Path>
Remarks: mandatory
The specified input files are imported in the same order as defined.
To append data to an ODM file, specify the ODM file as input and outFile. Note that wildcard characters (*,?) can be used for the selection of multiple input files at once (e.g. strip*.las)
-iFormatfile format [auto, xyz, bxyz, wnp, bwnp, las, shape, sdw, fwf, odm, scop, <opals format def. xml file>, gdal, <GDAL driver name>]
Type: opals::Vector<opals::String>
Remarks: estimable
In case of multiple input files with present iFormat parameter, the number of iFormat and inFiles entries have to match or only a single iFormat entry is specified (=all input files have the same format).
This also applies if an additional data file is to be appended to an existing ODM in which case the ODM has to be specified twice, as input and output.
Estimation rule: auto. The file content is used to recognise the file format.
-trjFiletrajectory files
Type: opals::Vector<opals::Path>
Remarks: optional
If trajectory file(s) are provided, trajectory information (e.g. beam vector, see parameter storeBeamInfo) for each point is computed and stored as attributes within the manager. Prerequisite are inFile formats that contain gps time information (las, sdw, fwf). Trajectory files have to contain the following information: x, y, z, gps time, roll, pitch and yaw
-tFormattrajectory file format [auto, trj, btrj, odm, <opals format def. xml file>]
Type: opals::Vector<opals::String>
Remarks: estimable
In case of multiple trajectory files with present tFormat parameter, the number of tFormat and trjFiles entries have to match or only a single tFormat entry is specified (=all trajectory files have the same format).
Estimation rule: auto. The file content is used to recognise the file format.
-outFileoutput manager file name
Type: opals::Path
Remarks: estimable
Estimation rule: The current path and the name (body) of the first input file are used as file name basis
-filtermodify the input using a (tree of) filter(s)
Type: opals::String
Remarks: optional
A filter string in EBNF syntax as described in section 'Filter syntax' can be passed to restrict the input data set (e.g. to consider last echoes only)
-trafogeometrically transform the data during import
Type: opals::TrafPars3dAffine
Remarks: optional
An eventually passed affine 3-d transformation is applied during data import. The passed value is converted internally into a single filter tree for parameters 'filter' and 'trafo'. If both are to be used, make sure to mark the position where in the filter's string 'trafo' shall be introduced, using curly brackets {}
-storeOrderstore natural order of data
Type: bool
Remarks: optional
If set, the import order of the data is preserved, which allows processing and exporting data in the original order (Not implemented yet).
-tilePointCountaverage tile point count
Type: int
Remarks: default=200000
After importing a specific number of points the odm estimates the point density of the data. Based on the estimated density and the specified tilePointCount a tile size is computed. After that, the ODM administers the current and all subsequent point data in a tiling strategy.
-tileSizetile size length of spatial index
Type: double
Remarks: optional
The tile size length (the ODM uses squared tiles) can only be set for new ODMs or ODMs that are not in tiling mode yet. This parameter should be use with caution, since inappropriate tile sizes can lead to memory problems or slow performance in later processing steps. On average, 100,000 to 200,000 points should be stored within one tile (check index statistics after import).
-storeBeamInfobeam information to attach while import with trajectory
Type: opals::Vector<opals::BeamInfo>
Remarks: default=BeamVector
If trajectory files are provided, the module computes and attaches the specified beam information to each imported point.
-mergePolygonmerge polygon mode
Type: opals::MergePolygonMode
Remarks: default=safe

Possible values:  
  safe ... incremental merge of polygon parts (slow but robust)
  fast ... build polygon object at once (fast but fails in case of topological incorrect polygon parts)

Per default the datamanager imports nested polygons in a safe and incremental manner such that polygon parts that are topological incorrected, still result in valid polygon objects. However, this algorithm can be very slow in case of complex nested polygon parts. If secured that only topological correct polygons are imported, using 'fast' mode can result in a huge import performance gain.

Examples

All data used in the following examples are located in the $OPALS_ROOT/demo/ directory of the OPALS distribution. To import the LAS file strip31.laz into the ODM, simply type:

opalsImport -inFile strip31.laz

Since no output file is specified, the name of the corresponding ODM file is estimated from the input file name and the file ...

strip31.odm

... constituting the ODM is created.

To add more than one strip to a single ODM file, multiple input files can be specified in a single command:

opalsImport -inFile "strip31.laz" "strip21.laz" "strip11.laz" -outFile allStrips.odm

Please note that quotes are only necessary if the file names contain blanks, which is not the case in this example. For convenience multiple files can be specified using wildcard characters (*,?) as well:

opalsImport -inFile strip?1.laz -outFile allStrips.odm

The same result could have been achieved by calling Module Import multiple times and by appending the data to a single ODM.

opalsImport -inFile strip31.laz -outFile allStrips.odm
opalsImport -inFile allStrips.odm strip21.laz -outFile allStrips.odm
opalsImport -inFile allStrips.odm strip11.laz -outFile allStrips.odm

In case, the automatic recognition of the input files' data format fails, it is nonetheless possible to import the files, if the proper input data format is specified. If multiple input files are used, then the number of files must match the number of input formats. This also applies for appending additional data files to an existing ODM. Thus, the very same example as above can be written as:

opalsImport -inFile strip?1.laz -iformat las -outFile allStrips.odm

Data in generic ASCII (or binary) format can be imported by providing a OPALS Generic Format file in XML format. The file fullwave.fwf, for instance, contains eight columns (x, y, z, time, amplitude, echo width, echo number, echo qualifier) per line (=data set).

24820.774 311160.141 322.452 314358.431470 16 2.606 2 0
24820.035 311161.159 319.200 314358.431470 69 1.736 3 1
24820.599 311160.576 322.863 314358.431485 18 3.541 1 2
...

With the following format definition file ...

<?xml version="1.0" encoding="ISO-8859-1" ?>
<?xml-stylesheet type="text/xsl" href="opalsFormatDefinitionStyle.xsl" ?>
<!-- OPALS Format Definition Section -->
<opalsFormatDefinition>
<description>XYZ ASCII Format</description>
<!-- Example for a simple XYZ ASCII Format Definition for Import and Export -->
<ascii>
<commentInitiator val="#" />
<data>
<entry val="x" />
<entry val="y" />
<entry val="z" />
<entry val="GPSTime" />
<entry val="Amplitude" />
<entry val="EchoWidth" />
<entry val="EchoNumber" />
<entry val="_echoQualifier" externalType="byte" internalType="byte" />
</data>
</ascii>
</opalsFormatDefinition>

... stored as myFormatDef.xml, the file can be imported as follows:

opalsImport -inFile fullwave.fwf -iFormat myFormatDef.xml

In fact, the file satisfies the I.P.F. full waveform format specification, and thus there is no need for a user defined format definition file in this place. But instead, sometimes it is desired to read only specific parts of a data file. Filters can be used to read, e.g. only the first echoes or echoes with amplitudes above a certain threshold.

opalsImport -inFile fullwave.fwf -iFormat fwf -filter Echo[First]
opalsImport -inFile fullwave.fwf -iFormat fwf -filter "Echo[First] and Generic[Amplitude>75]"

Please mind that quoting the filter string is necessary in the latter case due to the embedded blanks and special characters. It is recommended always to quote complex filter strings. For more detailed information about selective data import please refer to the Filter manual.

Author
J. Otepka, G. Mandlburger
Date
29.05.2016

References

Otepka, J., Briese, C. and Nothegger, C., 2006. First steps to a topographic information system of the next generation. In: Symposium of ISPRS Commission IV - Geo Spatial Databases for Sustainable Development, Goa, India.

Mandlburger, G., Otepka, J., Karel, W., Wagner. W. and Pfeifer, N., 2009. Orientation and Processing of Airborne Laser Scanning data (OPALS) - concept and first results of a comprehensive ALS software. ISPRS Workshop Laserscanning 2009, Paris, France.