Ingest of raw data | Airborne Solutions Software Documentation

Data captured with the sensor system is handled as a raw session within HxMap. Ingest is the process whereby raw sessions are converted into processed session for use in quality control, data adjustment and product generation.

Raw sessions allow basic review of the captured extent and provide low resolution data (thumbnails) for determining QC of clouds and other environmental factors with direct georeferencing based on real time trajectories. Processed sessions provide access to the full resolution image and lidar data with direct georefencing based on a final post-processed trajectory.

Available Ingest options depend on the individual sensor types that make up the sensor systems and the downstream processing required.

Sensor System	Image Component	LiDAR Component
Leica CityMapper Hybrid system combining 1 nadir RGBN and 4 oblique RGB medium format cameras as well as different variants of the Hyperion LiDAR unit.	RCD30 CH82 (1x) RCD30 CH81m (4x)	Hyperion2
Leica CityMapper-2 Hybrid system combining 1 nadir RGB, 1 nadir NIR and 4 oblique RGB medium format cameras as well as the Hyperion 2+ LiDAR unit.	MFC150 (5x) MFC150-NIR (1x)	Hyperion2+
Leica CoastalMapper High performance hybrid topo-bathymetric system providing highest collection efficiency for airborne lidar bathymetric survey available in the market	MFC250 (1x) MFC150-NIR (1x)	Hyperion3+ (topo) Theia (bathymetric)
Leica ContentMapper Large format imagery system designed for large-area and high-altitude data acquisition of RGBN imagery, especially for orthoimage production.	MFC150 (3x) MFC150-NIR (2x)	none
Leica CountryMapper Hybrid system combining large format nadir RGBN imaging with latest generation Hyperion LiDAR unit.	MFC150 (3x) MFC150-NIR (2x)	Hyperion3+
Leica DMC-4 Large format imagery system designed for large-area and high-altitude data acquisition of RGBN imagery	MFC150 (3x) MFC150-NIR (2x)	none
Leica TerrainMapper Linear mode LiDAR system with an optional medium-format camera.	Webcam (1x) or RCD30 CH82 (1x)	Hyperion2 or Hyperion2+
Leica TerrainMapper-2 Linear mode LiDAR system with medium format RGBN imaging.	MFC150 (1x) MFC150-NIR (1x)	Hyperion2+
Leica TerrainMapper-3 Linear mode LiDAR system with medium format imaging. Flexible LiDAR scan patterns allow to tailor data collection to a broad variety of land cover.	MFC250 (1x) MFC150- NIR (1x)	Hyperion3+
Leica SPL100 Single Photon LiDAR (SPL) system with or without imaging sensor.	SPL100: RCD30 CH82 (1x) SPL100HA: none	SPL100: HRQLSII SPL100HA: HRQLSIIHA
Leica DMC III Large format imagery system collecting RGBN imagery.	DMC III PAN and MSRGBN	none
Leica RCD30 Medium format imagery system collecting RGBN or RGB imagery.	RCD30 CH82 (RGBN) or RCD30 CH81 (RGB)	none

In HxMap 5.0 CoastalMapper bathy lidar is supported for ingest and point cloud viewing in Lidar Viewer. Support in other point cloud workflow features will be provided in future HxMap releases. Bathy point clouds formatted for Lidar Survey Studio are available in the session’s \bathyIntermediate directory.

Selecting data for ingest

To perform raw data ingest, select the data from the data source tree or through the main map window.

It is necessary to load a valid calibration for each system into the repository prior to working with its data in HxMap. The repository should configured in a centralized location for Local and Cluster processing. See Application Settings for more details.

Ingest requires sufficient imagery data to calculate atmospheric corrections and perform normalization. It is therefore recommended to pick a complete mission rather than individual strips/images.

For LiDAR data a defined AOI can be used to reduce ingested data to only a selected area. Be sure to define the AOI first and then select the defined AOI together with the LiDAR takes before starting the ingest process.

Click the 'Ingest' icon on the menu bar.

The 'Ingest' wizard pops up, where on several tabs all relevant settings for the Ingest process are collected via user input.

Product Settings tab

First configure Ingest and L0 Product Settings for the selected image and/or lidar takes to ingest raw sensor data for the workflow.

Ingest always includes georeferencing with the sensor calibration, installation parameters and trajectory.

Update Geoereferencing Only: recompute image EOP of already ingested data with a new trajectory solution.
Georeference and…: compute georeference for image and lidar data during ingest.

Import QC Status: import the QC Status marked by FlightPro during acquisition, and any QC markings made by the operator on the raw session in Workflow Manager.

For each sensor listed in the table on the left, select and configure the L0 Product and settings on the right.

Update Georefencing is only applicable to image captures. In case of a change in trajectory or calibration for lidar captures, please ingest the data again from the raw session.

Topo lidar settings (Hyperion, HRQLSII)

L0 Product: See table below for details.
Trim to full density: Classify the strip ends of circular scanner to class 108.
- Use with circle scan pattern where strip ends have partial density until both scan directions are overlapping on ground.
- Works best on flat and rolling terrain, and should not be used in steep terrain.
Keep all LiDAR points: write filtered noise and strip end points into the Ingest point cloud.
- Only use with HSPC L0 templates

Image settings

L0 Product: See table below for details.
Reduce image takes to minimum overlap (deprecated): thin out selected data while maintaining the specified minimum forward overlap for nadir imagery (computation does not consider oblique views)

Create statistics: compute Base and Extended radiometry corrections to be applied in Product Generation.
- Create statistics for Nadir images gives the same result at Ingest and in Blockwide Radiometry
- Create statistics for Nadir + Oblique blocks should be computed in Blockwide Radiometry, as the full block coverage can better support the Left and Right oblique camera statistics calculation
Water Polygons: optionally supply a water mask as polygons to override the default classification of water from create statistics
Run radiometric normalization: compute Balancing radiometry correction to be applied in Product Generation.
- without radiometric normalization, only Dark Pixel and Dark Pixel + Gradient radiometry is available for displaying images in QC Perspective.
- Run radiometric normalization should computed in Blockwide Radiometry, as the full block coverage can be normalized for brightness and contrast
Create Minifications – required for Create statistics, Radiometric normalization, and typical image workflows.
Create additional products: Allows to create additional image products directly after ingest step completes
- NOTE: this option is disabled if a JPG ingest template is selected as this ingest is not intended for creating products.
- NOTE: because of combining ingest with product generation it is currently not possible to define/attach a radiometric profile as this is normally done in between these two steps
- Additional product settings are listed according to the user’s defined product templates

Reduce image takes to minimum overlap has been deprecated.

If Create statistics is skipped during ingest, only Calibrated radiometry is available for displaying images in QC Perspective.

Bathy lidar settings

L0 Product: see table below for details.
4X: increases the point cloud density during processing (additional processing time)
Deep Channel: process returns from the sensor’s deep channel waveforms (additional processing time)

Click Next to continue to Calibration selection.

Appendix - L0 Product types

Image sensors

Component Sensor	L0 Product	Description
MFC250	COMPRESSED_TIF	radiometrically calibrated 16-bit TIFF with lossless JPEG-LS compression. Use for all HxMap workflows.
	UNCOMPRESSED_TIF	radiometrically calibrated 16-bit TIFF. Intended for HxMap workflows and viewing in 3rd party software.
	JPG	8-bit JPEG RGB. Intended for 3rd party applications. Not supported in HxMap workflows after ingest.
MFC150	COMPRESSED_TIF	radiometrically calibrated 16-bit TIFF with lossless JPEG-LS compression. Use for all HxMap workflows.
	UNCOMPRESSED_TIF	radiometrically calibrated 16-bit TIFF. Intended for HxMap workflows and viewing in 3rd party software
	JPG	8-bit JPEG RGB. Intended for 3rd party applications. Not supported in HxMap workflows after ingest.
DMC III	MSRGBN-PAN	radiometrically calibrated 16-bit TIFF (low resolution RGBN and high-resolution PAN)
RCD30	COMPRESSED_ORTHO	radiometrically calibrated 16-bit TIFF with lossless JPEG-LS compression. Use for all HxMap workflows.
	UNCOMPRESSED_ORTHO	radiometrically calibrated 16-bit TIFF. Intended for HxMap workflows and viewing in 3rd party software.
	ORTHO_JPG	8-bit JPEG RGB. Intended for 3rd party applications. Not supported in HxMap workflows after ingest.
	QUICK	radiometrically calibrated 16-bit TIFF at 1:16 resolution. Use for QuickQC in HxMap or 3rd party applications. Not supported in HxMap workflows after ingest.
	QUICK_JPEG	8-bit JPEG RGB at 1:16 resolution. Use for QuickQC in HxMap or 3rd party applications. Not supported in HxMap workflows after ingest.
	REMOTESENSING	radiometrically calibrated 16-bit TIFF without saturation correction. Compatible with HxMap and 3rd party remote sensing workflows.

ORTHO/REMOTESENSING/TIF products create 16bit TIFF RGB(N) imagery. Image data that is intended to be used throughout the upcoming HxMap workflow should be ingested with these L0 product types.

ORTHO-JPG/ xxJPGxx products create 8bit JPG RGB imagery with an initial radiometric enhancement applied (see also Project specification settings Tab). Create Statistics and Radiometric Normalization are skipped for these products. They are intended to be used in a 3rd party workflow directly after ingest and are not supported in the downstream HxMap workflows.

QUICK products create 1:16 RGB Thumbnail imagery (either TIFF or JPG). Create Statistics and Radiometric Normalization are skipped for these products. Image data is intended to be used for QuickQC in 3rd party applications and is not intended to be used further in HxMap workflows.

Topo LiDAR sensors

Component Sensor	L0 Product	Description
HYPERION3+ HYPERION3 HYPERION2 HRQLSII	HSPC	creates LiDAR point cloud data for the sensor. Use for all HxMap workflows.
	WFA	creates LiDAR point cloud data for the sensor with waveform attributes sampled by the sensor. Required to generate point cloud products with waveform attributes. (Hyperion3+ only)
	WFD	creates LiDAR point cloud data for the sensor with waveform samples recorded by the sensor. Required to generate point cloud products with waveform data packages. (Hyperion2 and 3 types only)
	CALIBRATION	creates LiDAR point cloud data and metadata required to Run LiDAR Calibration Tool in HxMap. Not used for any purpose other than input to LiDAR Calibration Tool.

Availability of WFA and WFD L0 templates in the raw data for ingest depends on corresponding configuration of flight plan settings in MissionPro to record the raw data with waveform attributes and waveform data packages.

Noise filtering for Hyperion ingest assumes a minimum range of 300m. In case of a low above ground level flight which would reduce expected range less than 300m, set the HxMap.ini [HyperionMPiA] low_range parameter to a value less than 300m.

Ingest of Waveform Attributes allows to generate the following attributes for each return and store them as LAS 1.4 Extra Bytes VLR during point cloud product generation.

Amplitude: Raw DN of the waveform peak
Pulse Assymetry: ratio of right-half area to left-half area of the waveform
Pulse Deviation: ratio of scaled “standard” received pulse area to sum of left half and right half return pulse area of waveform
Pulse Width: sum of the left-half and right half area of the waveform
Reflectance: Radiometrically calibrated DN of the waveform peak

Bathy LiDAR sensors

Component Sensor	L0 Product	Description
THEIA	CALIBRATION	creates LiDAR point cloud with no water surface correction
	INLANDWATER	creates LiDAR point cloud with a single water surface
	OPENOCEAN	creates LiDAR point cloud with one water surface per scan direction

Additional processing settings for HxMap Bathy ingest are also included in the bathy L0 templates, stored in the HxMap installation folder under etc\ingest\l0producttemplates.

Calibration tab

The ingest wizard will automatically select the most appropriate system calibration version in the global repository for the dataset, based on criteria described below. Optionally select a project calibration to use a recent misalignment update which hasn’t yet been installed to the global repository. The Sensor Description button shows the calibration creation date for individual sensors in the system.

Calibrations available in the global repository are filtered based on following characteristics
- Validity start and end date: the selected session start date must fall within the calibration’s validity start and end date.
- Installed sensors: the selected session’s system’s installed sensors must match the calibration’s installed sensors. Currently this applies to camera heads with exchangeable lenses.

Update Misalignment option is available when selecting a processed session is selected and Update Georeferencing is selected on the first page of the Ingest wizard. Update Georeferencing is only applied to image captures.

Click Next to continue to LiDAR Atmospheric Settings.

LiDAR Atmospheric Settings

LiDAR measurements will be adjusted for temperature and pressure during Ingest. Enter the temperature and pressure observations from the aerial survey mission. Observations entered by the sensor operator in FlightPro will automatically be populated in the dialog.

Temperature: enter the date and time, temperature, and height above sea level for each temperature observation.
- Multiple temperature observations may be added to reflect large variation in temperature during the session
Mean sea level pressure: enter the QNH pressure in kilopascals measured in the area of the flight during capture time.
- Only one pressure observation is needed for the session

In case an error is made in data entry, use the Remove button to remove the incorrect entries, then add a corrected entry. Click Next to continue to Project Specification Settings.

Project Specification Settings

The last part of the wizard is for any project related settings.

Session: populated automatically according the active Session Naming settings
Output path: location to write the Ingested session data
Minification path: optionally shown if Applications preferences are set to allow external minfications. Location to write external minifications required by the workflow for the full resolution images.

Trajectory: GNSS/INS trajectory used for georeferencing. Two trajectory formats are supported
- RNV: Real-time navigation format from FlightPro
- SOL: Post processed trajectory solution from Inertial Explorer Note that . (RNV, SOL). In case LiDAR data is ingested, the user shall in any case provide a post-processed trajectory (*.sol)
Radiometric profile: this option is only available if the JPG ingest template for RCD30 (ORHO-JPG) or MFC150 (JPG) has been selected. If no radiometric profile is provided, a default radiometric enhancement is applied to the imagery.
File Naming Preview: For any sensor, the user can pick a pre-defined file naming template (see also section Naming convention). The resulting file names are shown below.
Additional Product settings: activated when Additional Products are selected in the first tab, using the same parameters defined in Product Generation Project Specification settings.

Avoid use of the character “$” or “%” in user defined session name, as these can interfere with logging functions.

Minifications:

The moire filter for MFC images in Quality Control perspective requires images with embedded minifications for proper function. Do not use external minifications if you plan to apply the moire filter.

Trajectory:

Real time navigation (.rnv) trajectory should only be used when absolute accuracy is not important e.g. for quick QC or rapid response mapping.
For a useful RNV file, FlightPro must have the correct GNSS reference lever arms for the aircraft the sensor is installed in to obtain a reasonable result.
Post processed solution (.sol) trajectory from Inertial Explorer is required to achieve data sheet accuracy from direct georeferencing. Correct GNSS reference lever arms must be entered in Inertial Explorer during loosely-coupled or tightly-coupled processing to the data sheet accuracy.

Click 'Finish' to start the process as described Job Handling .

Once the process run completes, the ingested session data is ready to load into HxMap QC Perspective for Data QC .

Note on JPG ingest options

The JPG ingest option is made available to enable image processing in external workflows. It should not be used for processing in HxMap.

Only calibrated radiometry is available for JPG ingest. To create JPG format Aerial products with full radiometry correction use one of the ‘TIF’ or ‘ORTHO' L0 templates.

JPG ingest requires a map down of the sensor dynamic range to 8-bit range. Ingest will calculate an auto dynamic range adjustment if no radiometric profile is given. Its recommended to use a radiometric profile for more control over the result, by the following workflow.

Load a raw session into HxMap and run Ingest either for a few selected takes using a regular TIF ingest template
The goal of that first ingest run is to produce 16bit TIFF Imagery, which is the required input for the definition of a radiometric profile.
Load the processed session into HxMap's QC perspective, load a take into the image viewer and start the Image Adjustment tool to create a radiometric profile. Save it.
Re-open HxMap and load the same raw session. Now run Ingest for the full project using the JPG ingest template for your sensor. Pick the previously defined radiometric profile in the project specification tab and start the process.
Resulting imagery can be directly used in a third-party workflow.

Ingest Output

Once Ingest is finished all the files that make up a recording session that are used in the workflow are persisted in the output directory (some files are sensor specific):

*.hexrs	Recording session
"Flightproject"	Ingest results Flight project folder contents are organized by flightplan and flightline. Each flighline folder holds the L0 image and lidar data and metadata created during ingest. Lidar data are stored in \pc subdirectories. Waveform data packets are stored in \Waveform subdirectories.
*.hexstp	Strip files per \flightproject\flightplan\flightline, Metadata per strip
*.hexlik	Link files per \flightproject\flightplan\flightline, Metadata per strip
*.hexrad	Radiometric corrections per \flightproject\flightplan\flightline, Dark pixel, Gradient and Radiometric adjustment correction per strip (depending on the sensor)
*.eop	Exterior orientation information per \flightproject\flightplan\flightline, exterior orientation information for image data available in the flightline
*_Footprints.shp	Footprint information per \flightproject\flightplan\flightline, footprint polygons for data available in the flightline
/cam/<SystemType>	Original System calibration (RCD30, CM, etc) calibration used to ingest the data, stored for linage only
/cam/<SystemType>DF	Digital Frame calibration for Imaging sensors LiDAR sensor calibration Pinhole camera model that will be used in Triangulation. Lidar sensor calibration files used to georeference point clouds.
/gps-imu	GPS/IMU Real time or post-processed trajectory solution used to create EOP
/DSNU	RCD 30 DSNU images Dark Signal Non-Uniformity (DSNU) images for dynamic dark signal correction
/intermediate	Temporary raw processing results Sensor specific temporary files that are required only at ingest run (e.g, logfiles, progress files, etc)
/logs	Ingest logfiles Logfiles created by Ingest incl. Ingest subtasks