Ingest of raw data

The data processing model for HxMap is divided into two aisles: Raw Space and Processed Space. The gateway to move from the raw processing space to processed space is the Ingest step, which is responsible to convert all sensor specific data into a generic HxMap representation of metadata and computational model. Ingest georeferences and prepares raw data for quality control and further processing.

The main differentiator is that processes that run before Ingest could take advantage of the different personalities of the sensors. However, once Ingest is finished, all imaging sensor data will be treated as raster and LiDAR sensors will be treated as point cloud in most of the downstream processing. Leica sensors are composed of multiple sensor components that build a sensor system, designed for different use cases like wide area, city, corridor mapping etc. Available Ingest options depend on the individual sensor types that make up the sensor systems and the downstream processing required. 

Overview on Sensor Systems and Sensors:

Selecting data for ingest

HxMap will ingest all selected data based on the selected ingest template. To perform raw data ingest, select the data from the data source tree or through the main map window.

Ingest requires sufficient imagery data to calculate atmospheric corrections and perform normalization. It is therefore recommended to pick a complete mission rather than individual strips/images.

For LiDAR data a defined AOI can be used to reduce ingested data to only a selected area. Be sure to define the AOI first and then select the defined AOI together with the LiDAR takes before starting the ingest process.

​Click the 'Ingest' icon on the menu bar.

The 'Ingest' wizard pops up, where on several tabs all relevant settings for the Ingest process are collected via user input.

Product Settings tab

Configure settings for the selected image and/or lidar takes to ingest raw sensor data for the workflow.

Ingest always includes georeferencing with the sensor calibration, installation parameters and trajectory

• Update Geoereferencing Only: recompute image EOP of already ingested data with a new trajectory solution. Only applicable for image takes.

• Georeference and…: compute georeference for image and lidar data during ingest.

Import QC Status: import the QC Status marked by FlightPro during acquisition, and any QC markings made by the operator on the raw session in Workflow Manager.

Remaining Ingest settings are dependent on the data type(s) being ingested. User is only shown options relevant for selected data, whether image or lidar.

• L0 Image / LiDAR Product: choose the L0 format appropriate to the workflow being processed. The table below provides more detail on the formats available.

• Image specific settings:

  • Reduce image takes to minimum overlap: if checked, ingest will thin out selected data while maintaining the specified minimum forward overlap for nadir imagery (computation does not consider oblique views)

  • Create statistics: compute Base and Extended radiometry corrections to be applied in Product Generation.

    • without Create statistics, only Calibrated radiometry is available for displaying images in QC Perspective.

    • Create statistics for Nadir images gives the same result at Ingest and in Blockwide Radiometry

    • Create statistics for Nadir + Oblique blocks should be computed in Blockwide Radiometry, as the full block coverage can better support the Left and Right oblique camera statistics calculation

  • Run radiometric normalization: compute Balancing radiometry correction to be applied in Product Generation.

    • without radiometric normalization, only Dark Pixel and Dark Pixel + Gradient radiometry is available for displaying images in QC Perspective.

    • Run radiometric normalization should computed in Blockwide Radiometry, as the full block coverage can be normalized for brightness and contrast

  • Create Minifications – required for Create statistics, Radiometric normalization, and typical image workflows.

  • Create additional products: Allows to create additional image products directly after ingest step completes

    • NOTE: this option is disabled if a JPG ingest template is selected as this ingest is not intended for creating products.

    • NOTE: because of combining ingest with product generation it is currently not possible to define/attach a radiometric profile as this is normally done in between these two steps

    • Additional product settings are listed according to the user’s defined product templates

• LiDAR specific settings:

  • Trim to full density: Classify the strip ends of circular scanner (Hyperion, HRQLSII) to class 108. Circular scan geometry results that strip ends have partial density until both scan directions are overlapping on ground.

  • Keep all LiDAR points: write filtered noise and strip end points into the Ingest point cloud.

    • NOTE: The “Keep all LiDAR points” option must not be checked if a Calibration Ingest is intended for LiDAR data.

Click Next to continue to Calibration selection.

Image L0 Formats

For all ORTHO/REMOTESENSING/TIF products, HxMap creates per default 16bit TIFF RGB(N) imagery. If selected, the Ingest process also runs a Multiframe process, where atmospheric influences as well as a radiometric normalization of a session are determined. For all image data that is intended to be used throughout the upcoming HxMap workflow these L0 product types should be selected.

For all ORTHO-JPG/ xxJPGxx products, HxMap creates per default 8bit JPG RGB imagery with an initial radiometric enhancement applied (see also Project specification settings Tab). The Ingest process will skip the Multiframe process. These product templates are geared for image data that is intended to be used in a 3rd party workflow and are not fully supported throughout the downstream HxMap workflows.

For all QUICK products, the Ingest process creates 1:16 RGB Thumbnail imagery (either Tiff or JPG) and will skip the Multiframe process. Image data is intended to be used for QuickQC in 3rd party applications and is also not intended to be used further in HxMap workflows.

LiDAR L0 formats

Calibration tab

Verify the preselected sensor calibration for the session you wish to ingest and modify, if required:

• Latest calibration version found in Global calibration repository is shown first. If no calibration is found in the Global repository, flight calibration will be shown. Changing the calibration version for any individual image or lidar sensor will update the calibration version for all image and lidar sensors in the system

• Project Calibration is activated as an option when lidar data is selected for ingest.

• NOTE: Leica DMC III ingest requires that a calibration to be loaded to the Global repository..

Click Next to continue to LiDAR Atmospheric Settings.

LiDAR Atmospheric Settings

This tab is available only in case there is LiDAR data selected for Ingest. It requires to specify environmental parameter of the flight necessary for a proper LiDAR post-processing

• Time: enter the flight date and a time (UTC) when the flight took place. You may enter multiple times for an individual flight too (start/ end time) if it's a large flight project over a couple of hours. In case Ingest is started for a session with multiple flight projects, make sure you enter flight date and time for each of the flight projects.

• Temperature: enter a temperature measured in the area of the flight project during capture time (see next point).

• Above sea level height: enter the height where the temperature was measured

• Mean sea level pressure: enter the pressure in kPa measured in the area of the flight during capture time

Click Next to continue to Project Specification Settings

Project Specification Settings

The last part of the wizard is for any project related settings.

• Session name: populated automatically according the active Session Naming setting.

• Output path: location to write the Ingested session data

• Trajectory: GNSS/INS trajectory used for georeferencing. Two trajectory formats are supported

  • RNV: Real-time navigation format from FlightPro

  • SOL: Post processed trajectory solution from Inertial Explorer Note that . (RNV, SOL). In case LiDAR data is ingested, the user shall in any case provide a post-processed trajectory (*.sol)

RNV

• RNV should only be used to direct georeference for basic sensor function QC. Do not use RNV for production Ingest.

• FlightPro must have the correct GNSS reference lever arm for the aircraft the sensor is installed in to obtain a reasonable result.

SOL

Accurate direct georeferencing during HxMap ingest depends on a correct lever arm value between the GNSS Antenna and PAV reference location. Verify the GNSS reference lever arm used in trajectory processing is correct for the aircraft.

• Radiometric profile: this option is only available if the JPG ingest template for RCD30 (ORHO-JPG) or MFC150 (JPG) has been selected. If no radiometric profile is provided, a default radiometric enhancement is applied to the imagery.

• File Naming Preview: For any sensor, the user can pick a pre-defined file naming template (see also section Naming convention). The resulting file names are shown below.

• Additional Product settings: activated when Additional Products are selected in the first tab, using the same parameters defined in as shown during product generation.

Click 'Finish' to start the process as described Job Handling .

Once the process run completes, the ingested session data is ready to load into HxMap QC Perspective for Data QC .

Note on JPG ingest options

The JPG ingest option is made available to enable image processing in external workflows. It should not be used for processing in HxMap.

Only calibrated radiometry is available for JPG ingest. To create JPG format Aerial products with full radiometry correction use one of the ‘TIF’ or ‘ORTHO' L0 templates.

JPG ingest requires a map down of the sensor dynamic range to 8-bit range. Ingest will calculate an auto dynamic range adjustment if no radiometric profile is given. Its recommended to use a radiometric profile for more control over the result, by the following workflow.

• Load a raw session into HxMap and run Ingest either for a few selected takes using a regular TIF ingest template

• The goal of that first ingest run is to produce 16bit TIFF Imagery, which is the required input for the definition of a radiometric profile.

• Load the processed session into HxMap's QC perspective, load a take into the image viewer and start the Image Adjustment tool to create a radiometric profile. Save it.

• Re-open HxMap and load the same raw session. Now run Ingest for the full project using the JPG ingest template for your sensor. Pick the previously defined radiometric profile in the project specification tab and start the process.

• Resulting imagery can be directly used in a third-party workflow.

Ingest Output

Once Ingest is finished all the files that make up a recording session that are used in the workflow are persisted in the output directory (some files are sensor specific):