Distributed Installation

HxMap could be installed from a single workstation to high end cluster environment with hundreds of nodes for high volume production. For production jobs covering large areas or multiple sensor flights, a cluster setup is strongly recommended to distribute computationally intensive jobs onto multiple processing nodes.

The HTCondor system has been chosen as the distributed computing platform for HxMap. HTCondor is a sophisticated and unique distributed job scheduler developed by the Center for High Throughput Computing at the University of Wisconsin-Madison. It provides a high throughput computing environment that delivers large amounts of computational power over a long period of time even in case of failures.

As a workload management system for compute-intensive batch jobs HTCondor provides:

• a job queuing mechanism,

• very flexible scheduling policy

• sophisticated priority scheme (by job type, submitter, cluster type etc.)

• resource monitoring and management

• inbuilt fault tolerances against machine, network, storage and job failures

• ability to effectively harness wasted CPU power from idle desktop workstations

HTCondor has more extensive features that are well documented in the manual and appear in many publications from the research team. 

The following chapters will describe how to setup and configure a distributed processing cluster for HxMap.