Abstract

We introduce the S-EO dataset: a large-scale, high-resolution dataset designed to advance geometry-aware shadow detection. Collected from diverse public-domain sources, including challenge datasets and government providers such as USGS, our dataset comprises 702 georeferenced tiles across the USA, each covering 500 × 500 meters. Each tile includes multi-date, multi-angle WorldView-3 pansharpened RGB images, panchromatic images, and a ground-truth DSM of the area obtained from LiDAR scans. For each image, we provide a shadow mask derived from geometry and sun position, a vegetation mask based on the NDVI index, and a bundle-adjusted RPC model. With approximately 20,000 images, the S-EO dataset establishes a new public resource for shadow detection in remote sensing imagery and its applications to 3D reconstruction. To demonstrate the dataset’s impact, we train and evaluate a shadow detector, showcasing its ability to generalize even to aerial images. Finally, we extend EO-NeRF — a state-of-the-art NeRF approach for satellite imagery — to leverage our shadow predictions for improved 3D reconstructions.

The dataset consists of 702 georeferenced tiles of 500 × 500 m each, across the cities of Jacksonville, Omaha, and San Diego. For each tile, we provide multiple data products:

• Panchromatic (PAN) WorldView-3 images (30 cm resolution)
• Pansharpened RGB images (30 cm resolution)
• Bundle-adjusted RPC models for all PAN and pansharpened images
• Digital Surface Models (DSMs) obtained from LiDAR data (USGS) with a 50 cm grid resolution — two versions using min and max altitude aggregation
• Vegetation masks based on the Normalized Difference Vegetation Index (NDVI)
• Shadow masks — two versions corresponding to min and max DSMs

With multiple images per tile, captured on different dates and from various angles, we obtain a total of 19,162 images and masks of approximately 1500 × 1500 pixels. Note that the exact image size varies depending on the viewing angle. The S-EO dataset primarily functions as a large-scale training resource rather than a validation set, due to the inherent noise in its annotations.

We train a shadow detector on the S-EO dataset using both shadow masks and auxiliary masks using a U-Net architecture. To support generalization, we apply data augmentation strategies from prior work and initialize models with pretrained weights from ground-level imagery. For supervision, we use focal loss and apply masking strategies to exclude uncertain and vegetated regions from the loss computation. With minimal fine-tuning, our model outperforms existing methods on shadow segmentation in remote sensing imagery.