Automated historical data retrieval system for the Laboratorio Nacional de Observación de la Tierra
Main Article Content
Abstract
After eight years of operation, the storage system at the Laboratorio Nacional de Observación de la Tierra reached a historical satellite data volume approaching one petabyte. To facilitate querying and data retrieval from this repository, a comprehensive infrastructure comprising three distributed components was designed and implemented: a web interface enabling users to submit detailed data requests, an asynchronous service for data extraction, and a provisioning service that creates temporary virtual accounts for downloading. The system was tested in real-world projects requiring a fine-grained selection of dates and times, successfully completing the retrieval process for a substantial data volume—890,000 files and 7.02 terabytes. This was achieved efficiently and with almost no human intervention, demonstrating the system's operational effectiveness and yielding valuable insights for future enhancements.
Downloads
Article Details

This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License.
Este trabajo tiene la licencia CC BY-NC-ND 4.0
References
Abernathey, R. P., Blackmon-Luca, C. C., Crone, T. J., Henderson, N., Lepore, C., Augspurger, T., Banihirwe, A., Hamman, J. J., Gentemann, C. L., McCaie, T. A., Robinson, N. H., y Signell, R. P. (2021). Cloud-native repositories for big scientific data. Computing in Science & Engineering, 23(2), 26–35. https://doi.org/10.1109/MCSE.2021.3059437 DOI: https://doi.org/10.1109/MCSE.2021.3059437
Aguilar Sierra, A. (2020). Sistema de almacenamiento para el Laboratorio Nacional de Observación de la Tierra [tesis de especialidad, Universidad Nacional Autónoma de México]. TESIUNAM. https://tesiunam.dgb.unam.mx/cgi-bin/koha/opac-detail.pl?biblionumber=802512
Aguilar Sierra, A. (2025). Clúster de software libre del Laboratorio Nacional de Observación de la Tierra. Cuadernos Técnicos Universitarios de la DGTIC, 3(2), 24–34. https://doi.org/10.22201/dgtic.30618096e.2025.3.2.107 DOI: https://doi.org/10.22201/dgtic.30618096e.2025.3.2.107
Ansari, S., Del Greco, S., Kearns, E., Brown, O., Wilkins, S., Ramamurthy, M., Weber, J., May, R., Sundwall, J., Layton, J., Gold, A., Pasch, A., y Lakshmanan, V. (2018). Unlocking the potential of NEXRAD data through NOAA’s Big Data Partnership. Bulletin of the American Meteorological Society, 99(1), 189–204. https://doi.org/10.1175/BAMS-D-16-0021.1 DOI: https://doi.org/10.1175/BAMS-D-16-0021.1
Chard, K., Dart, E., Foster, I., Shifflett, D., Tuecke, S., y Williams, J. (2018). The Modern Research Data Portal: A design pattern for networked, data-intensive science. PeerJ Computer Science, 4, e144. https://doi.org/10.7717/peerj-cs.144 DOI: https://doi.org/10.7717/peerj-cs.144
Gomes, V. C. F., Queiroz, G. R., y Ferreira, K. R. (2020). An overview of platforms for big Earth observation data management and analysis. Remote Sensing, 12(8), 1253. https://doi.org/10.3390/rs12081253 DOI: https://doi.org/10.3390/rs12081253
Knapp, K. R., y Wilkins, S. L. (2018). Gridded Satellite (GridSat) GOES and CONUS data. Earth System Science Data, 10(3), 1417–1425. https://doi.org/10.5194/essd-10-1417-2018 DOI: https://doi.org/10.5194/essd-10-1417-2018
Kruskopf, M., Gilliland, L., Roberts, B., Markert, A., Ashmall, W., y Limaye, A. (2025). ClimateServ: An open source earth observation climate data access tool. Environmental Modelling & Software, 194, 106709. https://doi.org/10.1016/j.envsoft.2025.106709 DOI: https://doi.org/10.1016/j.envsoft.2025.106709
Mohr, M., Pebesma, E., Dries, J., Lippens, S., Janssen, B., Thiex, D., Milcinski, G., Schumacher, B., Briese, C., Claus, M., Jacob, A., Sacramento, P., & Griffiths, P. (2025). Federated and reusable processing of Earth observation data. Scientific Data, 12(1), 194. https://doi.org/10.1038/s41597-025-04513-y DOI: https://doi.org/10.1038/s41597-025-04513-y
Sudmanns, M., Tiede, D., Lang, S., Bergstedt, H., Trost, G., Augustin, H., Baraldi, A., y Blaschke, T. (2020). Big Earth data: Disruptive changes in Earth observation data management and analysis? International Journal of Digital Earth, 13(7), 832–850. https://doi.org/10.1080/17538947.2019.1585976 DOI: https://doi.org/10.1080/17538947.2019.1585976