QDOAS Latest changes

UV-Vis Home > Software > QDOAS > QDOAS Latest changes

Royal Belgian Institute for Space Aeronomy

QDOAS Latest changes

Version 3.6.0 - 1 December 2023

GROUND-BASED MEASUREMENTS

doas_cl can use trigger lists created by acquisition software
FRM4DOAS format adapted for imagers
Remove support for a few older ground-based instrument data formats which are no longer used

MISCELLANEOUS

Can save residual spectra in NetCDF output format
Write sensor, fitting interval and spectral band in NetCDF metadata.

CODE

License changed from GPLv2 to a BSD 3-clause type license.
Various changes to allow compilation with Microsoft Visual C++.
Replace GSL depedendency with Eigen
Remove HDF-EOS2 dependency.

Version 3.5.0 - 21 December 2022

SATELLITES MEASUREMENTS

support GEMS Level-1 format (it's recommended to work on files with dimensions correctly reordered for better performances)

an irradiance in GEMS format is always expected as ref1; a second reference spectrum can be used as ref2 in APEX format
the residual spectrum can be included in the output file in nc format (replaces "residual" option in analysis windows properties
doas_cl : new option -new_irrad to save the wavelength calibration after a run calibration on a irradiance spectrum this option works for GEMS but has never been tested on other formats. Send us a message if you are interested in this option.

added milliseconds in time variable output, for tropomi
due to an update of the codadef files, the GOME-2 reading routine wasn't working anymore for versions > 3.4.1, this has been fixed now.
minor error fixed in netCDF output

GROUND-BASED MEASUREMENTS

ASCII column extended format : a third column can be added with the errors on spectra.
MFC format : measurements with elevation angles > 100 are tagged as off-axis measurements (useful for instruments pointing to two opposite directions)
For groundbased measurements, when a site is specified to recalculate solar angles, it's now possible to specify the convention for calculated solar azimuth angles

MISCELLANEOUS

"spline interpolation requests increasing absissae ..." turns from fatal error to warning (useful for imagers in order not to exclude all rows)
Convolution tool outputs cross sections in ASCII or netCDF format. Cross sections files in netCDF format can be used directly in QDOAS analysis windows properties (useful for imagers). Reference spectra in projects properties should still be in ASCII format even if convolved
hdf-eos output format has been removed

Version 3.4.5 - 21 January 2021

FILE FORMATS

update of reading routine for gome-2 due to an update of .codadef files.
MFC format : fix issue with the selection of reference spectra

MISCELLANEOUS

Ground-based measurements : add the wanted saa convention if angles are recalculated for a specific observation site
Windows OS : 'save plot' in ascii systematically added 'asc' extension even if not wanted

Version 3.4 - 11 March 2020

FILE FORMATS

netCDF format used in the frame of FRM4DOAS is supported. The description of the file format is downloadable from https://frm4doas.aeronomie.be/index.php/frm4doas-guidelines. Python routines are proposed to convert data from most common file formats. It's important to use the latest version of these routines.
selection of an earthshine reference spectrum available for TROPOMI although for better efficiency, it is recommended to provide the reference spectrum as an external file

ANALYSIS

use of Pukite cross sections (preconvolved or calculated by the S/W)
molecular ring correction to account for non linear effects in the spectral analysis windows

MISCELLANEOUS

The QDOAS S/W user manual has not been updated yet since the version 3.2. Except for some minor bug fixed and new advanced options, there shouldn't be too many differences between versions 3.2 and 3.4.

Version 3.2 - 12 September 2017

FILE FORMATS

ASCII file format: QDOAS proposes a new option "column extended", more flexible than the current "line" and "column" ones. Spectra (preceeded or not by wavelengths) can be provided as column vectors and records can be described by pre-defined fields. See "Input File Format" annex for further details.

CONVOLUTION

Super-Gaussian: this line shape is a generalization of the Gaussian with the introduction of two independent parameters determining the width and shape. This line shape was proposed by Steffen Beirle (Max-Planck Institut für Chemie, Mainz), and is discussed in a recent paper in AMT. The current implementation in QDOAS is operational but not optimized in terms of performance.

CALIBRATION

Sliding windows: during the calibration procedure, the calibration interval is divided into a number of smaller intervals. By default, the complete calibration interval is divided into non-overlapping sub-windows of equal size. Now, the user can specify the width of the sub-windows, independently from their number.
Calculation of a pre-shift: if the wavelength calibration of the reference spectrum is really too bad, the calibration procedure couldn't converge and return a wrong value of the shift that corresponds to a local minimum of the chi square. QDOAS proposes to approximate the value of the shift by calculating the correlation between the reference spectrum and the solar spectrum convolved at the resolution of the instrument and use this value as initial shift value of the usual NLLS algorithm.

MISCELLANEOUS

Export data/spectra: browses spectra and saves information and/or spectra in an ASCII file. If only data on records are exported, the format of the generated file is the same as QDOAS output; otherwise, the file including spectra (and eventually the wavelengths) can be read by QDOAS with the new "column extended" ASCII format.
Sequential analysis (use the reference of the MAXDOAS scan) should work for most of ground-based file formats including MFC formats generated by DOASIS and MS-DOAS.
The linear and non-linear fitting routines were tuned to improve performance. This should lead to time savings of 10% up to 50%, depending on the configuration used. The linear fitting routines now use a QR decomposition from the GNU Scientific Library. The changes in the non-linear fitting algorithm have a minimal impact on results, leading to relative differences of 0.1% or less with respect to results obtained with previous versions of QDOAS.
QDOAS' netCDF output files now contain fletcher32 checksums for all variables.
Instructions to compile QDOAS on macOS were updated, and, where necessary, the code was adjusted to allow smooth compilation on macOS.
QDOAS can now read ascii input files with legacy Macintosh newline encoding (single carriage return).
Improved handling of non-ascii characters in configuration file names.
QDOAS is now compatible with Qt5.

FIXED BUGS

When processing GOME-2 files, geolocation data is passed to the output also for pixels with invalid spectra.
The MFC binary format can now also be used on 64-bit linux systems.

Version 2.111 - 5 April 2016

FILE FORMATS

MFC original binary format generated with DOASIS or MS-DOAS is supported.
MAXDOAS measurements: extended options for the automatic selection of the zenith-sky spectrum of the scan, currently supported by a limited number of formats including MFC one. Please, contact authors if you want to apply this option to your format.
BIRA-IASB airborne and mobile DOAS measurements are split. MobileDOAS can be selected under ground-based measurements; hundreds of seconds are accounted for in the fractional time.
Extra output fields available for GOME-2: “MDR number”, “Observation index” (=index within an MDR), “Observation Mode” (“Nadir”/”North Pole Scanning”/…), solar zenith and azimuth angles in satellite reference system, satellite latitude and longitude.
Extra output fields for OMI: Spacecraft altitude/longitude/latitude, instrument configuration id.
QDOAS can now handle older OMI L1B files which do not contain “XtrackQualityFlags”, when the configuration doesn't use these flags.

MISCELLANEOUS

Fill Values are used in the output data when processing fails, with proper fill value attributes set in netCDF and HDF-EOS5 output formats.
New output fields “number of wavelength bands used in the fit”, “processing error flag”, “Analysis window center pixel wavelength”.
“Spectrum” and “Irradiance” plots for satellite data now use the wavelength grid from the L1B data if available (GOME-2, OMI). The axis labels have been changed from “Counts” to the physical units for the radiometrically calibrated (ir-)radiances.
New meta-data attributes in netCDF output files: “QDOASConfig” contains the configuration file name, “QDOASConfigProject” contains the name of project within that file that was used to produce the results.
Improved error messages for some incorrect configuration settings.
The calibration procedure can be preceded by the calculation of a pre-shift. This additional step approximates the value of the shift by calculating the correlation between the reference spectrum and the solar spectrum convolved at the resolution of the instrument using the initial value(s) of the slit function parameter(s). This step is useful if the wavelength calibration is not stable or when the initial wavelength calibration is really bad.
New error message if a cross section file does not have the expected number of columns. This implies that the two last columns of previous pre-convolved ring cross sections generated by the QDOAS ring tool should be removed. Such ring cross sections are usually introduced to fit the wavelength calibration. If the option “convolve ring” is used, the file should have the 4 columns (the two last ones are the convolved solar and Raman spectra). The QDOAS ring tool includes now a new option to save or not the convolved solar and Raman spectra and generates a four or two columns file.

FIXED BUGS

The new wavelength calibration procedure announced in version 2.109 was not actually used for GOME-2. This has now been fixed.
A number of bugs related to netCDF output have been fixed.
Fixed a bug in the output of calibration fields with more than one dimension (e.g. calibration stretch).
A minor bug appeared in version 2.108 (determination of the width of the calibration sub-windows) and has been fixed; it is then recommended to switch to this release.
GOME-2 satellite altitude output fixed.

Version 2.110

FIXED BUGS

When working with GOME-2 L1B files in previous versions of QDOAS, the “Scan index” (1 = forward scan, 2 = backward scan, 0 = “other”) property was not stored properly.
Milliseconds of OMI measurement timestamps are now properly reported in the output.

MISCELLANEOUS

The configuration of the “2n-Lorentz” line shape is changed. Until now, the user had to specify the polynomial degree “2n” in the line shape properties when using a 2n-Lorentz line shape. The interface made it possible to enter an odd number, which lead to errors when running the analysis. The interface now requests the number “n”, making such invalid configurations impossible. Old configuration files will automatically be adapted when saved with this new version.
QDOAS can now provide output in the netCDF format, in addition to the existing ASCII and HDF-EOS5 output options.

Version 2.109 - 9 April 2015

WAVELENGTH CALIBRATION FOR SATELLITE FORMATS

For GOME-2 and OMI, radiances are now interpolated onto the original level 1 irradiance wavelength grid before the start of the analysis, using the wavelength calibration provided in the level 1 files. For projects using a fitted shift/stretch for the radiance (as recommended), this change should have negligible impact on retrieved slant column densities. Fitted shift and stretch values will be different compared to previous versions, because part of the shift/stretch between radiance and irradiance is now accounted for by using the level 1 calibration. Though this change makes it possible to obtain results relying purely on the original level 1 wavelength calibration, we still recommend fitting an additional shift/stretch of the radiance in QDOAS for optimal accuracy.

FIXED BUGS

For OMI data, conversion of measurement timestamps from TAI to UTC did not take into account UTC leap seconds, leading to an error in the displayed datetime of up to 8 seconds (8 seconds for data after 1 July 2012, when the last UTC leap second occurred). Qdoas now displays the correct UTC time for each OMI measurement.
Selecting 'output calibration' in a project that doesn't use calibration no longer crashes the program.
Calibration output for GOME/GOME-2 was broken in the previous release; now fixed.
Qdoas can now handle slit function files and ascii spectra with a large number of columns (before, ASCII files with very long lines could cause a problem).
The output field 'Time (hh:mm:ss)' caused problems when writing HDF-EOS5 output files due to restrictions on HDF-EOS5 field names. It has been renamed to 'Time (hh-mm-ss)'.
Reading ASCII files on systems with locale settings that expect comma as a decimal separator (for example certain German language locales) was broken.
Trying to open an incorrect file in a GOME-2 project no longer causes the program to crash. An error message is displayed instead.
When processing OMI or GOME-2 spectra using a reference spectrum generated from averaged (earthshine) radiances, the “ref1/ref2 alignment” used to align this reference on to the irradiance grid was not used anymore in the previous version 2.108.
For OMI, the “ref1/ref2” alignment plot no longer causes the calibration results plot to disappear.

MISCELLANEOUS

At the end of a processing session in the GUI, the session is closed automatically; the user doesn't need to press the red 'stop' button when all available records have been processed.
Modified layout of the analysis window properties and project properties menus, so they take up less space and fit on smaller screens.
There is now an option to use a linearized “offset” in optical density fitting mode, where a polynomial divided by the reference spectrum (instead of the analyzed spectrum) is added to the linear fit components.
Minor change in the calculation of the “resol.” cross section in order to better quantify the difference in resolution between the spectrum and the reference.
Ring tool: add convolution option “None” to disable convolution by the slit function.
Plots are now drawn with antialiasing, which gives a much smoother look.
Improved layout for plot images generated using the “export as image” feature.

Version 2.108 - 29 April 2014

FIXED BUGS

HDF-EOS5 output did not work when no automatic calibration was used for one of the analysis windows.
When an analysis window is configured to re-use the fit result for a given cross section from a previous analysis window, Qdoas now displays an error message when it cannot find the exact same cross section (i.e. same cross section file) in a previous analysis window, instead of failing silently.
When using a pre-convolved, 60 row high-resolution reference spectrum for OMI, where each row is the result of the convolution of a high-resolution solar spectrum with the slit function for the corresponding row of the OMI detector, the spectrum for row 1 was used in the wavelength calibration of the irradiance spectrum for all rows, and the other rows were being ignored. Qdoas now uses the correct spectrum to calibrate the wavelengths for each detector row.
The calculation of undersampling was sometimes wrong due to a bad selection of the slit function type.
The name of the slit function was wrong in the headers.
Resol cross section introduced in version 2.106 was centered around 1. We subtract 1. now in order to have the cross section around 0. The Fit Store button value was not correctly read from the configuration file.
The disabled status of analysis windows was not accounted for by doas_cl.
In the case of multiple analysis windows and different criteria of selection for the reference spectrum, if the Ref2/Ref1 alignment was not possible in an analysis window due to a bad reference spectrum, the alignment of reference spectra in the following analysis windows was not performed but the spectra were processed anyway conducting to bad results.
For multiple analysis windows using different reference spectra (which leads to different wavelength calibrations for those windows), calibration output only contained the results for the first analysis window. Now, calibration data for all analysis windows is included. Likewise, the display of calibration results in the GUI now includes results for multiple analysis windows, when different calibrations are used.
When writing output files to directories yyyy/mm/dd, previous versions of Qdoas used the date of the first successfully processed spectrum to choose the output directory. For satellite orbit files starting shortly before midnight, the first successfully processed spectrum would often come from a measurement taken after midnight, and therefore Qdoas put the results in the directory for the next day. Now, Qdoas uses the date of the first observation in a satellite orbit file to determine the output directory, regardless of whether the spectrum from this observation was processed.
A time conversion bug was fixed. Due to this bug, results from satellite measurements taken on the first day of a given year would be written to a folder for the “zeroth” day of the previous year.

GROUND-BASED MEASUREMENTS

For MAXDOAS measurements, the automatic search of the reference spectrum of the scan should work for CCD-EEV, ASCII, MFC STD and MFC BIRA-IASB binary formats.
For MFC format, a converter exists to create daily binary files (BIRA format) with spectra, dark currents and offsets. QDOAS automatically corrects spectra using the dark currents and offsets measurements.

MISCELLANEOUS

Ring tool: new check button to normalize the Raman spectrum before calculating the ring.
When the user has chosen to write output files, Qdoas checks if it can write to the configured output directory before starting the analysis.
Interface improvements: layout of calibration properties and analysis window properties is now adapted to the font size to improve readability; same for text fields in the slit function properties tab, which were sometimes partly hidden, depending on screen resolution and font size.
Configuration files: update of the xml format. The change is backwards-compatible and only affects config files where an instrument transmission function is specified (see the section about the 'instrument' tab of the project properties in the manual). Configuration files created with older versions of Qdoas can still be used with this version, but for configuration files saved with the new version, the transmission function will not be loaded correctly in older versions of Qdoas.
In the output page of project properties, the color index field is replaced by the bandwidth on which to calculate the flux (as it is possible to save different central wavelengths separated by semi-colons, color indexes can be calculated outside QDOAS)
Addition of a button to save successful records only (default behavior) or to save all records including reference spectra and records for which the analysis fails (log error on the spectrum for example). Sometimes, it is useful to have all records (for example to retrieve information on the reference spectra from the output files or to always have the same number of records in the output files whatever the analysis fails or not.
Qdoas now supports Qwt version 6.1.0, as well as 6.0.2.

Version 2.106 - 4 November 2013

FIXED BUGS

An error in the linear offset fitting routine, leading to incorrect offset values, was fixed.
Running analysis with incomplete calibration settings no longer causes Qdoas to segfault. Qdoas now continues to work after reporting the error.
Output to a file in the current working directory without leading "./" was broken, now fixed.

MISCELLANEOUS

The format of the residual output file (see manual, section "Analysis windows properties") was slightly changed to be compatible with the spike removal algorithm.
Manual update/harmonization: all information regarding configuration of the calibration procedure moved to a single section.
Linux: installation script now automatically installs the online Help files in the right location during "make install" (default location: /home/username/doc/qdoas/Help).
Resol cross section introduced in predefined parameters. It is built from the reference spectrum with a gaussian FWHM of 0.05 nm
Add 2D slit function in the calibration procedure. A stretch can be fitted as for 1D file.
Possibility to save the record number of the automatically selected reference spectrum
Calculation of fluxes: the interval is defined now in wavelengths instead of pixels

Version 2.105 - 29 May 2013

FIXED BUGS

HDF5 output of polynomial components works.
Preconvolved solar reference spectra and cross sections are no longer required to have the same number of points.

MISCELLANEOUS

The spike removal test now uses the RMS residual to decide if a wavelength should be removed (previously, the average magnitude was used).

Version 2.103 - 29 April 2013

OMI

QDOAS can now process OMI level one files. Separate reference spectra/slit functions/calibration parameters are used for each of the 60 detector rows.
OMI PixelQualityFlags and XtrackQualityFlags can be used to exclude pixels or spectra from the analysis.

File formats

SCIAMACHY: possibility to correct the non linearity of the detector (new file to enter in the instrumental page of projects properties.
GOME2: compatibility with BEAT 6.7.0/CODA 2.7.2 definitions.
GOME2 and automatic reference selection: days with a lot of empty files make thes program crash.
GOME-ERS: version 2.1 couldn't read anymore the geolocation coordinates of old versions of GOME.

Analysis windows properties

Resol: in predefined parameters, this new parameter automatically generated from the reference spectrum allows to account for very small differences in the determined resolution.

Convolution

Asymmetric line shapes can be fitted with a asymmetric Gaussian line shapes (fit of an asymmetric factor in addition to the Gaussian FWHM Slit Function Parameter).
Functions of type « File » can be fitted in the Calibration page of Projects properties; a different stretch factor is fitted for each side of the slit function; lookup tables of functions are accepted.
Before, if the slit function was not fitted in the Calibration page of Projects properties, a pre-convolved solar spectrum was expected in this page. Now, the program uses the information of the Slit page to convolve the solar spectrum. If the slit function type is « None », the pre-convolved solar spectrum has to be entered in this Slit page.

Output

HDF5 format is supported

Miscellaneous

QDOAS S/W user manual has been updated with new features; algorithms description chapter has been completely reviewed.
A « spike removal » function was added, which can automatically remove pixels with exceptionally high residuals.
The display of analysis results in the spreadsheet was fixed so that long numbers are no longer truncated.
Review Chi square/RMS formulas. The value previously reported as RMS was the square root of the reduced Chi square value. Now it is the RMS of the fitting residuals. Formulas are given in the user manual.
A bug was fixed which caused the analysis to stop when using least squares weighting by instrumental errors.
Pixels within gaps are now ignored in the plots.
This version includes a number of small performance improvements.
In intensity fitting only, possibility to authorize the concentrations of molecules within a given ranges of values (not recommended except for tests purposes).
Doas_cl: new -xml switches to configure some options of the xml file during batch processing

Last update on 13 Dec 2023