<No description has been provided for this image

DESI DR1 Spectroscopic Redshifts¶

Inspection of basic properties of the DESI DR1 catalog.

Contact: Luigi Silva (luigi.silva@linea.org.br)

Last verified: Apr. 08, 2025

Acknowledgments¶

This notebook used computational resources from the Associação Laboratório Interinstitucional de e-Astronomia (LIneA) with financial support from the INCT of e-Universo (Process no. 465376/2014-2).

This notebook used data obtained with the Dark Energy Spectroscopic Instrument (DESI). DESI construction and operations is managed by the Lawrence Berkeley National Laboratory. This material is based upon work supported by the U.S. Department of Energy, Office of Science, Office of High-Energy Physics, under Contract No. DE–AC02–05CH11231, and by the National Energy Research Scientific Computing Center, a DOE Office of Science User Facility under the same contract. Additional support for DESI was provided by the U.S. National Science Foundation (NSF), Division of Astronomical Sciences under Contract No. AST-0950945 to the NSF’s National Optical-Infrared Astronomy Research Laboratory; the Science and Technology Facilities Council of the United Kingdom; the Gordon and Betty Moore Foundation; the Heising-Simons Foundation; the French Alternative Energies and Atomic Energy Commission (CEA); the National Council of Humanities, Science and Technology of Mexico (CONAHCYT); the Ministry of Science and Innovation of Spain (MICINN), and by the DESI Member Institutions: www.desi.lbl.gov/collaborating-institutions. The DESI collaboration is honored to be permitted to conduct scientific research on I’oligam Du’ag (Kitt Peak), a mountain with particular significance to the Tohono O’odham Nation. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author(s) and do not necessarily reflect the views of the U.S. National Science Foundation, the U.S. Department of Energy, or any of the listed funding agencies.

See complete DESI Data License and Acknowledgements information here.

Introduction¶

This notebook contains a brief characterization of the spectroscopic redshifts (spec-z) measurements in the DESI DR1 catalog, which has been publicly distributed and described in detail in scientific literature by DESI project.

The data presented here were obtained from the DESI data release web page.

The schema of the catalog can be found here.

Reference paper¶

DESI Collaboration et al. (2025), Data Release 1 of the Dark Energy Spectroscopic Instrument

Imports and configurations of libraries¶

Importing the required libraries¶

Importing the libraries.

In [1]:
### GENERAL
import os
import getpass
import numpy as np
import pandas as pd
from IPython.display import Markdown, display
from IPython.display import IFrame

### DASK
import dask.dataframe as dd
from dask.distributed import Client

### VISUALIZATION
# BOKEH
from bokeh.io import output_notebook

# GEOVIEWS
import geoviews as gv
import geoviews.feature as gf
from cartopy import crs

# HOLOVIEWS
import holoviews as hv
from holoviews import opts
from holoviews.operation.datashader import rasterize, dynspread

# DATASHADER
import datashader as ds
In [2]:
hv.extension('bokeh')
gv.extension('bokeh')
output_notebook()
No description has been provided for this image No description has been provided for this image
Loading BokehJS ...
In [3]:
pd.set_option("display.float_format", lambda x: f"{x:.6f}" if 1e-7 <= abs(x) <= 1e8 else f"{x:.3e}")

Setting up a local cluster with dask¶

Defining a local Dask Client.

In [4]:
client = Client(n_workers=2, threads_per_worker=2, memory_limit="7GB")

Printing info about the Client.

In [5]:
client
Out[5]:

Client

Client-cfad6e26-14b8-11f0-91bf-c2d3b49f35e5

Connection method: Cluster object Cluster type: distributed.LocalCluster
Dashboard: http://127.0.0.1:8787/status

Cluster Info

LocalCluster

f7524b6d

Dashboard: http://127.0.0.1:8787/status Workers: 2
Total threads: 4 Total memory: 13.04 GiB
Status: running Using processes: True

Scheduler Info

Scheduler

Scheduler-7ea5a17e-1031-474a-a674-6a1d712ead0a

Comm: tcp://127.0.0.1:45620 Workers: 2
Dashboard: http://127.0.0.1:8787/status Total threads: 4
Started: Just now Total memory: 13.04 GiB

Workers

Worker: 0
Comm: tcp://127.0.0.1:33151 Total threads: 2
Dashboard: http://127.0.0.1:39430/status Memory: 6.52 GiB
Nanny: tcp://127.0.0.1:44213
Local directory: /tmp/dask-scratch-space/worker-okveid10

Worker: 1
Comm: tcp://127.0.0.1:40428 Total threads: 2
Dashboard: http://127.0.0.1:32772/status Memory: 6.52 GiB
Nanny: tcp://127.0.0.1:44762
Local directory: /tmp/dask-scratch-space/worker-j3aywwft

Printing info about the Dask Dashboard.

In [6]:
user = getpass.getuser()
dashboard_url = f"https://jupyter.linea.org.br/user/{user}/proxy/8787/status"

#display(Markdown(f"You can see the dashboard at: [{dashboard_url}]({dashboard_url})"))

Reading the entire catalog with Dask¶

Defining the path to the input files¶

Defining the path to the input files.

In [7]:
input_dir = "/home/luigi.silva/DESI/DR1/DATA/PARQUET"
input_pattern = "*.parquet"

input_files = os.path.join(input_dir, input_pattern)

Reading the catalog¶

Reading the catalog with Dask.

In [8]:
ddf_input = dd.read_parquet(input_files)

Data characterization¶

Full catalog¶

In [9]:
input_nrows = ddf_input.shape[0].compute()

print(f"Total number of lines (full catalog): {input_nrows}")

Total number of lines (full catalog): 28425963
In [10]:
input_cols = ddf_input.columns
input_ncols = len(input_cols)

print(f"Total number of columns (full catalog): {input_ncols}")

Total number of columns (full catalog): 136
In [11]:
print("Columns names (full catalog): \n")
print(input_cols.to_list())

Columns names (full catalog): 

['TARGETID', 'SURVEY', 'PROGRAM', 'HEALPIX', 'SPGRPVAL', 'Z', 'ZERR', 'ZWARN', 'CHI2', 'COEFF', 'NPIXELS', 'SPECTYPE', 'SUBTYPE', 'NCOEFF', 'DELTACHI2', 'COADD_FIBERSTATUS', 'TARGET_RA', 'TARGET_DEC', 'PMRA', 'PMDEC', 'REF_EPOCH', 'FA_TARGET', 'FA_TYPE', 'OBJTYPE', 'SUBPRIORITY', 'OBSCONDITIONS', 'RELEASE', 'BRICKNAME', 'BRICKID', 'BRICK_OBJID', 'MORPHTYPE', 'EBV', 'FLUX_G', 'FLUX_R', 'FLUX_Z', 'FLUX_W1', 'FLUX_W2', 'FLUX_IVAR_G', 'FLUX_IVAR_R', 'FLUX_IVAR_Z', 'FLUX_IVAR_W1', 'FLUX_IVAR_W2', 'FIBERFLUX_G', 'FIBERFLUX_R', 'FIBERFLUX_Z', 'FIBERTOTFLUX_G', 'FIBERTOTFLUX_R', 'FIBERTOTFLUX_Z', 'MASKBITS', 'SERSIC', 'SHAPE_R', 'SHAPE_E1', 'SHAPE_E2', 'REF_ID', 'REF_CAT', 'GAIA_PHOT_G_MEAN_MAG', 'GAIA_PHOT_BP_MEAN_MAG', 'GAIA_PHOT_RP_MEAN_MAG', 'PARALLAX', 'PHOTSYS', 'PRIORITY_INIT', 'NUMOBS_INIT', 'CMX_TARGET', 'DESI_TARGET', 'BGS_TARGET', 'MWS_TARGET', 'SCND_TARGET', 'SV1_DESI_TARGET', 'SV1_BGS_TARGET', 'SV1_MWS_TARGET', 'SV1_SCND_TARGET', 'SV2_DESI_TARGET', 'SV2_BGS_TARGET', 'SV2_MWS_TARGET', 'SV2_SCND_TARGET', 'SV3_DESI_TARGET', 'SV3_BGS_TARGET', 'SV3_MWS_TARGET', 'SV3_SCND_TARGET', 'PLATE_RA', 'PLATE_DEC', 'COADD_NUMEXP', 'COADD_EXPTIME', 'COADD_NUMNIGHT', 'COADD_NUMTILE', 'MEAN_DELTA_X', 'RMS_DELTA_X', 'MEAN_DELTA_Y', 'RMS_DELTA_Y', 'MEAN_PSF_TO_FIBER_SPECFLUX', 'MEAN_FIBER_RA', 'STD_FIBER_RA', 'MEAN_FIBER_DEC', 'STD_FIBER_DEC', 'MIN_MJD', 'MAX_MJD', 'MEAN_MJD', 'TSNR2_GPBDARK_B', 'TSNR2_ELG_B', 'TSNR2_GPBBRIGHT_B', 'TSNR2_LYA_B', 'TSNR2_BGS_B', 'TSNR2_GPBBACKUP_B', 'TSNR2_QSO_B', 'TSNR2_LRG_B', 'TSNR2_GPBDARK_R', 'TSNR2_ELG_R', 'TSNR2_GPBBRIGHT_R', 'TSNR2_LYA_R', 'TSNR2_BGS_R', 'TSNR2_GPBBACKUP_R', 'TSNR2_QSO_R', 'TSNR2_LRG_R', 'TSNR2_GPBDARK_Z', 'TSNR2_ELG_Z', 'TSNR2_GPBBRIGHT_Z', 'TSNR2_LYA_Z', 'TSNR2_BGS_Z', 'TSNR2_GPBBACKUP_Z', 'TSNR2_QSO_Z', 'TSNR2_LRG_Z', 'TSNR2_GPBDARK', 'TSNR2_ELG', 'TSNR2_GPBBRIGHT', 'TSNR2_LYA', 'TSNR2_BGS', 'TSNR2_GPBBACKUP', 'TSNR2_QSO', 'TSNR2_LRG', 'MAIN_NSPEC', 'MAIN_PRIMARY', 'SV_NSPEC', 'SV_PRIMARY', 'ZCAT_NSPEC', 'ZCAT_PRIMARY', 'DESINAME']

Filtered catalog¶

Applying the SPECTYPE==GALAXY and ZCAT_PRIMARY==True filters¶

Applying the SPECTYPE==GALAXY and ZCAT_PRIMARY==True filters in the catalog.

In [12]:
ddf_filtered = ddf_input[ddf_input["SPECTYPE"]=="GALAXY"]
In [13]:
ddf_filtered = ddf_filtered[ddf_filtered["ZCAT_PRIMARY"]==True]

Selecting the columns of interest¶

Selecting the columns of interest in the catalog.

In [14]:
selected_cols = [
    'TARGETID', 'HEALPIX', 'Z', 'ZERR', 'ZWARN',
    'TARGET_RA', 'TARGET_DEC', 'EBV', 'FLUX_G', 'FLUX_R', 'FLUX_Z', 'FLUX_W1',
    'FLUX_W2', 'FLUX_IVAR_G', 'FLUX_IVAR_R', 'FLUX_IVAR_Z', 'FLUX_IVAR_W1',
    'FLUX_IVAR_W2'
]

print(selected_cols)

['TARGETID', 'HEALPIX', 'Z', 'ZERR', 'ZWARN', 'TARGET_RA', 'TARGET_DEC', 'EBV', 'FLUX_G', 'FLUX_R', 'FLUX_Z', 'FLUX_W1', 'FLUX_W2', 'FLUX_IVAR_G', 'FLUX_IVAR_R', 'FLUX_IVAR_Z', 'FLUX_IVAR_W1', 'FLUX_IVAR_W2']
In [15]:
ddf_filtered = ddf_filtered[selected_cols]

ddf_filtered = ddf_filtered.persist()

Number of lines and columns¶

In [16]:
filtered_nrows = ddf_filtered.shape[0].compute()

print(f"Total number of lines (filtered catalog): {filtered_nrows}")

Total number of lines (filtered catalog): 21044698
In [17]:
filtered_cols = ddf_filtered.columns
filtered_ncols = len(filtered_cols)

print(f"Total number of columns (filtered catalog): {filtered_ncols}")

Total number of columns (filtered catalog): 18
In [18]:
print("Columns names (filtered catalog): \n")
print(filtered_cols.to_list())

Columns names (filtered catalog): 

['TARGETID', 'HEALPIX', 'Z', 'ZERR', 'ZWARN', 'TARGET_RA', 'TARGET_DEC', 'EBV', 'FLUX_G', 'FLUX_R', 'FLUX_Z', 'FLUX_W1', 'FLUX_W2', 'FLUX_IVAR_G', 'FLUX_IVAR_R', 'FLUX_IVAR_Z', 'FLUX_IVAR_W1', 'FLUX_IVAR_W2']

First lines of the catalog¶

In [19]:
ddf_filtered.head()
Out[19]:
TARGETID HEALPIX Z ZERR ZWARN TARGET_RA TARGET_DEC EBV FLUX_G FLUX_R FLUX_Z FLUX_W1 FLUX_W2 FLUX_IVAR_G FLUX_IVAR_R FLUX_IVAR_Z FLUX_IVAR_W1 FLUX_IVAR_W2
0 39628473198710603 2152 0.804206 0.000010 0 23.764862 29.832379 0.054335 1.354010 2.976295 10.395741 27.525110 48.779099 531.460571 123.641281 19.293917 2.664506 0.641028
2 39628473198711342 2152 0.932752 2.232e-46 518 23.802207 29.832150 0.053668 2.653994 11.634748 23.412300 50.114071 35.767471 595.788574 118.594986 25.252254 2.198009 0.608285
3 39628473198711925 2152 0.390723 0.000020 0 23.832176 29.858176 0.053723 3.122899 13.893675 35.096653 78.427452 54.463345 436.552338 94.306648 18.671127 1.769941 0.524213
4 39628473202901028 2152 0.665803 0.000076 0 23.866668 29.816397 0.053278 0.331562 1.404510 4.291292 24.670589 14.502599 1265.926758 370.455292 56.139481 2.710926 0.662417
5 39628473202901626 2152 0.328418 0.000050 0 23.898508 29.849485 0.053137 8.766316 30.099854 60.953991 88.483856 50.180351 173.101532 25.908314 5.784760 1.403655 0.399923

Basic statistics¶

In [20]:
ddf_filtered.describe().compute()
Out[20]:
TARGETID HEALPIX Z ZERR ZWARN TARGET_RA TARGET_DEC EBV FLUX_G FLUX_R FLUX_Z FLUX_W1 FLUX_W2 FLUX_IVAR_G FLUX_IVAR_R FLUX_IVAR_Z FLUX_IVAR_W1 FLUX_IVAR_W2
count 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000 21044698.000000
mean 5.282e+10 18979.565458 0.686863 0.000068 256.741457 184.477115 16.480597 0.041694 5.785048 13.613173 25.804079 31.345707 21.482312 543.947215 207.887829 43.991272 2.003729 0.522845
std 1.623e+17 10127.589749 0.464044 0.000058 674.851214 86.142084 21.210698 0.038225 49.013611 101.211184 167.558305 1976.882889 453.540389 653.421766 319.336411 54.513966 1.566050 0.466398
min -8.265e+08 0.000e+00 -0.004312 2.232e-46 0.000e+00 0.000068 -31.601866 0.000e+00 -1770.626953 -248.088272 -444.662659 -34785.710938 -260170.796875 -99.000000 -99.000000 -99.000000 0.000e+00 0.000e+00
25% 3.963e+16 10512.000000 0.407712 0.000040 0.000e+00 156.217883 3.152669 0.025922 0.339624 0.453288 0.828235 0.856831 0.104785 175.050026 50.422132 11.490018 1.296442 0.370481
50% 3.963e+16 20071.000000 0.816868 0.000074 0.000e+00 212.811514 17.130446 0.036964 0.822830 1.952401 7.305542 20.402670 13.988712 436.097748 153.232582 40.473501 2.593952 0.625707
75% 3.963e+16 27247.000000 1.184093 0.000109 5.000000 314.948742 40.435969 0.062465 11.264684 27.309690 51.519180 63.298538 45.356457 1441.455536 530.845047 113.992617 3.476955 0.849641
max 2.882e+18 49151.000000 1.697883 0.007043 2695.000000 359.999980 84.609140 3.964358 70843.726562 149822.156250 193771.828125 9016309.000000 1548954.875000 12979.132812 5741.590332 808.293274 54.963043 16.473513

Spatial distribution plot¶

Getting RA and DEC columns in the Dask Dataframe.

In [21]:
ddf_ra_dec = ddf_filtered[['TARGET_RA', 'TARGET_DEC']]

Changing RA to the $[-180^{\circ}, 180^{\circ})$ interval.

In [22]:
ddf_ra_dec['RA_SHIFTED'] = ((ddf_ra_dec['TARGET_RA'] + 180) % 360) - 180

Plot using the Plate Carrée projection¶

Defining the points element.

In [23]:
hv_points = hv.Points(ddf_ra_dec[['RA_SHIFTED', 'TARGET_DEC']], kdims=['RA_SHIFTED', 'TARGET_DEC'])

Defining the title, the axis labels and the plot sizes.

In [24]:
title = 'Spatial Distribution - SPECTYPE==GALAXY and ZCAT_PRIMARY==True - Plate Carrée Projection'
height = 500
width = 1000
padding = 0.05

Making the plot with geoviews and datashader.

In [25]:
Plate_Carree_rasterized_points = rasterize(hv_points, aggregator=ds.count()).opts(cmap="Viridis", cnorm='log')

Plate_Carree_spread_points = dynspread(Plate_Carree_rasterized_points).opts(
    width=width, height=height, padding=padding, title=title, toolbar='above', colorbar=True,
    tools=['box_select'], show_grid=True, 
    invert_xaxis=True  
)

# For a dynamic plot, uncoment the line below.
#Plate_Carree_spread_points
In [26]:
hv.save(Plate_Carree_spread_points, 'plate_carree_spatial_distribution.html', backend='bokeh')
In [27]:
IFrame('plate_carree_spatial_distribution.html', width=1200, height=600)
Out[27]:

Plot using the Mollweide projection¶

Defining the points element.

In [28]:
gv_points = gv.Points(ddf_ra_dec[['RA_SHIFTED', 'TARGET_DEC']], kdims=['RA_SHIFTED', 'TARGET_DEC']).opts(
    projection=crs.PlateCarree()
)

Defining the title, the axis labels and the plot sizes.

In [29]:
title = 'Spatial Distribution - SPECTYPE==GALAXY and ZCAT_PRIMARY==True - Mollweide Projection'
height = 500
width = 1000
padding = 0.05

Defining the RA and DEC ticks for the Mollweide projection.

In [30]:
longitudes = np.arange(30, 360, 30)
latitudes = np.arange(-75, 76, 15)

lon_labels = [f"{lon}°" for lon in longitudes]
lon_labels.reverse()
lat_labels = [f"{lat}°" for lat in latitudes]

labels_data = {
    "lon": list(np.flip(longitudes)) + [180] * len(latitudes),
    "lat": [0] * len(longitudes) + list(latitudes),
    "label": lon_labels + lat_labels,
}

df_labels = pd.DataFrame(labels_data)

labels_plot = gv.Labels(df_labels, kdims=["lon", "lat"], vdims=["label"]).opts(
    text_font_size="12pt",
    text_color="black",
    text_align='right',
    text_baseline='bottom',
    projection=crs.Mollweide()
)

Making the plot with datashader.

In [31]:
Mollweide_rasterized_points = rasterize(gv_points, aggregator=ds.count()).opts(cmap="Viridis", cnorm='log')

Mollweide_spread_points = dynspread(Mollweide_rasterized_points).opts(
    width=width, height=height, padding=padding, title=title, toolbar='above', colorbar=True,
    invert_xaxis=True
)

grid = gf.grid()

final_mollweide_plot = (Mollweide_spread_points).opts(
    opts.RGB(projection=crs.Mollweide())
) * grid * labels_plot

# For a dynamic plot, uncoment the line below.
#final_mollweide_plot
In [32]:
hv.save(final_mollweide_plot, 'mollweide_spatial_distribution.html', backend='bokeh')
In [33]:
IFrame('mollweide_spatial_distribution.html', width=1200, height=600)
Out[33]:

Redshift plot¶

In [34]:
z_values = ddf_filtered['Z'].compute()
In [35]:
redshift = hv.Dimension('z', label='spec-z')
(count, z_bin) = np.histogram(z_values, bins=np.arange(0,2,0.01))
In [36]:
z_distribution = hv.Histogram((count, z_bin), kdims=redshift).opts(
    title='Redshift Distribution', xlabel='spec-z', height=400, width=800,  xlim=(0.,2.))   
z_distribution
Out[36]: