Pipelines

Before delving into the structure of the toast package, it is sometimes useful to look at (and use!) an example. One such program is the simple script below which simulates a fake satellite scanning strategy with a focalplane of detectors and then makes a map.

Simple Satellite Simulation

The current version of this tool simulates parameterized boresight pointing and then uses the given focalplane (loaded from a pickle file) to compute the detector pointing. Noise properties of each detector are used to simulate noise timestreams.

In order to create a focalplane file, you can do for example:

import pickle
import numpy as np

fake = {}
fake['quat'] = np.array([0.0, 0.0, 1.0, 0.0])
fake['fwhm'] = 30.0
fake['fknee'] = 0.05
fake['alpha'] = 1.0
fake['NET'] = 0.000060
fake['color'] = 'r'
fp = {}
fp['bore'] = fake

with open('fp_lb.pkl', 'wb') as p:
    pickle.dump(fp, p)

Note that until the older TOAST mapmaking tools are ported, this script requires the use of libmadam (the –madam option).

usage: toast_satellite_sim.py [-h] [--samplerate SAMPLERATE]
                              [--spinperiod SPINPERIOD]
                              [--spinangle SPINANGLE]
                              [--precperiod PRECPERIOD]
                              [--precangle PRECANGLE] [--hwprpm HWPRPM]
                              [--hwpstep HWPSTEP] [--hwpsteptime HWPSTEPTIME]
                              [--obs OBS] [--gap GAP] [--numobs NUMOBS]
                              [--obschunks OBSCHUNKS] [--outdir OUTDIR]
                              [--debug] [--nside NSIDE] [--baseline BASELINE]
                              [--noisefilter] [--madam] [--fp FP]

Simulate satellite boresight pointing and make a noise map.

optional arguments:
  -h, --help            show this help message and exit
  --samplerate SAMPLERATE
                        Detector sample rate (Hz)
  --spinperiod SPINPERIOD
                        The period (in minutes) of the rotation about the spin
                        axis
  --spinangle SPINANGLE
                        The opening angle (in degrees) of the boresight from
                        the spin axis
  --precperiod PRECPERIOD
                        The period (in minutes) of the rotation about the
                        precession axis
  --precangle PRECANGLE
                        The opening angle (in degrees) of the spin axis from
                        the precession axis
  --hwprpm HWPRPM       The rate (in RPM) of the HWP rotation
  --hwpstep HWPSTEP     For stepped HWP, the angle in degrees of each step
  --hwpsteptime HWPSTEPTIME
                        For stepped HWP, the the time in seconds between steps
  --obs OBS             Number of hours in one science observation
  --gap GAP             Cooler cycle time in hours between science obs
  --numobs NUMOBS       Number of complete observations
  --obschunks OBSCHUNKS
                        Number of chunks to subdivide each observation into
                        for data distribution
  --outdir OUTDIR       Output directory
  --debug               Write diagnostics
  --nside NSIDE         Healpix NSIDE
  --baseline BASELINE   Destriping baseline length (seconds)
  --noisefilter         Destripe with the noise filter enabled
  --madam               If specified, use libmadam for map-making
  --fp FP               Pickle file containing a dictionary of detector
                        properties. The keys of this dict are the detector
                        names, and each value is also a dictionary with keys
                        "quat" (4 element ndarray), "fwhm" (float, arcmin),
                        "fknee" (float, Hz), "alpha" (float), and "NET"
                        (float). For optional plotting, the key "color" can
                        specify a valid matplotlib color string.

Example: Proposed CoRE Satellite Boresight

Here is one example using this script to generate one day of scanning with a single boresight detector, and using one proposed scan strategy for a LiteCoRE satellite:

toast_satellite_sim.py --samplerate 175.86 --spinperiod 1.0 --spinangle 45.0
--precperiod 5760.0 --precangle 50.0 --hwprpm 0.0 --obs 23.0 --gap 1.0
--obschunks 24 --numobs 1 --nside 1024 --baseline 5.0 --madam --noisefilter
--fp fp_core.pkl --outdir out_core_nohwp_fast

Example: Proposed LiteBIRD Satellite Boresight

Here is how you could do a similar thing with a boresight detector and one proposed lightbird scanning strategy for a day:

toast_satellite_sim.py --samplerate 23.0 --spinperiod 10.0 --spinangle 30.0
--precperiod 93.0 --precangle 65.0 --hwprpm 88.0 --obs 23.0 --gap 1.0
--obschunks 24 --numobs 1 --nside 1024 --baseline 60.0 --madam --fp fp_lb.pkl
--debug --outdir out_lb_hwp

Creating Your Own Pipeline

TOAST is designed to give you tools to piece together your own data processing workflow. Here is a slightly modified version of the pipeline script above. This takes a boresight detector with 1/f noise properties, simulates a sort-of Planck scanning strategy, generates a noise timestream, and then generates a fake signal timestream and adds it to the noise. Then it uses madam to make a map.

#!/usr/bin/env python3

import mpi4py.MPI as MPI

import os
import re

import numpy as np

import quaternionarray as qa

import toast
import toast.tod as tt
import toast.map as tm


# scanning parameters

samplerate = 25.0 # Hz
spinperiod = 1.0 # minutes
spinangle = 85.0 # degrees
precperiod = 0 # minutes
precangle = 0 # degrees

# we add a HWP here, since we have one detector for just a couple days

hwprpm = 60.0

# map making

nside = 256
baseline = 60.0

# make a fake focalplane

fake = {}
fake['quat'] = np.array([0.0, 0.0, 1.0, 0.0])
fake['fwhm'] = 30.0
fake['fknee'] = 0.1
fake['alpha'] = 1.5
fake['NET'] = 0.0002
fp = {}
fp['bore'] = fake

# Since madam only supports a single observation, we use
# that here so that we can use the same data distribution whether
# or not we are using libmadam.  Normally we would have multiple 
# observations with some subset assigned to each process group.

# This is the 2-level toast communicator.  By default,
# there is just one group which spans MPI_COMM_WORLD.

comm = toast.Comm()

# construct the list of intervals.  We'll use 55 minute intervals
# with a 5 minute gap. and observe for 4 days.

numobs = 4 * 24
science = 55 * 60
gap = 5 * 60

intervals = tt.regular_intervals(numobs, 0.0, 0, samplerate, science, gap)

# how many samples in one interval (plus the gap)?

interval_samples = intervals[1].first - intervals[0].first

# when we distribute the data, we don't want to split these intervals
# between processes.  So make a list of the samples in each interval
# and pass that list when constructing the TOD class.  This way it 
# distributes the data as evenly as possible among the processes.

distsizes = []
for it in intervals:
    distsizes.append(interval_samples)

# how many total samples do we have now?

totsamples = np.sum(distsizes)

# create the single TOD for this observation

detectors = sorted(fp.keys())
detquats = {}
for d in detectors:
    detquats[d] = fp[d]['quat']

tod = tt.TODSatellite(
    comm.comm_group, 
    detquats,
    totsamples,
    firsttime=0.0,
    rate=samplerate,
    spinperiod=spinperiod,
    spinangle=spinangle,
    precperiod=precperiod,
    precangle=precangle,
    sampsizes=distsizes
)

# Create the noise model for this observation

fmin = 2.0 / samplerate
fknee = {}
alpha = {}
NET = {}
for d in detectors:
    fknee[d] = fp[d]['fknee']
    alpha[d] = fp[d]['alpha']
    NET[d] = fp[d]['NET']

noise = tt.AnalyticNoise(rate=samplerate, fmin=fmin, detectors=detectors, fknee=fknee, alpha=alpha, NET=NET)

# The distributed timestream data

data = toast.Data(comm)

# Create the (single) observation

ob = {}
ob['name'] = 'mission'
ob['tod'] = tod
ob['intervals'] = intervals
ob['baselines'] = None
ob['noise'] = noise
ob['id'] = 0

data.obs.append(ob)

# Constantly slewing precession axis, so that it makes a circle in one year

degday = 360.0 / 365.25

precquat = tt.slew_precession_axis(nsim=tod.local_samples[1], 
    firstsamp=tod.local_samples[0], samplerate=samplerate, degday=degday)

# we set the precession axis now, which will trigger calculation
# of the boresight pointing.

tod.set_prec_axis(qprec=precquat)

# simulate noise

nse = tt.OpSimNoise(out='simdata')
nse.exec(data)

# make a Healpix pointing matrix.

pointing = tt.OpPointingHpix(nside=nside, nest=True, mode='IQU', hwprpm=hwprpm)
pointing.exec(data)

# Simulate a gradient signal and accumulate it to the same output
# as the noise simulation.

grad = tt.OpSimGradient(out='simdata', nside=nside, min=-1.0, max=1.0, nest=True)
grad.exec(data)

# Mapmaking.  For purposes of this simulation, we use detector noise
# weights based on the NET (white noise level).  If the destriping
# baseline is too long, this will not be the best choice.

detweights = {}
for d in detectors:
    net = fp[d]['NET']
    detweights[d] = 1.0 / (samplerate * net * net)

# Set up MADAM map making.  By setting purge=True, we will
# purge all data after copying it into the madam
# buffers.  This is ok, as long as madam is the last step of 
# the pipeline.

outdir = 'out_example_customize'
if not os.path.isdir(outdir):
    os.mkdir(outdir)

pars = {}

cross = int(nside / 2)
submap = int(nside / 8)

pars[ 'temperature_only' ] = 'F'
pars[ 'force_pol' ] = 'T'
pars[ 'kfirst' ] = 'T'
pars[ 'base_first' ] = baseline
pars[ 'nside_map' ] = nside
pars[ 'nside_cross' ] = cross
pars[ 'nside_submap' ] = submap
pars[ 'write_map' ] = 'T'
pars[ 'write_binmap' ] = 'T'
pars[ 'write_matrix' ] = 'T'
pars[ 'write_wcov' ] = 'T'
pars[ 'write_hits' ] = 'T'
pars[ 'kfilter' ] = 'F'
pars[ 'run_submap_test' ] = 'T'                
pars[ 'fsample' ] = samplerate
pars[ 'path_output' ] = outdir

madam = tm.OpMadam(params=pars, detweights=detweights, name='simdata', purge=True)
madam.exec(data)