Project

General

Profile

Purpose

The STDWeb / STDPipe group coordinates the maintenance, development, and expert use of the GRANDMA optical data processing tool STDWeb/STDPipe (Simple Transient Detection Pipeline (Karpov, S. 2025)):

Group Mission

  1. Provide robust, well-documented tools for optical data analysis within GRANDMA.
  2. Act as the central hub of expertise for STDPipe and STDWeb, both scientifically (photometry) and technically (code base, infrastructure).
  3. Assist with data validation and troubleshooting analysis issues.
  4. Coordinate development of new features, improvements, and bug fixes based on user feedback and scientific needs.
  5. Implement automated processing (from image deposit to SkyPortal upload).
  6. Facilitate communication between scientists, developers, telescope teams, and SkyPortal teams.

Group Structure

Chairs:

S. Karpov, D. Akl
  • Core developer and primary referent for STDWeb/STDPipe.
  • Coordinates tasks, roadmap, and communication.
  • Liaises with GRANDMA management and work-package chairs.

Collaborators

  1. Program chairs : for implementing new techniques that could help with different aspects of analysis, e.g. GW program chair could ask for the development of certain feature such as viewing unstacked frames of a candidate (Already implemented!)
  2. SkyPortal Team for automation and ease of analysis
  3. Developers for front/backend development who must have:
    • Strong technical, Python, and data-processing background.
    • Deep knowledge of optical photometry (PSF, aperture, calibration,…).

Functional Overview: (aka What the Group Aims To Do)

Expertise and User Support:

Act as reference contact for:
  • STDPipe usage questions.
  • Interpretation of photometric results.
  • Pipeline behaviour in unusual conditions (bad weather, non-standard filters, galaxies, etc.).
  • Validate tasks involving:
  • High color terms and non-standard photometric systems.
  • Non-constant backgrounds and complex sky structures.
  • Near-IR filters and strong sky variability.
  • Strong galaxy contamination or crowded fields.

Validate descriptions of STDWeb/STDPipe in publications.

Data Flow and Automation (OwnCloud Integration)

Aim to implement an automated process for GRANDMA Observations as follows:
  1. OwnCloud repositories are created automatically for telescopes/programs.
    When a user deposits an image in the relevant OwnCloud folder, it:
  2. Triggers STDWeb.
    • Runs calibration, astrometry, photometry, and TBD (transient detection, subtraction, etc…Folders must be flagged with the correct task needed)
    • Produces reduced products and measurements.
    • SkyPortal integration: Automatically uploads results to SkyPortal photometry plots (will need monitoring to ensure it appears under the correct source (in case of tasks with unspecified coordinates), instrument, and filter)
    • Flag certain detections (such as marginal measurements).

Tools and Features

The group maintains, improves, and validates the full suite of STDPipe and STDWeb analysis capabilities. These include, but are not limited to:
  • Image stacking options
  • Calibration & Preprocessing
  • Source Detection & Photometry
  • Galaxy Contamination & Host Subtraction
  • Filter & Instrument Support
  • Quality Control & Diagnostics
  • Transient Detection
  • User Interface & Workflow Tools

Operational Responsibilities

Monitor :
  • Stuck / crashed tasks and restart or debug them as necessary.
  • Free disk space on STDWeb / associated storage.
Ensure that the automated pipelines are running as expected, especially during:
  • Major follow-up campaigns (GW alerts, GRBs, etc.).
  • Periods of high data volume.

Access and Enrollment

Getting Access to STDWeb:

Access to STDWeb is provided via request to S. Karpov.

Joining the STDWeb/STDPipe Group:

  • STDWeb Slack Channel #stdweb-group ; Includes:
  • Communications of any issues faced (primary method of reporting issues, errors, and warnings)
  • Validation of tasks/analysis
  • Updates to the web portal (STDWeb)

Development Workflow

Feature Requests

Can originate from :
  • Program chairs
  • Group Leaders
  • SkyPortal / Program chairs
  • Active users during campaigns

Recommended workflow:

  1. Inform the chair of the issue via #stdweb group.
    Chair will assess the scope of the bug/feature:
    For larger, long-term work, the chair will ask the user to open an issue in the STDPipe / STDWeb repository (link below)
  2. Chair triages:
    • Bug vs feature request.
    • Priority (urgent / medium / low).
      1. Task is assigned to:
        • Chair.
        • External developer when appropriate.

Development & Deployment

  • Implement changes in a dedicated branch
  • Add / update tests and documentation when relevant
  • Deploy to production STDWeb instance.
  • Announce major changes on the Slack channel and the STDWeb wiki changes log.

Documentation and Use

Wiki:

  • Group Overview/Functionalities
  • Changes page
  • Common issues/workarounds
  • Links to training material.
  • All important decisions and procedures should be mirrored in the wiki so they are not “lost” in chat history.

Guides and Tutorials

  • STDWeb User Guide (Overleaf)
    Current main guide:
    https://fr.overleaf.com/read/jkmyyxkfnjhj#a8caa1
    Upcoming Tasks:
  • Update sections reflecting new features (e.g. stacking).
  • Update contacts
  • Video Tutorials
  • Maintain a list of recorded sessions
  • When new significant features are deployed, schedule short sessions will highlight the updates.

Useful Links

STDweb: simple transient detection pipeline for the web. S. Karpov
https://doi.org/10.14311/AP.2025.65.0050

STDWeb: https://grandma-stdpipe.ijclab.in2p3.fr or http://stdweb.favor2.info/
STDPipe Git Repository: https://github.com/karpov-sv/stdpipe
STDWeb Wiki: https://forge.in2p3.fr/projects/stdweb/wiki
STDWeb User Guide (Overleaf): https://fr.overleaf.com/project/65f816e5b812270262eb60ce