This page explains how we collect, process, and analyze artifact evaluation data, including detailed calculation formulas for all metrics displayed on this site.

Overview

Data Schema Version0.2.0
Total Artifacts3031
Conferences Tracked15 (ACSAC, ATC, CAIS, CHES, EUROSYS, FAST, NDSS, OSDI, PETS, SC, SOSP, SP, SYSTEX, USENIXSEC, WOOT)
Years Covered2017-2026
Total Authors8922
AE Committee Members4591 (2554 unique)
Last Updated2026-06-23 10:54:01 UTC

Conferences Covered

Data is collected from conferences tracked by sysartifacts and secartifacts:

  • Systems: ATC, CAIS, EUROSYS, FAST, OSDI, SC, SOSP
  • Security: ACSAC, CHES, NDSS, PETS, SP, SYSTEX, USENIXSEC, WOOT

Data Collection

We scrape artifact evaluation results from sysartifacts/secartifacts websites, extract paper titles, authors, badges (Available, Functional, Reproducible, Reusable) and repository URLs. For USENIX conferences (ATC, FAST) we also scrape badge data from technical session pages. AE committee data is gathered from sysartifacts/secartifacts plus direct scraping (USENIX, CHES, PETS websites).

Repository statistics (GitHub stars/forks, Zenodo/Figshare downloads) are collected via their public APIs. Artifact URLs are checked monthly for availability (HTTP HEAD requests).

Author names are matched to DBLP for disambiguation and total-publication counts. Author affiliations are enriched using a priority cascade:

  1. Author Index (canonical enriched source, when available)
  2. DBLP person pages (<www> tags)
  3. CSRankings faculty records
  4. OpenAlex API
  5. Semantic Scholar API (fallback)
  6. AE committee data (last resort)

Affiliation names are normalized to canonical forms (e.g., “MIT” → “Massachusetts Institute of Technology”).

All scripts are in the reprodb-pipeline repository. Full CLI reference, API documentation, and data model definitions are in the pipeline documentation.

Pipeline

The pipeline is orchestrated by a stage dependency graph and runs monthly (1st of each month) via GitHub Actions:

  1. Extract DBLP data — download and parse the DBLP XML database (~3 GB compressed) for author–paper mappings and affiliations
  2. Scrape artifact results from sysartifacts/secartifacts websites and USENIX technical session pages
  3. Collect repository statistics (GitHub stars/forks, Zenodo/Figshare downloads) and check artifact URL availability
  4. Compute author statistics — match papers to DBLP authors, filter by AE-active years, compute per-author metrics
  5. Generate area-specific author data — separate systems, security, and combined stats
  6. Aggregate committee statistics — compile AE membership and chair roles from scraped data
  7. Compute combined rankings — apply weighted scoring formula, enforce minimum threshold, assign ranks with dense tie-breaking
  8. Aggregate institution rankings — sum across affiliated authors, classify institution roles
  9. Build author profiles and search index — generate per-author detail records
  10. Record ranking history — snapshot current rankings for trend tracking
  11. Export data (JSON/YAML) to this website

Independent stages run in parallel where the dependency graph allows. All output data structures are formally defined in the Data Schemas documentation.

The complete pipeline takes ~30 minutes and processes 3031+ artifact papers.

Author Metrics

Individual author statistics are computed by matching artifact papers to DBLP records. Each metric is calculated as follows:

Artifacts

The total number of evaluated artifacts (papers with at least one badge) authored by this person across all tracked conferences.

Total Papers

The total number of papers this author published at tracked conferences, counting only years when that conference was conducting artifact evaluation. For example:

  • If ACSAC started AE in 2017, only papers from 2017–present are counted
  • If an author published at ACSAC in 2010–2024, only 2017–2024 papers contribute to the denominator
  • This prevents artificial deflation of Artifact Rate by excluding pre-AE papers

The paper count is determined by matching author names to DBLP records and filtering by conference and year.

Clamping rule: If DBLP undercounts papers (i.e., artifact_count > total_papers due to incomplete DBLP records), total_papers is clamped to equal artifact_count. This guarantees that Artifact Rate ≤ 100%.

Artifact Rate (AR%)

The percentage of an author’s papers (at AE-active conferences) that have artifact badges: AR% = (Artifacts / Total Papers) × 100.

Key point: The denominator includes only papers from years when the venue had artifact evaluation. This ensures the rate reflects artifact adoption within the relevant time window, avoiding both over-inflation (counting only artifact papers) and under-inflation (counting all historical publications).

Cross-area handling: For authors active in both systems and security, contributions are summed. If an author has 10 systems papers and 5 security papers (all in AE-active years), the denominator is 15. This additive approach is correct because systems and security conferences are disjoint publication venues.

Reproducibility Rate (RR%)

Among papers with artifacts, the percentage achieving the highest-tier badge (Reproduced or Reusable): RR% = (Reproduced badges / Total artifacts) × 100. This measures the depth of reproducibility beyond mere artifact availability.

Artifact:Evaluation Ratio (A:E)

The A:E ratio characterizes the balance between artifact production and evaluation service: A:E = Artifact Score / AE Score, where:

  • Artifact Score = sum of badge points (Available+1, Functional+1, Reproduced+1 per artifact)
  • AE Score = committee service points (member=3, chair=5)

AE Memberships

The number of times this author served on an artifact evaluation committee across all tracked conferences.

Chair Count

The number of times this author served as an AE chair or co-chair.

Combined Score

A composite metric balancing artifact production, artifact quality, and AE service: Combined Score = Σᵢ(Aᵢ + Fᵢ + Rᵢ) + Σⱼ(3 + Bⱼ × 2), summing over n artifacts and m AE terms, where:

  • First sum (per artifact i):
    • Aᵢ = 1 if artifact i is Available, 0 otherwise
    • Fᵢ = 1 if artifact i is Functional, 0 otherwise
    • Rᵢ = 1 if artifact i is Reproduced/Reusable, 0 otherwise
    • Maximum per artifact: 3 points (all three badges)
  • Second sum (committee service, per AE term j):
    • Each committee membership contributes 3 points
    • Bⱼ = 1 if term j is a chair role, 0 otherwise — chairs receive a +2 bonus for a total of 5 points per chair term

Minimum Score Threshold: Only individuals and institutions with combined score ≥ 3 are included in rankings. Institutions with placeholder affiliations (“Unknown”, etc.) are excluded.

Ranking Method: Dense ranking with ties — authors with the same combined score receive the same rank; the next rank is incremented by the number of tied entries.

Why These Weights?

  • Additive badge scoring (1 point each) reflects that each badge level requires distinct effort (availability, functionality, reproducibility)
  • AE membership = 3 points estimates the substantial time investment (~50 hours per evaluation cycle)
  • Chair role = 5 points recognizes leadership and coordination responsibilities
  • This formula balances artifact producers and evaluators, countering the traditional invisibility of evaluation labor in academic metrics

Institution Metrics

Institution-level statistics aggregate contributions from all authors affiliated with that institution. Affiliations are determined from DBLP person pages and CSRankings faculty data.

How Institution Data is Aggregated

All metrics are summed across affiliated authors:

  • Artifacts: Total artifacts from all affiliated authors
  • Total Papers: Total papers from all affiliated authors (AE-active years only)
  • AE Memberships: Total committee memberships from all affiliated authors
  • Combined Score: Sum of all affiliated authors’ combined scores
  • Researchers: Count of unique authors affiliated with the institution

Artifact Rate and Reproducibility Rate are then computed from these aggregated totals:

  • Institution AR% = (Total artifacts / Total papers) × 100
  • Institution RR% = (Total reproduced badges / Total artifacts) × 100

Institution Role Classification

Each institution is classified based on its Artifact:Evaluation ratio (artifact_score / ae_score):

  • Artifact-focused (Producer): A:E ratio > 2.0, or artifact-only (no AE service)
  • Evaluation-focused (Consumer): A:E ratio < 0.5, or AE-only (no artifacts)
  • Balanced: 0.5 ≤ A:E ratio ≤ 2.0

Cross-Area Aggregation

For institution rankings broken down by area (systems vs. security):

  • Systems rankings: Include only artifacts, papers, and AE service from systems conferences
  • Security rankings: Include only artifacts, papers, and AE service from security conferences
  • Overall rankings: Sum of systems + security contributions

Badge Definitions

We rely on each conference’s official badge definitions. We treat the same badge name as comparable across venues (e.g., Available in one venue is assumed to mean the same or similar level of availability in another). We make the same assumption for Functional. For the highest tier, Reproduced (security) and Reusable (systems) are treated as equivalent.

Repository Statistics

For artifacts with GitHub/GitLab repositories or Zenodo/Figshare archives, we collect engagement metrics as supplementary signals of community uptake:

GitHub/GitLab Metrics

  • Stars: Number of users who starred the repository
  • Forks: Number of times the repository was forked

Zenodo/Figshare Metrics

  • Downloads: Total download count from the archive platform
  • Views: Number of views/visits to the artifact page

Important notes:

  • Repository statistics are displayed separately and do not contribute to the combined score
  • These metrics reflect external reuse signals but are subject to biases:
    • Age effects (older artifacts accumulate more stars)
    • Repository type differences (libraries vs. experiment code)
    • Discovery algorithm effects (GitHub trending, recommendation systems)
  • We report these as observational data, not as quality judgments

Artifact Citations (Experimental)

We attempted to track academic citations to artifact DOIs using OpenAlex, querying citation counts for artifact DOIs (Zenodo and Figshare).

Why Citation Data Is Not Included in Rankings

OpenAlex reported 14 artifacts with a total of 43 citing DOIs. We verified each citing DOI by checking Crossref publisher-submitted reference lists for the actual artifact DOI, and detected self-citations by comparing author lists between the artifact and the citing paper.

Result: zero genuine third-party artifact citations. All 43 were:

  • 36 false positives — the citing paper’s bibliography contains the paper DOI, not the artifact DOI. OpenAlex conflates these when the artifact and paper share a title.
  • 6 self-citations — the paper cites its own artifact (same authors).
  • 1 unknown — an arXiv preprint whose references could not be resolved.

Because current bibliographic indexes do not reliably distinguish artifact citations from paper citations, citation counts are excluded from the combined score and ranking tables. The citation collection pipeline remains available as an optional, experimental module for future use as citation infrastructure matures.

See the verification scripts and detailed results for the full analysis.

Data Sources

  • sysartifacts.github.io — Systems conference artifact evaluation results (ATC, CAIS, EUROSYS, FAST, OSDI, SC, SOSP)
  • secartifacts.github.io — Security conference artifact evaluation results (ACSAC, CHES, NDSS, PETS, SP, SYSTEX, USENIXSEC, WOOT)
  • usenix.org — Badge information and AE committee data for USENIX conferences
  • DBLP — Author name matching, disambiguation, and total publication counts
  • OpenAlex — Author affiliation enrichment
  • Semantic Scholar — Author affiliation fallback lookup
  • CSRankings — Faculty affiliation data
  • GitHub, GitLab — Repository statistics (stars, forks)
  • Zenodo, Figshare — Archive statistics (downloads, views)
  • Data Schemas — JSON Schema definitions for all data structures used by this site (versioned; see all schema versions)

Acknowledgements

This project celebrates the work of artifact authors who go the extra mile to make research reproducible, and artifact evaluation committees (AE chairs and members) who invest time reviewing and certifying artifacts. Their contributions strengthen our scientific record. We thank the communities maintaining sysartifacts and secartifacts for publishing detailed evaluation results. Inspired by Systems Circus and csrankings.org.

API Access

The full artifact dataset is available as a public JSON endpoint for programmatic access:

GET https://reprodb.github.io/assets/data/search_data.json

Returns an array of all 3031 artifacts with title, authors, affiliations, conference, year, badges, and repository/artifact URLs. No authentication required.

Example using curl:

# Get all artifacts
curl -s https://reprodb.github.io/assets/data/search_data.json | python3 -c "
import sys, json
data = json.load(sys.stdin)
# Filter: fuzzing papers from 2024
results = [a for a in data if 'fuzz' in a['title'].lower() and a['year'] == 2024]
print(json.dumps(results, indent=2))
"

Search Keywords

The search bar supports special # keywords that filter results by specific criteria. Keywords can be combined with each other and with free text.

Keyword Description
#unavailable Artifacts whose URLs may no longer be accessible (checked periodically via automated HTTP probes)
#awarded Award-winning artifacts only
#github Artifacts hosted on GitHub
#zenodo Artifacts hosted on Zenodo
#nourl Artifacts with no artifact URL recorded

Examples:

  • #unavailable — all artifacts with potentially dead links
  • #awarded OSDI — award-winning OSDI artifacts
  • #github #unavailable 2022 — GitHub-hosted artifacts from 2022 with dead links
  • #zenodo fuzzing — Zenodo-hosted fuzzing artifacts

Keywords also work alongside the year, venue, and area dropdown filters.

Note on #unavailable: Availability is determined by automated URL checks that run as part of the monthly pipeline. A URL marked as unavailable may be temporarily down, rate-limited, or require authentication. The hover tooltip on each flagged artifact shows when the check was last performed.