artifinder

src.scrapers.artifinder

Loader for ArtiFinder-Data (https://github.com/DistriNet/ArtiFinder-Data).

ArtiFinder scrapes conference papers directly and identifies links to their artifacts. The published data set is organised as::

data/<venue>/<year>.yaml

where <venue> is one of ccs, ndss, sp, usenix and each YAML file is a list of entries::

- title: "Paper title."
  authors: ["Jane Doe", "John Roe 0001"]
  page_link: "https://doi.org/..."
  artifacts_discovered:                 # ordered best score first; [] if none
    - "https://github.com/org/repo"
    - "https://doi.org/10.5281/zenodo.1"
  validated: false

The older schema used a single discovered_artifact: <url> | null key; it is still accepted. Only entries with at least one link are returned, but the total number of scanned papers per conference-year is also reported so that a discovery rate can be computed. All links are kept, normalised and de-duplicated, in ArtiFinder's order (best score first) as artifact_urls.

Public API

load_artifinder(conf_regex=None) -> ArtiFinderData load_artifinder_data(conf_regex=None) -> list[dict] # entries only

ArtiFinderNoResultsError

Bases: RuntimeError

Raised when ArtiFinder yields no discovered-artifact entries at all.

This usually means the upstream data could not be fetched (network/proxy problem, GitHub rate limiting, a changed repository layout) or that the conf_regex / min_year filters excluded everything. Failing loudly prevents silently overwriting previously generated outputs with empty ones.

Source code in src/scrapers/artifinder.py

class ArtiFinderNoResultsError(RuntimeError):
    """Raised when ArtiFinder yields no discovered-artifact entries at all.

    This usually means the upstream data could not be fetched (network/proxy
    problem, GitHub rate limiting, a changed repository layout) or that the
    ``conf_regex`` / ``min_year`` filters excluded everything. Failing loudly
    prevents silently overwriting previously generated outputs with empty ones.
    """

ArtiFinderData

Bases: NamedTuple

Parsed ArtiFinder data set.

Attributes:

Name	Type	Description
entries	list[dict]	One dict per paper that has a discovered artifact link, with keys conference, category, year, title, authors, page_link and artifact_urls (all discovered links, best score first).
counts	list[dict]	One dict per conference-year with keys conference, category, year, total_papers (scanned) and discovered (papers with a non-null artifact link).
source_updated	str | None	YYYY-MM-DD date the ArtiFinder-Data source was last updated (its latest commit), or None if it could not be found.

Source code in src/scrapers/artifinder.py

class ArtiFinderData(NamedTuple):
    """Parsed ArtiFinder data set.

    Attributes:
        entries: One dict per paper that has a discovered artifact link, with
            keys ``conference``, ``category``, ``year``, ``title``, ``authors``,
            ``page_link`` and ``artifact_urls`` (all discovered links, best
            score first).
        counts: One dict per conference-year with keys ``conference``,
            ``category``, ``year``, ``total_papers`` (scanned) and
            ``discovered`` (papers with a non-null artifact link).
        source_updated: ``YYYY-MM-DD`` date the ArtiFinder-Data source was last
            updated (its latest commit), or ``None`` if it could not be found.
    """

    entries: list[dict]
    counts: list[dict]
    source_updated: str | None = None

normalize_artifact_url(url: str) -> str

Return url with an explicit scheme and no trailing slash.

ArtiFinder sometimes records bare hosts (github.com/org/repo); prefix those with https://. Values that already carry a scheme are returned unchanged apart from trailing-slash trimming.

Source code in src/scrapers/artifinder.py

def normalize_artifact_url(url: str) -> str:
    """Return *url* with an explicit scheme and no trailing slash.

    ArtiFinder sometimes records bare hosts (``github.com/org/repo``); prefix
    those with ``https://``.  Values that already carry a scheme are returned
    unchanged apart from trailing-slash trimming.
    """
    u = (url or "").strip()
    if not u:
        return ""
    if not re.match(r"^[a-zA-Z][a-zA-Z0-9+.-]*://", u):
        u = "https://" + u.lstrip("/")
    return u.rstrip("/")

load_artifinder(conf_regex: str | None = None, min_year: int | None = DEFAULT_MIN_YEAR, local_dir: str | Path | None = None) -> ArtiFinderData

Download and parse the ArtiFinder data set.

Parameters:

Name	Type	Description	Default
conf_regex	str | None	Optional regex applied to f"{conf.lower()}{year}" (e.g. "usenixsec2023") to restrict which conference-years are loaded.	None
min_year	int | None	Earliest conference edition year to include (inclusive). Defaults to :data:DEFAULT_MIN_YEAR (2017, the artifact-evaluation era). Pass None to load the full history back to the 2000s.	DEFAULT_MIN_YEAR
local_dir	str | Path | None	Optional path to a local ArtiFinder-Data checkout (repo root or its data directory). Defaults to the REPRODB_ARTIFINDER_DIR environment variable. When present, the loader reads from disk and skips all network access.	None

Source code in src/scrapers/artifinder.py

def load_artifinder(
    conf_regex: str | None = None,
    min_year: int | None = DEFAULT_MIN_YEAR,
    local_dir: str | Path | None = None,
) -> ArtiFinderData:
    """Download and parse the ArtiFinder data set.

    Args:
        conf_regex: Optional regex applied to ``f"{conf.lower()}{year}"`` (e.g.
            ``"usenixsec2023"``) to restrict which conference-years are loaded.
        min_year: Earliest conference edition year to include (inclusive).
            Defaults to :data:`DEFAULT_MIN_YEAR` (2017, the artifact-evaluation
            era).  Pass ``None`` to load the full history back to the 2000s.
        local_dir: Optional path to a local ArtiFinder-Data checkout (repo root
            or its ``data`` directory).  Defaults to the ``REPRODB_ARTIFINDER_DIR``
            environment variable.  When present, the loader reads from disk and
            skips all network access.
    """
    base = _resolve_local_dir(local_dir)
    if base is not None:
        logger.info("  ArtiFinder: reading from local checkout %s", base)
        top = _local_venues(base)
    else:
        top = _list_dir(ARTIFINDER_API_BASE)
    if not top:
        logger.warning("  ArtiFinder: no data directories found (network issue?)")
        return ArtiFinderData([], [])

    all_entries: list[dict] = []
    counts: list[dict] = []
    for item in top:
        if item.get("type") != "dir":
            continue
        venue = item.get("name", "")
        mapping = _VENUE_MAP.get(venue.lower())
        if not mapping:
            logger.debug("  ArtiFinder: skipping unmapped venue %r", venue)
            continue
        conf, area = mapping

        files = _local_files(base / venue) if base is not None else _list_dir(f"{ARTIFINDER_API_BASE}/{venue}")
        for f in files:
            if f.get("type") != "file":
                continue
            m = _YEAR_RE.match(f.get("name", ""))
            if not m:
                continue
            year = int(m.group(1))
            if min_year is not None and year < min_year:
                continue
            if conf_regex and not re.search(conf_regex, f"{conf.lower()}{year}"):
                continue
            if base is not None:
                raw = _local_read(base / venue / f["name"])
            else:
                raw = download_file(f"{ARTIFINDER_RAW_BASE}/{venue}/{f['name']}")
            if not raw:
                continue
            year_entries, total = _parse_year_file(conf, area, year, raw)
            all_entries.extend(year_entries)
            if total:
                counts.append(
                    {
                        "conference": conf,
                        "category": area,
                        "year": year,
                        "total_papers": total,
                        "discovered": len(year_entries),
                    }
                )

    logger.info(
        "  ArtiFinder: loaded %d discovered artifacts across %d venues (%d conference-years scanned, min_year=%s)",
        len(all_entries),
        len({e["conference"] for e in all_entries}),
        len(counts),
        min_year,
    )
    source_updated = _local_source_updated(base) if base is not None else _remote_source_updated()
    return ArtiFinderData(all_entries, counts, source_updated)

load_artifinder_data(conf_regex: str | None = None, min_year: int | None = DEFAULT_MIN_YEAR, local_dir: str | Path | None = None) -> list[dict]

Convenience wrapper returning only the entries with discovered artifacts.

Source code in src/scrapers/artifinder.py

def load_artifinder_data(
    conf_regex: str | None = None,
    min_year: int | None = DEFAULT_MIN_YEAR,
    local_dir: str | Path | None = None,
) -> list[dict]:
    """Convenience wrapper returning only the entries with discovered artifacts."""
    return load_artifinder(conf_regex, min_year=min_year, local_dir=local_dir).entries