Photo by Robin Mathlener on Unsplash

Artefact Evaluation

FORTE this year includes an artefact evaluation performed by the artefact evaluation committee (AEC). The goal of the artefact evaluation is to enable future researchers to more effectively build on and compare with previous work. The Artefact Evaluation Committee (AEC) will review the artefact. We will be attributing 3 badges, according to EAPLS guidelines:

  1. Artefact functional: documented, consistent, complete, exercisable;
  2. Artefact reusable: exceeding functional, by being carefully documented and well-structured for reuse and repurposing, see below for details;
  3. Artefact available: available on a publicly accessible archival repository to a permanent repository that provides a Digital Object Identifier (DOI).

Authors of regular papers are invited (but not required) to submit a relevant artefact for evaluation by the artefact evaluation committee (AEC). Authors of tool papers are required to submit an artefact to the AEC and this artefact should satisfy at least the requirements for the functional badge. Acceptance of tool papers is conditional on successful artefact evaluation.

Members of the artefact evaluation committee and the program committee are asked to use submitted artefacts for the sole purpose of evaluating the contribution associated with the artefact.

Important Dates

Submission Guidelines

Submission site: https://easychair.org/conferences/?conf=forte2024. Please select the “FORTE AEC 2024” track when making a new submission and use the same title (and pdf of the paper) as for the FORTE submission.

A final artefact submission should consist of

When uploading your artefact to the URL, please update the SHA256 checksum of the .zip file in the abstract. You can generate the checksum using the following command-line tools.

Packaging Guidelines

Your artefact .zip file must contain the following elements.

If you are not in a position to prepare the artefact as above, please contact PC chairs for an alternative arrangement. For instance, if you cannot provide us with a VM that contains licensed software, e.g., MatLab, please contact us, so we can find a solution.

Evaluation Criteria

All artefacts are evaluated by the artefact evaluation committee. Each artefact will be reviewed by at least two committee members, ideally three. Reviewers will read the paper and explore the artefact to evaluate how well the artefact supports the claims and results of the paper.

Criteria for the “functional” badge

The evaluation and the awarding of the functional badge is based on the following questions:

Criteria for the “available” badge

To get the available badge, please upload your VM to a permanent repository that provides a DOI, such as Zenodo, figshare, or Dryad and use this DOI link in your artefact submission.

Additional criteria for the “reusable” badge

Artefacts seeking the “reusable” badge need to clear a significantly higher bar than functional artefacts. First, they must be available, i.e., receive an “available” badge. Second, we expect a higher level of quality during the evaluation of the functional level. Third, in addition to the criteria from the functional level, they are evaluated against the following criteria:

Contact

For any questions please contact the two AE chairs: