Fuzzing workshop 2022

April 24, 2022
Co-located with NDSS'22 @ San Diego, California

Fuzzing'22 Artifact Evaluation

• Goal of the Artifact Evaluation
• Badges and their meaning
• Important Dates
• Artifact Evaluation Process
• Submission Instructions
• Instructions for the short paper
• Instructions for the artifact
• Reviewing Guidelines
• Additional Information
• Artifact Evaluation Committee

Goal of the Artifact Evaluation

The two main goals of the artifact evaluation are:

1. the consistency check between the results reported in the research paper and the submitted artifact, and
2. the evaluation of the provided artifact concerning the official ACM evaluation badges.

All submissions should aim to get all three available badges (available, functional, reusable) to ensure the reproducibility of their research.

Badges and their meaning

Since the accepted full papers in Fuzzing'2022 will be published in TOSEM, we have the option to assign the official ACM badges for Available, Functional, and Reusable (https://www.acm.org/publications/policies/artifact-review-and-badging-current).

Artifacts Available v1.1

Author-created artifacts relevant to this paper have been placed on a publicly accessible archival repository. A DOI or link to this repository along with a unique identifier for the object is provided.

Artifacts Evaluated - Functional v1.1

The artifacts associated with the research are found to be documented, consistent, complete, exercisable, and include appropriate evidence of verification and validation.

Artifacts Evaluated - Reusable v1.1

Functional + The artifacts associated with the paper are of a quality that significantly exceeds minimal functionality. That is, they have all the qualities of the Artifacts Evaluated - Functional level, but, in addition, they are very carefully documented and well-structured to the extent that reuse and repurposing is facilitated. In particular, norms and standards of the research community for artifacts of this type are strictly adhered to.

Important Dates

Artifact Evaluation Process

This artifact evaluation is performed in three phases:

(Phase 1) Test phase: The Artifact Evaluation Committee (AEC) may contact the authors within the Test Phase to request clarifications on the basic installation and start-up procedures or to resolve simple installation problems. Given the short review time available, the authors are expected to respond within a 48-hour period but no later than September 12, 2022. Authors may update their research artifact after submission only for changes requested by reviewers and only until September 12, 2022. The authors submitting an open-source repository link are expected to give a tag to time-stamp your submission.

(Phase 2) First evaluation round: After the test phase, the AEC will conduct the full artifact evaluation and submit reviews and badge recommendations for each artifact. Afterwards, there will be the possibility for the authors to revise their artifacts and improve them according to the reviewer's feedback. The author response and artifact improvement deadline is October 2, 2022.

(Phase 3) Second evaluation round (if needed): If the AEC requested changes with their Phase 2 feedback, the AEC would conduct another round of reviews for the improved artifacts. Afterwards, the decision on the artifact badges will be finalized.

Note that HotCRP will serve as the main communication medium between authors and reviewers. The authors are expected to respond timely to the reviewer's questions, in particular, during Phase 1.

Submission Instructions

Instructions for the short paper

The short paper, i.e., the RCR report, should be titled: "[Paper Title] - RCR Report" where "[Paper Title]" is the title of the submitted TOSEM paper. RCR Reports shall use ACM journal format of short papers, and preferably comply with the following suggested structure:

• Abstract: Summary of the artifact and its relation to the paper
• Overview
• Paper:
• Summary of the submitted TOSEM Paper
• Key claims in the submitted TOSEM paper
• Key results in the submitted TOSEM paper
• Artifact(s):
• Table showing which artifacts are made available, where they are located (link/DOI), which license is used (e.g., CC-BY).
• Evidence generation: Prototype / tool, benchmarks, questionnaires
• Generated evidence: empirical results, survey results, labeled data,..
• Data analysis: Jupyter notebook, R/Pandas script..
• Prerequisites and Requirements
• E.g., Hardware or OS requirements
• OS req(e.g., platform, [pinned] dependencies)
• Setup (e.g., installation instructions)
• Steps to Reproduce
• How to generate the data (if possible)?
• How to analyze the data (if possible)?

Instructions for the artifact

Authors must perform the following steps to submit an artifact:

1. Prepare the artifact
2. Making the artifact available
3. Documenting the artifact
4. Submit the artifact

1. Prepare the artifact

The configuration and installation of the artifact should take less than 30 minutes. Otherwise, the artifact is unlikely to be endorsed.

The artifact shall include an installation package so the tool can be installed and run in commonly available environments. Provide enough associated instruction, code, and data such that some CS person with a reasonable knowledge of scripting, build tools, etc., could install, build, and run the code. If the artifact contains or requires a special tool or any other non-trivial piece of software, the authors shall provide a VirtualBox VM image or a Docker container image with a working environment containing the artifact and all the necessary tools.

We expect the artifacts to be vetted on a clean machine before submission.

2. Making the artifact available

The authors shall make the packaged artifact available so that the readers of the RCR report can access it. The digital artifact should be available on a long-term archival platform that comply with the FAIR principles (findable, accessible, interoperable, reusable, more details can be found here https://www.nature.com/sdata/policies/repositories#general). We strongly recommend either https://www.figshare.com or https://www.zenodo.org.

3. Documenting the artifact

The package shall include a documentation explaining how to (i) obtain the artifact package, (ii) unpack the artifact, (iii) get started, and (iv) use the artifacts. The artifact submission shall only describe the technicalities of the artifacts and uses of the artifact that are not already described in the RCR report and associated paper.

The submission should contain the following documents (in plain text or pdf format) in a zip archive:

• A README file describing what the artifact does and where it can be obtained (with hidden links and access password if necessary). Also, there should be a clear description of how to repeat/replicate/reproduce the results presented in the paper. Artifacts that focus on data should clearly describe all aspects relevant to understand the context, data provenance, ethical and legal statements (as long as relevant), and storage requirements. Artifacts that focus on software should clearly describe all aspects relevant to how to install and use it (and be accompanied by a small example).
• A REQUIREMENTS file for artifacts that focus on software. This file should, in principle, describe all aspects of hardware environment requirements (e.g., performance, storage, or non-commodity peripherals) and software environments (e.g., Docker, VM, and operating system), as well as a requirements.txt with explicit versioning information (e.g., for Python-only environments), when relevant. Any deviation from standard environments needs to be reasonably justified.
• A STATUS file stating what kind of badge(s) the authors are applying for and why the authors believe that the artifact deserves that badge(s). Note that all submissions need to apply for the "available" badge. Applying for the "functional" and "reusable" badges is strongly recommended.
• A LICENSE file describing the distribution rights. Due to the requirement of the “available” badge, that license must be some form of open source license.
• An INSTALL file with installation instructions. These instructions should include notes illustrating a very basic usage example or a method to test the installation. This could be, for instance, on what output to expect that confirms that the code is installed and working; and that the code is doing something interesting and useful.

4. Submit the artifact

By September 1, 2022, submit your research artifact at the Manuscript Central TOSEM website (choosing the RCR paper type) and by uploading the abstract describing your artifact. The abstract should include the DOI for your artifact, the paper title, the purpose of the research artifact, the badge(s) you are claiming, and the technology skills assumed by the reviewer evaluating the artifact. Please also mention if running your artifact requires specific Operation Systems or other environments.

After submitting your artifact to TOSEM, please also upload the same artifact to our HotCRP instance: https://fuzzing22-ae.hotcrp.com.

Reviewing Guidelines

For AEC members, we are currently creating custom reviewing guidelines, which will be based on the ESEC/FSE'2021 for the AE guidelines: https://2021.esec-fse.org/getImage/orig/fse_artifacts_submission_reviewing_guidelines.pdf.

Additional Information

If you need more input on how to produce a high-quality artifact, please check out the following additional resources:

• the blog post by Rohan Padhye, and
• the AE resource collection by Manuel Rigger.

In case of any questions, please do not hesitate to contact us!

Artifact Evaluation Committee

• Chair: Yannic Noller (National University of Singapore)


• David Ackermann, ETH Zurich
• Vincent Ahlrichs, Fraunhofer AISEC
• Carolin Brandt, Delft University of Technology
• Alexandra Bugariu, ETH Zurich
• Karine Even-Mendoza, Imperial College London
• Rahul Gopinath, University of Sydney
• Ahmad Hazimeh, EPFL
• Adrian Herrera, Australian National University
• Stephan Lukasczyk, University of Passau
• Numair Mansur, MPI-SWS
• Ruijie Meng, National University of Singapore
• Jonathan Metzman, Google
• Ridwan Shariffdeen, National University of Singapore
• Jingbo Wang, University of Southern California
• Chenxi Zhang, Fudan University