Rubrics

Rubric clipart

Essay

General (30%)

Insufficient
There is not a clear link between some references and the content of the paper, or some references come from dubious sources.
Sufficient
The work includes relevant references that motivate the work.
Good
The work includes relevant references that motivate the work. There is a clear and brief explanation of the contributions of each related work.
Excellent
The work includes relevant references that motivate the work. There is a clear and brief explanation of the contributions of each related work. The paper pinpoints the differences between the proposed solution and the related work.

Soundness

Insufficient
There are arguments that are not technically sound.
Sufficient
The arguments laid out are technically sound but should have been better explained.
Good
The arguments laid out are technically sound. There are some arguments that lack adequate technical depth but those are not essential to the narrative.
Excellent
The arguments laid out are technically sound, and of adequate technical depth.

Background

Insufficient
Key concepts should have been presented in the paper.
Sufficient
There are some key concepts that should be defined but they do not affect the clarity of the paper.
Good
The paper presents key concepts related to the solution and respective validation.
Excellent
Clearly, concisely, and logically presents key concepts related to the solution and respective validation.

Format (30%)

Coherence

Insufficient
Sentences, paragraphs, and sections have a clear change in structure and style. This lack of coherence hinders the clarity of the narrative.
Sufficient
Sentences, paragraphs, and sections have a clear change in structure and style. However, this change in style does not hinder the clarity of the narrative.
Good
Sometimes, sentences, paragraphs, and sections have a slight change of style. However, this change in style does not hinder the clarity of the narrative.
Excellent
The text is well-structured. Sentences, paragraphs, and sections are coherent.

Development

Insufficient
The flow of narrative is a bit jumpy – i.e, often, the sections do not naturally build upon each other. The conclusion misses a few messages that were developed in the narrative.
Sufficient
The sections naturally build upon each other and work towards a clear message. However, in 1 or 2 sections, the flow of the narrative is broken. The conclusion misses a few messages that were developed in the narrative.
Good
The sections naturally build upon each other and work towards a clear message. However, in 1 or 2 sections, the flow of the narrative is sometimes broken. There is a compelling conclusion.
Excellent
The sections naturally build upon each other and work towards a clear message. There is a compelling conclusion.

Correctness

Insufficient
The text clearly misses proof reading. The clarity of the narrative is hindered by the grammatical and spelling errors.
Sufficient
The text has a few typos and grammatical mistakes. However, it is still easy to follow the narrative.
Good
The English writing is grammatically correct. The text is written in correct standard English, with complete sentences. There are a few typos here an there.
Excellent
The English writing is grammatically correct. The text is written in correct standard English, with complete sentences, and error-free.

Focus

Insufficient
The text has no clear goal and it is difficult to grasp the main takeaways.
Sufficient
Sometimes the goal of the text is not clear and some takeaways are not clear.
Good
Sometimes the goal of the text is not clear but it is easy to grasp the main takeaways.
Excellent
The text has a clear goal. It is easy to grasp the main takeaways.

Unit

Insufficient
Paragraphs often lack a clear main idea, hindering the flow and understandability of the narrative.
Sufficient
Some paragraphs deviate from their main idea (c.f. topic sentence).
Good
All paragraphs follow one main idea and does not deviate from it. The paper is not fully clear for someone that did not follow the course.
Excellent
All paragraphs follow one main idea and do not deviate from it. The paper is independently readable.

Graphics

Insufficient
The paper should have more graphics supporting the narrative, and their quality should be improved. Sometimes, the link between the text and the figures is missing.
Sufficient
There are a few ideas that could have been made clear with an illustrating graphic. Sometimes, the link between the text and the figures is missing.
Good
The story-line is illustrated with meaningful images and infographics. There is a clear link between the text and the figures.
Excellent
The story-line is illustrated with meaningful and appealing images and infographics. There is a clear link between the text and the figures.

Solution (40%)

Impact in the Process

How the solution fits the process-pipeline

Insufficient
It is not clear which part of the process/pipeline the solution is addressing.
Sufficient
It is clear which stage of the process/pipeline the solution is addressing. It is not clear how the proposed solution improves that stage.
Good
The solution provides a clear improvement in a particular stage of the ML development and release process.
Excellent
The solution provides a clear improvement in a particular stage of the ML development and release process. The connection between the improvement and the stage is justified with arguments of adequate technical depth.

Novelty

Insufficient
The solution already exists.
Sufficient
The solution is not necessarily innovative but it is useful for practitioners.
Good
The solution is relevant to the fields of Release Engineering or MLOps.
Excellent
The work creates a novel solution that pushes the state of the art of Release Engineering or MLOps.

Description

Insufficient
The solution is not well-described or presents limitations that should be addressed to solve the proposed problem.
Sufficient
The solution is well-described but some decisions are not well-grounded. Yet, it does not pose any serious limitation in its ability to the solve the proposed problem.
Good
The solution is well-described and technically sound. Sometimes, a clear contextualization referring to best practices is missing.
Excellent
The solution is well-described and technically sound, reflecting best practices in this area of study.

Validation

Insufficient
Even though the paper presents a validation of the solution, it is incomplete.
Sufficient
The paper presents a validation of the solution. The limitations of the validation are clearly described.
Good
The paper presents a thorough validation of the solution. The limitations of the validation are minimal and clearly described.
Excellent
The paper presents a thorough validation of the solution using a real-world context. The limitations of the validation are minimal and clearly described.

Generalizability

Insufficient
The solution is not ready to be used with the lab project.
Sufficient
The solution is fully integrated with the lab project.
Good
The solution is fully integrated with the lab project. On top of that, it is clear that the solution works with real-world projects.
Excellent
The solution is fully integrated with the lab project. On top of that, there are clear instructions on how to transfer the knowledge or apply the solution to real-world projects.

Final Pipeline

Pass/fail

To be considered as an appropriate submission, the final pipeline needs to fullfil the following requirements.

Grading

The pipeline extension introduces a substantial change over the tutorial pipeline.

Insufficient
Minor changes/extension compared to “tutorial pipeline”
Sufficient
Group has provided a feature that extends the existing pipeline with obvious value
Good
The provided feature is substantial, both conceptually and also from the required code/configuration.
Excellent
The provided feature touches multiple phases and/or has several subcomponents

The used technology is appropriate.

Insufficient
Integrating the extension is unnatural and breaks the workflows of a common Docker/Kubernetes setup, like the tutorial project.
Sufficient
Proposed extension fits into the workflow of a common Docker/Kubernetes setup.
Good
The pipeline extension naturally fits a Docker/Kubernetes setup and applies existing battle-proven technology or concepts.
Excellent
The pipeline extension naturally fits a Docker/Kubernetes setup and applies existing battle-proven technology or concepts in a new way or implements a new technique.

The pipeline extension can be ported to other projects/is usable by others.

Insufficient
Pipeline cannot be replicated
Sufficient
All necessary code and configurations are available in a public repository.
Good
Pipeline extension can be migrated to another project, the process is clearly documented.
Excellent
The pipeline extension is easy to configure and reusing/integrating it in other projects is made easy (e.g., by providing reusable/configurable containers or Maven goals).

The pipeline extension is automated and supports the release decision.

Insufficient
The extension requires manual execution.
Sufficient
The execution is automatically triggered, results have to be manually inspected (e.g., looking at a dashboard).
Good
The execution is automatically triggered, inspection of the results can be planned (e.g., in code review).
Excellent
Both the execution and the impact on the release decision are automated processes.