Evaluations¶
An evaluation object represents a collection of Synapse Entities that will be processed in a particular way. This could mean scoring Entries in a challenge or executing a processing pipeline.
Imports:
from synapseclient import Evaluation, Submission, SubmissionStatus
Evaluations can be retrieved by ID:
evaluation = syn.getEvaluation(1901877)
Like entities, evaluations are access controlled via ACLs. The synapseclient.Synapse.getPermissions()
and
synapseclient.Synapse.setPermissions()
methods work for evaluations:
access = syn.getPermissions(evaluation, user_id)
The synapseclient.Synapse.submit()
method returns a Submission object:
entity = syn.get(synapse_id)
submission = syn.submit(evaluation, entity, name='My Data', team='My Team')
The Submission object can then be used to check the status of the submission:
status = syn.getSubmissionStatus(submission)
- The status of a submission may be:
- INVALID the submitted entity is in the wrong format
- SCORED in the context of a challenge or competition
- OPEN indicating processing has not completed
- CLOSED indicating processing has completed
Submission status objects can be updated, usually by changing the status and score fields, and stored back to
Synapse using synapseclient.Synapse.store()
:
status.score = 0.99
status.status = 'SCORED'
status = syn.store(status)
See:
synapseclient.Synapse.getEvaluation()
synapseclient.Synapse.getEvaluationByContentSource()
synapseclient.Synapse.getEvaluationByName()
synapseclient.Synapse.submit()
synapseclient.Synapse.getSubmissions()
synapseclient.Synapse.getSubmission()
synapseclient.Synapse.getSubmissionStatus()
synapseclient.Synapse.getPermissions()
synapseclient.Synapse.setPermissions()
Evaluation¶
-
class
synapseclient.evaluation.
Evaluation
(**kwargs)¶ An Evaluation Submission queue, allowing submissions, retrieval and scoring.
Parameters: - name – Name of the evaluation
- description – A short description of the evaluation
- contentSource – Synapse Project associated with the evaluation
- submissionReceiptMessage – Message to display to users upon submission
- submissionInstructionsMessage – Message to display to users detailing acceptable formatting for submissions.
To create an Evaluation and store it in Synapse:
evaluation = syn.store(Evaluation( name="Q1 Final", description="Predict progression of MMSE scores for final scoring", contentSource="syn2290704"))
The contentSource field links the evaluation to its
synapseclient.entity.Project
. (Or, really, any synapse ID, but sticking to projects is a good idea.)Evaluations can be retrieved from Synapse by ID:
evaluation = syn.getEvaluation(1901877)
…by the Synapse ID of the content source (associated entity):
evaluation = syn.getEvaluationByContentSource('syn12345')
…or by the name of the evaluation:
evaluation = syn.getEvaluationByName('Foo Challenge Question 1')
-
__init__
(**kwargs)¶ Initialize self. See help(type(self)) for accurate signature.
Submission¶
-
class
synapseclient.evaluation.
Submission
(**kwargs)¶ Builds an Synapse submission object.
Parameters: - name – Name of submission
- entityId – Synapse ID of the Entity to submit
- evaluationId – ID of the Evaluation to which the Entity is to be submitted
- versionNumber – Version number of the submitted Entity
- submitterAlias – A pseudonym or team name for a challenge entry
-
__init__
(**kwargs)¶ Initialize self. See help(type(self)) for accurate signature.
Submission Status¶
-
class
synapseclient.evaluation.
SubmissionStatus
(**kwargs)¶ Builds an Synapse submission status object.
Parameters: - score – The score of the submission
- status – Status can be one of {‘OPEN’, ‘CLOSED’, ‘SCORED’, ‘INVALID’}.
-
__init__
(**kwargs)¶ Initialize self. See help(type(self)) for accurate signature.