Synapse Multipart Upload¶
Implements the client side of Synapse multipart upload, which provides a robust means of uploading large files (into the 10s of GB). End users should not need to call any of these functions directly.
-
synapseclient.multipart_upload.
calculate_part_size
(fileSize, partSize=None, min_part_size=8388608, max_parts=10000)¶ Parts for multipart upload must be at least 5 MB and there must be at most 10,000 parts
-
synapseclient.multipart_upload.
count_completed_parts
(part_status)¶ Given a string of the form “1001110”, where 1 and 0 indicate a status of completed or not, return the count of parts already completed.
-
synapseclient.multipart_upload.
find_parts_to_upload
(part_status)¶ Given a string of the form “1001110”, where 1 and 0 indicate a status of completed or not, return the part numbers that aren’t completed.
-
synapseclient.multipart_upload.
get_data_chunk
(data, n, chunksize=8388608)¶ Return the nth chunk of a buffer.
-
synapseclient.multipart_upload.
get_file_chunk
(filepath, n, chunksize=8388608)¶ Read the nth chunk from the file.
-
synapseclient.multipart_upload.
multipart_upload
(syn, filepath, filename=None, contentType=None, storageLocationId=None, **kwargs)¶ Upload a file to a Synapse upload destination in chunks.
Parameters: - syn – a Synapse object
- filepath – the file to upload
- filename – upload as a different filename
- contentType – contentType
- partSize – number of bytes per part. Minimum 5MB.
- storageLocationId – a id indicating where the file should be stored. Retrieved from Synapse’s UploadDestination
Returns: a File Handle ID
Keyword arguments are passed down to
_multipart_upload()
and_start_multipart_upload()
.
-
synapseclient.multipart_upload.
multipart_upload_string
(syn, text, filename=None, contentType=None, storageLocationId=None, **kwargs)¶ Upload a string using the multipart file upload.
Parameters: - syn – a Synapse object
- text – a string to upload as a file.
- filename – a string containing the base filename
- contentType – contentType
- partSize – number of bytes per part. Minimum 5MB.
- storageLocationId – a id indicating where the text should be stored. Retrieved from Synapse’s UploadDestination
Returns: a File Handle ID
Keyword arguments are passed down to
_multipart_upload()
and_start_multipart_upload()
.
-
synapseclient.multipart_upload.
_multipart_upload
(syn, filename, contentType, get_chunk_function, md5, fileSize, partSize=None, storageLocationId=None, **kwargs)¶ Multipart Upload.
Parameters: - syn – a Synapse object
- filename – a string containing the base filename
- contentType – contentType
- get_chunk_function – a function that takes a part number and size and returns the bytes of that chunk of the file
- md5 – the part’s MD5 as hex.
- fileSize – total number of bytes
- partSize – number of bytes per part. Minimum 5MB.
- storageLocationId – a id indicating where the file should be stored. retrieved from Synapse’s UploadDestination
Returns: a MultipartUploadStatus object
Keyword arguments are passed down to
_start_multipart_upload()
.
-
synapseclient.multipart_upload.
_start_multipart_upload
(syn, filename, md5, fileSize, partSize, contentType, preview=True, storageLocationId=None, forceRestart=False)¶ Returns: A MultipartUploadStatus