Release Notes¶
2.0.0 (2020-03-23)¶
This release requires Python 3.6+. Python 2 is no longer supported as of this release
Highlights:¶
Multi-threaded download of files from Syanpse can be toggled by setting
syn.multi_threaded=True
on asynapseclient.Synapse
object. This will become the default implementation in the future, but to ensure stability for the first release of this feature, it must be intentionally toggled.import synapseclient syn = synapseclient.login() syn.multi_threaded = True # syn123 now will be downloaded via the multi-threaded implementation syn.get("syn123")
Multi-threaded download currently will only work with files stored in AWS S3, where most files on Synapse reside. Files not stored in S3 will fall back to single-threaded download even if
syn.multi_threaded==True
synapseutils.copy()
will no longer copy entities that have access requirements nor entities to which the user does not have download permissionscontentTypes
andfileNames
are optional parameters insynapseutils.copyFileHandles()
Synapse Docker Repository(
synapseclient.DockerRepository
) objects can be used as theentity
argument insynapseclient.Synapse.submit()
. Optional argumentdocker_tag="latest"
has also been added tosynapseclient.Synapse.submit()
Full set of addressed are issues are listed below.
Bug¶
[SYNPY-271] - cache.remove fails to return the file handles we removed
[SYNPY-1032] - Support new columnTypes defined in backend
Task¶
[SYNPY-999] - Remove unsafe copy functions from client
[SYNPY-1027] - Copy function should copy things when users are part of a Team that has DOWNLOAD access
Improvement¶
[SYNPY-389] - submission of Docker repository
[SYNPY-537] - synapseutils.copyFileHandles requires fields that does not require at rest
[SYNPY-680] - synapseutils.changeFileMetaData() needs description in documentation
[SYNPY-682] - improve download speeds to be comparable to AWS
[SYNPY-807] - Drop support for Python 2
[SYNPY-907] - Replace `from <module> import …` with `import <module>`
[SYNPY-962] - remove ‘password’ as an option in default synapse config file
[SYNPY-972] - Link on Synapse Python Client Documentation points back at itself
1.9.4 (2019-06-28)¶
Bug¶
[SYNPY-881] - Synu.copy fails when copying a file with READ permissions
[SYNPY-888] - Docker repositories cannot be copied
[SYNPY-927] - trying to create a project with name that already exists hangs
[SYNPY-1005] - cli docs missing sub-commands
[SYNPY-1018] - Synu.copy shouldn’t copy any files with access restrictions
Improvement¶
[SYNPY-608] - Add how to contribute md to github project
[SYNPY-735] - command line for building a table
[SYNPY-864] - docstring for the command line client doesn’t have complete list of sub-commands available
[SYNPY-926] - allow forceVersion false for command line client
[SYNPY-1013] - Documentation of “store” command for Synapse command line client
[SYNPY-1021] - change email contact for code of conduct
1.9.3 (2019-06-28)¶
1.9.2 (2019-02-15)¶
In version 1.9.2, we improved Views’ usability by exposing set_entity_types() function to change the entity types that will show up in a View:
import synapseclient
from synapseclient.table import EntityViewType
syn = synapseclient.login()
view = syn.get("syn12345")
view.set_entity_types([EntityViewType.FILE, EntityViewType.FOLDER])
view = syn.store(view)
Bug Fixes¶
Tasks¶
1.9.1 (2019-01-20)¶
In version 1.9.1, we fix various bugs and added two new features:
Python 3.7 is supported.
Deprecation warnings are visible by default.
Features¶
Bug Fixes¶
[SYNPY-454] - Some integration tests do not clean up after themselves
[SYNPY-456] - Problems with updated query system
[SYNPY-515] - sphinx documentation not showing for some new classes
[SYNPY-526] - deprecate downloadTableFile()
[SYNPY-578] - switch away from POST /entity/#/table/deleterows
[SYNPY-594] - Getting error from dev branch in integration test against staging
[SYNPY-796] - fix or remove PyPI downloads badge in readme
[SYNPY-799] - Unstable test: Test PartialRow updates to entity views from rowset queries
[SYNPY-846] - error if password stored in config file contains a ‘%’
Tasks¶
[SYNPY-491] - Figure out custom release note fitlers
[SYNPY-840] - Install not working on latest python
[SYNPY-847] - uploadFileHandle should not be deprecated nor removed
[SYNPY-852] - Check and update docs.synapse.org to reflect the change in the Python client
[SYNPY-860] - vignette for how to upload a new version of a file directly to a synapse entity
[SYNPY-863] - Update public documentation to move away from the query services
[SYNPY-866] - Update Synapse logo in Python docs :)
[SYNPY-873] - consolidate integration testing to platform dev account
Improvements¶
[SYNPY-473] - Change syn.list to no longer use deprecated function chunkedQuery
[SYNPY-573] - synapse list command line shouldn’t list the parent container
[SYNPY-581] - <entity>.annotations return object is inconsistent with getAnnotations()
[SYNPY-612] - Rename view_type to viewType in EntityViewSchema for consistency
[SYNPY-777] - Python client _list still uses chunckedQuery and result seem out of date
[SYNPY-804] - Update styling in the python docs to more closely match the Docs site styling
[SYNPY-815] - Update the build to use test user instead of migrationAdmin
[SYNPY-848] - remove outdated link to confluence for command line query
[SYNPY-856] - build_table example in the docs does not look right
[SYNPY-858] - Write file view documentation in python client that is similar to synapser
[SYNPY-870] - Submitting to an evaluation queue can’t accept team as int
1.9.0 (2018-09-28)¶
In version 1.9.0, we deprecated and removed query() and chunkedQuery(). These functions used the old query services which does not perform well. To query for entities filter by annotations, please use EntityViewSchema.
We also deprecated the following functions and will remove them in Synapse Python client version 2.0. In the Activity object:
usedEntity()
usedURL()
In the Synapse object:
getEntity()
loadEntity()
createEntity()
updateEntity()
deleteEntity()
downloadEntity()
uploadFile()
uploadFileHandle()
uploadSynapseManagedFileHandle()
downloadTableFile()
Please see our documentation for more details on how to migrate your code away from these functions.
Bug Fixes¶
SYNPY-195 - Dangerous exception handling
SYNPY-261 - error downloading data from synapse (python client)
SYNPY-694 - Uninformative error in copyWiki function
SYNPY-805 - Uninformative error when getting View that does not exist
SYNPY-819 - command-line clients need to be updated to replace the EntityView ‘viewType’ with ‘viewTypeMask’
Tasks¶
Improvements¶
1.8.2 (2018-08-17)¶
In this release, we have been performed some house-keeping on the code base. The two major changes are:
making syn.move() available to move an entity to a new parent in Synapse. For example:
import synapseclient from synapseclient import Folder syn = synapseclient.login() file = syn.get("syn123") folder = Folder("new folder", parent="syn456") folder = syn.store(folder) # moving file to the newly created folder syn.move(file, folder)exposing the ability to use the Synapse Python client with single threaded. This feature is useful when running Python script in an environment that does not support multi-threading. However, this will negatively impact upload speed. To use single threaded:
import synapseclient synapseclient.config.single_threaded = True
Bug Fixes¶
SYNPY-535 - Synapse Table update: Connection Reset
SYNPY-603 - Python client and synapser cannot handle table column type LINK
SYNPY-688 - Recursive get (sync) broken for empty folders.
SYNPY-744 - KeyError when trying to download using Synapse Client 1.8.1
SYNPY-750 - Error in downloadTableColumns for file view
SYNPY-758 - docs in Sphinx don’t show for synapseclient.table.RowSet
SYNPY-760 - Keyring related error on Linux
SYNPY-766 - as_table_columns() returns a list of columns out of order for python 3.5 and 2.7
SYNPY-776 - Cannot log in to Synapse - error(54, ‘Connection reset by peer’)
SYNPY-795 - Not recognizable column in query result
Features¶
Tasks¶
1.8.1 (2018-05-17)¶
This release is a hotfix for a bug. Please refer to 1.8.0 release notes for information about additional changes.
1.8.0 (2018-05-07)¶
This release has 2 major changes:
The client will no longer store your saved credentials in your synapse cache (~/synapseCache/.session). The python client now relies on keyring to handle credential storage of your Synapse credentials.
The client also now uses connection pooling, which means that all method calls that connect to Synapse should now be faster.
The remaining changes are bug fixes and cleanup of test code.
Below are the full list of issues addressed by this release:
Bug Fixes¶
Tasks¶
1.7.5 (2018-01-31)¶
v1.7.4 release was broken for new users that installed from pip. v1.7.5 has the same changes as v1.7.4 but fixes the pip installation.
1.7.4 (2018-01-29)¶
- This release mostly includes bugfixes and improvements for various Table classes:
Fixed bug where you couldn’t store a table converted to a pandas.Dataframe if it had a INTEGER column with some missing values.
EntityViewSchema can now automatically add all annotations within your defined scopes as columns. Just set the view’s addAnnotationColumns=True before calling syn.store(). This attribute defaults to True for all newly created EntityViewSchemas. Setting addAnnotationColumns=True on existing tables will only add annotation columns that are not already a part of your schema.
You can now use synapseutils.notifyMe as a decorator to notify you by email when your function has completed. You will also be notified of any Errors if they are thrown while your function runs.
- We also added some new features:
syn.findEntityId() function that allows you to find an Entity by its name and parentId, set parentId to None to search for Projects by name.
The bulk upload functionality of synapseutils.syncToSynapse is available from the command line using: synapse sync.
Below are the full list of issues addressed by this release:
Improvements¶
SYNPY-267 - Update Synapse tables for integer types
SYNPY-304 - Table objects should implement len()
SYNPY-416 - warning message for recursive get when a non-Project of Folder entity is passed
SYNPY-482 - Create a sample synapseConfig if none is present
SYNPY-489 - Add a boolean parameter in EntityViewSchema that will indicate whether the client should create columns based on annotations in the specified scopes
SYNPY-494 - Link should be able to take an entity object as the parameter and derive its id
SYNPY-511 - improve exception handling
SYNPY-512 - Remove the use of PaginatedResult’s totalNumberOfResult
SYNPY-539 - When creating table Schemas, enforce a limit on the number of columns that can be created.
Bug Fixes¶
SYNPY-235 - can’t print Row objects with dates in them
SYNPY-272 - bug syn.storing rowsets containing Python datetime objects
SYNPY-297 - as_table_columns shouldn’t give fractional max size
SYNPY-404 - when we get a SynapseMd5MismatchError we should delete the downloaded file
SYNPY-425 - onweb doesn’t work for tables
SYNPY-438 - Need to change ‘submit’ not to use evaluation/id/accessRequirementUnfulfilled
SYNPY-496 - monitor.NotifyMe can not be used as an annotation decorator
SYNPY-521 - inconsistent error message when username/password is wrong on login
SYNPY-536 - pre-signed upload URL expired warnings using Python client sync function
SYNPY-555 - EntityViewSchema is missing from sphinx documentation
SYNPY-558 - synapseutils.sync.syncFromSynapse throws error when syncing a Table object
SYNPY-595 - Get recursive folders filled with Links fails
SYNPY-605 - Update documentation for getUserProfile to include information about refreshing and memoization
1.7.3 (2017-12-08)¶
Release 1.7.3 introduces fixes and quality of life changes to Tables and synapseutils:
Changes to Tables:
You no longer have to include the etag column in your SQL query when using a tableQuery() to update File/Project Views. just SELECT the relevant columns and etags will be resolved automatically.
The new PartialRowSet class allows you to only have to upload changes to individual cells of a table instead of every row that had a value changed. It is recommended to use the PartialRowSet.from_mapping() classmethod instead of the PartialRowSet constructor.
Changes to synapseutils:
Improved documentation
You can now use ~ to refer to your home directory in your manifest.tsv
We also added improved debug logging and use Python’s builtin logging module instead of printing directly to sys.stderr
Below are the full list of issues addressed by this release:
Bug Fixes¶
SYNPY-419 - support object store from client
SYNPY-499 - metadata manifest file name spelled wrong
SYNPY-504 - downloadTableFile changed return type with no change in documentation or mention in release notes
SYNPY-508 - syncToSynapse does not work if “the file path in “used” or “executed” of the manifest.tsv uses home directory shortcut “~”
SYNPY-516 - synapse sync file does not work if file is a URL
SYNPY-525 - Download CSV file of Synapse Table - 416 error
SYNPY-572 - Users should only be prompted for updates if the first or second part of the version number is changed.
Features¶
1.7.1 (2017-11-17)¶
Release 1.7 is a large bugfix release with several new features. The main ones include:
We have expanded the synapseutils packages to add the ability to:
Bulk upload files to synapse (synapseutils.syncToSynapse).
Notify you via email on the progress of a function (useful for jobs like large file uploads that may take a long time to complete).
The syncFromSynapse function now creates a “manifest” which contains the metadata of downloaded files. (These can also be used to update metadata with the bulk upload function.
File View tables can now be created from the python client using EntityViewSchema. See fileviews documentation.
The python client is now able to upload to user owned S3 Buckets. Click here for instructions on linking your S3 bucket to synapse.
We’ve also made various improvements to existing features:
The LARGETEXT type is now supported in Tables allowing for strings up to 2Mb.
The –description argument when creating/updating entities from the command line client will now create a Wiki for that entity. You can also use –descriptionFile to write the contents of a markdown file as the entity’s Wiki
Two member variables of the File object, file_entity.cacheDir and file_entity.files is being DEPRECATED in favor of file_entity.path for finding the location of a downloaded File
pandas dataframe`s containing `datetime values can now be properly converted into csv and uploaded to Synapse.
We also added a optional convert_to_datetime parameter to CsvFileTable.asDataFrame() that will automatically convert Synapse DATE columns into datetime objects instead of leaving them as long unix timestamps
Below are the full list of bugs and issues addressed by this release:
Features¶
SYNPY-53 - support syn.get of external FTP links in py client
SYNPY-179 - Upload to user owned S3 bucket
SYNPY-412 - allow query-based download based on view tables from command line client
SYNPY-487 - Add remote monitoring of long running processes
SYNPY-415 - Add Docker and TableViews into Entity.py
SYNPY-89 - Python client: Bulk upload client/command
SYNPY-413 - Update table views via python client
SYNPY-301 - change actual file name from python client
SYNPY-442 - set config file path on command line
Improvements¶
SYNPY-407 - support LARGETEXT in tables
SYNPY-360 - Duplicate file handles are removed from BulkFileDownloadRequest
SYNPY-187 - Move –description in command line client to create wikis
SYNPY-224 - When uploading to a managed external file handle (e.g. SFTP), fill in storageLocationId
SYNPY-315 - Default behavior for files in cache dir should be replace
SYNPY-381 - Remove references to “files” and “cacheDir”.
SYNPY-396 - Create filehandle copies in synapseutils.copy instead of downloading
SYNPY-403 - Use single endpoint for all downloads
SYNPY-435 - Convenience function for new service to get entity’s children
SYNPY-471 - docs aren’t generated for synapseutils
SYNPY-472 - References to wrong doc site
SYNPY-347 - Missing dtypes in table.DTYPE_2_TABLETYPE
SYNPY-463 - When copying filehandles we should add the files to the cache if we already donwloaded them
SYNPY-475 - Store Tables timeout error
Bug Fixes¶
SYNPY-190 - syn.login(‘asdfasdfasdf’) should fail
SYNPY-344 - weird cache directories
SYNPY-346 - ValueError: cannot insert ROW_ID, already exists in CsvTableFile constructor
SYNPY-351 - Versioning broken for sftp files
SYNPY-366 - file URLs leads to wrong path
SYNPY-393 - New cacheDir causes cache to be ignored(?)
SYNPY-409 - Python client cannot depend on parsing Amazon pre-signed URLs
SYNPY-418 - Integration test failure against 167
SYNPY-421 - syn.getWikiHeaders has a return limit of 50 (Need to return all headers)
SYNPY-423 - upload rate is off or incorrect
SYNPY-424 - File entities don’t handle local_state correctly for setting datafilehandleid
SYNPY-426 - multiple tests failing because of filenameOveride
SYNPY-427 - test dependent on config file
SYNPY-428 - sync function error
SYNPY-431 - download ending early and not restarting from previous spot
SYNPY-443 - tests/integration/integration_test_Entity.py:test_get_with_downloadLocation_and_ifcollision AssertionError
SYNPY-461 - On Windows, command line client login credential prompt fails (python 2.7)
SYNPY-465 - Update tests that set permissions to also include ‘DOWNLOAD’ permission and tests that test functions using queries
SYNPY-468 - Command line client incompatible with cache changes
SYNPY-470 - default should be read, download for setPermissions
SYNPY-483 - integration test fails for most users
SYNPY-484 - URL expires after retries
SYNPY-486 - Error in integration tests
SYNPY-488 - sync tests for command line client puts file in working directory
SYNPY-142 - PY: Error in login with rememberMe=True
SYNPY-464 - synapse get syn4988808 KeyError: u’preSignedURL’
1.6.1 (2016-11-02)¶
What is New¶
In version 1.6 we introduce a new sub-module _synapseutils_ that provide convenience functions for more complicated operations in Synapse such as copying of files wikis and folders. In addition we have introduced several improvements in downloading content from Synapse. As with uploads we are now able to recover from an interrupted download and will retry on network failures.
Improvements¶
We have improved download robustness and error checking, along with extensive recovery on failed operations. This includes the ability for the client to pause operation when Synapse is updated.
SYNPY-270 - Synapse READ ONLY mode should cause pause in execution
SYNPY-308 - Add md5 checking after downloading a new file handle
SYNPY-309 - Add download recovery by using the ‘Range’: ‘bytes=xxx-xxx’ header
SYNPY-353 - Speed up downloads of fast connections
SYNPY-356 - Add support for version flag in synapse cat command line
SYNPY-357 - Remove failure message on retry in multipart_upload
SYNPY-380 - Add speed meter to downloads/uploads
SYNPY-387 - Do exponential backoff on 429 status and print explanatory error message from server
SYNPY-390 - Move recursive download to Python client utils
Bug Fixes¶
SYNPY-154 - 500 Server Error when storing new version of file from command line
SYNPY-168 - Failure on login gives an ugly error message
SYNPY-253 - Error messages on upload retry inconsistent with behavior
SYNPY-261 - error downloading data from synapse (python client)
SYNPY-274 - Trying to use the client without logging in needs to give a reasonable error
SYNPY-331 - test_command_get_recursive_and_query occasionally fails
SYNPY-337 - Download error on 10 Gb file.
SYNPY-343 - Login failure
SYNPY-351 - Versioning broken for sftp files
SYNPY-352 - file upload max retries exceeded messages
SYNPY-358 - upload failure from python client (threading)
SYNPY-361 - file download fails midway without warning/error
SYNPY-362 - setAnnotations bug when given synapse ID
SYNPY-363 - problems using provenance during upload
SYNPY-382 - Python client is truncating the entity id in download csv from table
SYNPY-383 - Travis failing with paramiko.ssh_exception.SSHException: No hostkey
SYNPY-384 - resuming a download after a ChunkedEncodingError created new file with correct size
SYNPY-388 - Asynchronous creation of Team causes sporadic test failure
SYNPY-391 - downloadTableColumns() function doesn’t work when resultsAs=rowset is set for for syn.tableQuery()
SYNPY-397 - Error in syncFromSynapse() integration test on Windows
SYNPY-399 - python client not compatible with newly released Pandas 0.19