[{"url":"https://api.github.com/repos/huggingface/evaluate/releases/248299087","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/248299087/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/248299087/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.4.6","id":248299087,"author":{"login":"lhoestq","id":42851186,"node_id":"MDQ6VXNlcjQyODUxMTg2","avatar_url":"https://avatars.githubusercontent.com/u/42851186?v=4","gravatar_id":"","url":"https://api.github.com/users/lhoestq","html_url":"https://github.com/lhoestq","followers_url":"https://api.github.com/users/lhoestq/followers","following_url":"https://api.github.com/users/lhoestq/following{/other_user}","gists_url":"https://api.github.com/users/lhoestq/gists{/gist_id}","starred_url":"https://api.github.com/users/lhoestq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lhoestq/subscriptions","organizations_url":"https://api.github.com/users/lhoestq/orgs","repos_url":"https://api.github.com/users/lhoestq/repos","events_url":"https://api.github.com/users/lhoestq/events{/privacy}","received_events_url":"https://api.github.com/users/lhoestq/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4OzL5P","tag_name":"v0.4.6","target_commitish":"main","name":"v0.4.6","draft":false,"immutable":false,"prerelease":false,"created_at":"2025-09-18T13:05:57Z","updated_at":"2025-09-18T13:07:20Z","published_at":"2025-09-18T13:07:20Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.4.6","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.4.6","body":"## What's Changed\r\n* Remove deprecated `HfFolder` by @Wauplin in https://github.com/huggingface/evaluate/pull/701\r\n  * this change adds support for `huggingface_hub>=1.0`\r\n* Update index.mdx by @meg-huggingface in https://github.com/huggingface/evaluate/pull/694\r\n* Fix parity tests ci by @lhoestq in https://github.com/huggingface/evaluate/pull/696\r\n* add leaderboards to docs by @burtenshaw in https://github.com/huggingface/evaluate/pull/697* Pin hfh in CI for updating repos by @lhoestq in https://github.com/huggingface/evaluate/pull/702\r\n\r\n## New Contributors\r\n* @burtenshaw made their first contribution in https://github.com/huggingface/evaluate/pull/697\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.4.5...v0.4.6","mentions_count":4},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/231469025","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/231469025/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/231469025/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.4.5","id":231469025,"author":{"login":"lhoestq","id":42851186,"node_id":"MDQ6VXNlcjQyODUxMTg2","avatar_url":"https://avatars.githubusercontent.com/u/42851186?v=4","gravatar_id":"","url":"https://api.github.com/users/lhoestq","html_url":"https://github.com/lhoestq","followers_url":"https://api.github.com/users/lhoestq/followers","following_url":"https://api.github.com/users/lhoestq/following{/other_user}","gists_url":"https://api.github.com/users/lhoestq/gists{/gist_id}","starred_url":"https://api.github.com/users/lhoestq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lhoestq/subscriptions","organizations_url":"https://api.github.com/users/lhoestq/orgs","repos_url":"https://api.github.com/users/lhoestq/repos","events_url":"https://api.github.com/users/lhoestq/events{/privacy}","received_events_url":"https://api.github.com/users/lhoestq/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4Ny-_h","tag_name":"v0.4.5","target_commitish":"main","name":"v0.4.5","draft":false,"immutable":false,"prerelease":false,"created_at":"2025-07-10T13:26:13Z","updated_at":"2025-07-10T13:26:45Z","published_at":"2025-07-10T13:26:45Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.4.5","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.4.5","body":"## What's Changed\r\n* Support datasets 4 by @lhoestq in https://github.com/huggingface/evaluate/pull/689\r\n\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.4.4...v0.4.5","mentions_count":1},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/226756613","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/226756613/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/226756613/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.4.4","id":226756613,"author":{"login":"lhoestq","id":42851186,"node_id":"MDQ6VXNlcjQyODUxMTg2","avatar_url":"https://avatars.githubusercontent.com/u/42851186?v=4","gravatar_id":"","url":"https://api.github.com/users/lhoestq","html_url":"https://github.com/lhoestq","followers_url":"https://api.github.com/users/lhoestq/followers","following_url":"https://api.github.com/users/lhoestq/following{/other_user}","gists_url":"https://api.github.com/users/lhoestq/gists{/gist_id}","starred_url":"https://api.github.com/users/lhoestq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lhoestq/subscriptions","organizations_url":"https://api.github.com/users/lhoestq/orgs","repos_url":"https://api.github.com/users/lhoestq/repos","events_url":"https://api.github.com/users/lhoestq/events{/privacy}","received_events_url":"https://api.github.com/users/lhoestq/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4NhAgF","tag_name":"v0.4.4","target_commitish":"main","name":"v0.4.4","draft":false,"immutable":false,"prerelease":false,"created_at":"2025-06-20T17:47:52Z","updated_at":"2025-06-20T17:49:29Z","published_at":"2025-06-20T17:49:29Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.4.4","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.4.4","body":"## Bug fixes\r\n* support jiwer 4.0 by @lhoestq in https://github.com/huggingface/evaluate/pull/685\r\n* Fix Perplexity Score For Tokenizers without bos_token_id by @kylehowells in https://github.com/huggingface/evaluate/pull/682\r\n* Fix size attribute error for precision/recall/f1  by @Maxwell-Jia in https://github.com/huggingface/evaluate/pull/656\r\n\r\n## Other changes\r\n* Add required hf_token secret to build main documentation by @albertvillanova in https://github.com/huggingface/evaluate/pull/635\r\n* Pin numpy<2 as required by tensorflow to fix doc building by @albertvillanova in https://github.com/huggingface/evaluate/pull/631\r\n* Support nltk>=3.9 to fix vulnerability by @albertvillanova in https://github.com/huggingface/evaluate/pull/629\r\n* add tip in docs and readme referring to lighteval by @MoritzLaurer in https://github.com/huggingface/evaluate/pull/618\r\n\r\n## New Contributors\r\n* @MoritzLaurer made their first contribution in https://github.com/huggingface/evaluate/pull/618\r\n* @Maxwell-Jia made their first contribution in https://github.com/huggingface/evaluate/pull/656\r\n* @kylehowells made their first contribution in https://github.com/huggingface/evaluate/pull/682\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.4.3...v0.4.4","mentions_count":5},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/174516840","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/174516840/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/174516840/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.4.3","id":174516840,"author":{"login":"lhoestq","id":42851186,"node_id":"MDQ6VXNlcjQyODUxMTg2","avatar_url":"https://avatars.githubusercontent.com/u/42851186?v=4","gravatar_id":"","url":"https://api.github.com/users/lhoestq","html_url":"https://github.com/lhoestq","followers_url":"https://api.github.com/users/lhoestq/followers","following_url":"https://api.github.com/users/lhoestq/following{/other_user}","gists_url":"https://api.github.com/users/lhoestq/gists{/gist_id}","starred_url":"https://api.github.com/users/lhoestq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lhoestq/subscriptions","organizations_url":"https://api.github.com/users/lhoestq/orgs","repos_url":"https://api.github.com/users/lhoestq/repos","events_url":"https://api.github.com/users/lhoestq/events{/privacy}","received_events_url":"https://api.github.com/users/lhoestq/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4KZupo","tag_name":"v0.4.3","target_commitish":"main","name":"0.4.3","draft":false,"immutable":false,"prerelease":false,"created_at":"2024-09-11T10:15:03Z","updated_at":"2024-09-11T10:17:30Z","published_at":"2024-09-11T10:17:30Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.4.3","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.4.3","body":"This release adds support for `datasets>=3.0` by removing calls to deprecated code\r\n\r\n## What's Changed\r\n* Fix CI with temporary pin nltk<3.9 by @albertvillanova in https://github.com/huggingface/evaluate/pull/623\r\n* Replace deprecated use_auth_token with token by @albertvillanova in https://github.com/huggingface/evaluate/pull/621\r\n* remove ignore_url_params by @lhoestq in https://github.com/huggingface/evaluate/pull/624\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.4.2...v0.4.3","mentions_count":2},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/153460461","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/153460461/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/153460461/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.4.2","id":153460461,"author":{"login":"lhoestq","id":42851186,"node_id":"MDQ6VXNlcjQyODUxMTg2","avatar_url":"https://avatars.githubusercontent.com/u/42851186?v=4","gravatar_id":"","url":"https://api.github.com/users/lhoestq","html_url":"https://github.com/lhoestq","followers_url":"https://api.github.com/users/lhoestq/followers","following_url":"https://api.github.com/users/lhoestq/following{/other_user}","gists_url":"https://api.github.com/users/lhoestq/gists{/gist_id}","starred_url":"https://api.github.com/users/lhoestq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lhoestq/subscriptions","organizations_url":"https://api.github.com/users/lhoestq/orgs","repos_url":"https://api.github.com/users/lhoestq/repos","events_url":"https://api.github.com/users/lhoestq/events{/privacy}","received_events_url":"https://api.github.com/users/lhoestq/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4JJZ7t","tag_name":"v0.4.2","target_commitish":"main","name":"v0.4.2","draft":false,"immutable":false,"prerelease":false,"created_at":"2024-04-30T09:43:44Z","updated_at":"2024-04-30T09:45:36Z","published_at":"2024-04-30T09:45:36Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.4.2","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.4.2","body":"## What's Changed\r\n* Update the documentation and citation of mauve by @krishnap25 in https://github.com/huggingface/evaluate/pull/416\r\n* Remove unused dependency by @daskol in https://github.com/huggingface/evaluate/pull/507\r\n* Add confusion matrix by @osanseviero in https://github.com/huggingface/evaluate/pull/528\r\n* Update python to 3.8 by @qubvel in https://github.com/huggingface/evaluate/pull/571\r\n* Fix FileFreeLock by @lhoestq in https://github.com/huggingface/evaluate/pull/578\r\n* Fix example doc in load function by @alexrs in https://github.com/huggingface/evaluate/pull/575\r\n* Speeding up mean_iou metric computation by @qubvel in https://github.com/huggingface/evaluate/pull/569\r\n\r\n## New Contributors\r\n* @rtrompier made their first contribution in https://github.com/huggingface/evaluate/pull/510\r\n* @daskol made their first contribution in https://github.com/huggingface/evaluate/pull/507\r\n* @qubvel made their first contribution in https://github.com/huggingface/evaluate/pull/571\r\n* @alexrs made their first contribution in https://github.com/huggingface/evaluate/pull/575\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.4.1...v0.4.2","mentions_count":7},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/125008387","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/125008387/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/125008387/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.4.1","id":125008387,"author":{"login":"lhoestq","id":42851186,"node_id":"MDQ6VXNlcjQyODUxMTg2","avatar_url":"https://avatars.githubusercontent.com/u/42851186?v=4","gravatar_id":"","url":"https://api.github.com/users/lhoestq","html_url":"https://github.com/lhoestq","followers_url":"https://api.github.com/users/lhoestq/followers","following_url":"https://api.github.com/users/lhoestq/following{/other_user}","gists_url":"https://api.github.com/users/lhoestq/gists{/gist_id}","starred_url":"https://api.github.com/users/lhoestq/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lhoestq/subscriptions","organizations_url":"https://api.github.com/users/lhoestq/orgs","repos_url":"https://api.github.com/users/lhoestq/repos","events_url":"https://api.github.com/users/lhoestq/events{/privacy}","received_events_url":"https://api.github.com/users/lhoestq/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4Hc3oD","tag_name":"v0.4.1","target_commitish":"main","name":"v0.4.1","draft":false,"immutable":false,"prerelease":false,"created_at":"2023-10-13T15:54:18Z","updated_at":"2023-10-13T15:57:18Z","published_at":"2023-10-13T15:57:18Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.4.1","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.4.1","body":"## What's Changed\r\n* Add code example to docstrings by @stevhliu in https://github.com/huggingface/evaluate/pull/374\r\n* [Minor fix] Typo by @cakiki in https://github.com/huggingface/evaluate/pull/403\r\n* [Docs] fixed a typo in bertscore readme by @hazrulakmal in https://github.com/huggingface/evaluate/pull/386\r\n* Add max_length kwarg to docstring of Perplexity measurement by @kdutia in https://github.com/huggingface/evaluate/pull/411\r\n* Fix minor typo in a_quick_tour.mdx by @tupini07 in https://github.com/huggingface/evaluate/pull/417\r\n* Fix Docs base_evaluator.mdx by @jorahn in https://github.com/huggingface/evaluate/pull/418\r\n* Update Gradio description to clarify text-based input by @BramVanroy in https://github.com/huggingface/evaluate/pull/427\r\n* fix `add` method by @hazrulakmal in https://github.com/huggingface/evaluate/pull/424\r\n* Fix broken link in docs/a_quick_tour.mdx by @tupini07 in https://github.com/huggingface/evaluate/pull/419\r\n* resolve #379 audio classification evaluator + docs  by @Plutone11011 in https://github.com/huggingface/evaluate/pull/405\r\n* fixed kwargs not being passed in combine by @Plutone11011 in https://github.com/huggingface/evaluate/pull/425\r\n* add r^2  metric by @TKaanKoc in https://github.com/huggingface/evaluate/pull/407\r\n* Update spaces gradio version to 3.19.1 by @BramVanroy in https://github.com/huggingface/evaluate/pull/426\r\n* replace evaluate DownloadConfig with datasets by @lvwerra in https://github.com/huggingface/evaluate/pull/447\r\n* Render Text2TextGenerationEvaluators' docstring examples by @mariosasko in https://github.com/huggingface/evaluate/pull/463\r\n* Trigger CI on ci-* branches by @Wauplin in https://github.com/huggingface/evaluate/pull/467\r\n* Update comet by @ricardorei in https://github.com/huggingface/evaluate/pull/443\r\n* Fix `datasets` import in Meteor metric by @mariosasko in https://github.com/huggingface/evaluate/pull/490\r\n* fix scikit-learn package name suggestion by @bzz in https://github.com/huggingface/evaluate/pull/498\r\n* Release: 0.4.1 by @lhoestq in https://github.com/huggingface/evaluate/pull/505\r\n\r\n## New Contributors\r\n* @cakiki made their first contribution in https://github.com/huggingface/evaluate/pull/403\r\n* @hazrulakmal made their first contribution in https://github.com/huggingface/evaluate/pull/386\r\n* @kdutia made their first contribution in https://github.com/huggingface/evaluate/pull/411\r\n* @tupini07 made their first contribution in https://github.com/huggingface/evaluate/pull/417\r\n* @jorahn made their first contribution in https://github.com/huggingface/evaluate/pull/418\r\n* @Plutone11011 made their first contribution in https://github.com/huggingface/evaluate/pull/405\r\n* @TKaanKoc made their first contribution in https://github.com/huggingface/evaluate/pull/407\r\n* @mariosasko made their first contribution in https://github.com/huggingface/evaluate/pull/463\r\n* @Wauplin made their first contribution in https://github.com/huggingface/evaluate/pull/467\r\n* @ricardorei made their first contribution in https://github.com/huggingface/evaluate/pull/443\r\n* @bzz made their first contribution in https://github.com/huggingface/evaluate/pull/498\r\n* @lhoestq made their first contribution in https://github.com/huggingface/evaluate/pull/505\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.4.0...v0.4.1","mentions_count":15},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/85867573","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/85867573/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/85867573/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.4.0","id":85867573,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4FHjw1","tag_name":"v0.4.0","target_commitish":"main","name":"v0.4.0","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-12-13T09:37:13Z","updated_at":"2022-12-13T13:35:51Z","published_at":"2022-12-13T13:35:51Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.4.0","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.4.0","body":"## What's Changed\r\n* add trainer integration docs by @lvwerra in https://github.com/huggingface/evaluate/pull/325\r\n* Stop using model-defined truncation in perplexity calculation by @mathemakitten in https://github.com/huggingface/evaluate/pull/333\r\n* Don't use eval for Evaluator instances in the doc by @fxmarty in https://github.com/huggingface/evaluate/pull/341\r\n* fix caching by @lvwerra in https://github.com/huggingface/evaluate/pull/336\r\n* Fix #327 set default row of gradio webui to 1 and drop empty/blank row by @Raibows in https://github.com/huggingface/evaluate/pull/335\r\n* Update pr docs actions by @mishig25 in https://github.com/huggingface/evaluate/pull/344\r\n* Fix `scikit-learn` install in spaces by @lvwerra in https://github.com/huggingface/evaluate/pull/345\r\n* added MASE, sMAPE and MAPE metrics by @kashif in https://github.com/huggingface/evaluate/pull/330\r\n* fix sklearn dependency in mape, mase and smape by @lvwerra in https://github.com/huggingface/evaluate/pull/346\r\n* Update link text by @stevhliu in https://github.com/huggingface/evaluate/pull/360\r\n* Corrected range of MAE by @clefourrier in https://github.com/huggingface/evaluate/pull/359\r\n* Revert \"Update pr docs actions\" by @mishig25 in https://github.com/huggingface/evaluate/pull/363\r\n* Evaluation suite by @mathemakitten in https://github.com/huggingface/evaluate/pull/337\r\n* Matthews correlation coefficient by @sanderland in https://github.com/huggingface/evaluate/pull/362\r\n* fix tf version by @lvwerra in https://github.com/huggingface/evaluate/pull/372\r\n* Add TextGeneration Evaluator by @NimaBoscarino in https://github.com/huggingface/evaluate/pull/350\r\n* Fix typo in rouge types by @davebulaval in https://github.com/huggingface/evaluate/pull/364\r\n* Add `Evaluate` usage for `scikit-learn` by @awinml in https://github.com/huggingface/evaluate/pull/368\r\n* Adding metric visualization by @sashavor in https://github.com/huggingface/evaluate/pull/342\r\n* Add NIST metric by @BramVanroy in https://github.com/huggingface/evaluate/pull/250\r\n* add GitHub Actions CI by @lvwerra in https://github.com/huggingface/evaluate/pull/375\r\n* Add Evaluate Usage for Keras and Tensorflow  by @arjunpatel7 in https://github.com/huggingface/evaluate/pull/370\r\n* fix version by @lvwerra in https://github.com/huggingface/evaluate/pull/380\r\n* CharacTER: MT metric by @BramVanroy in https://github.com/huggingface/evaluate/pull/286\r\n* CharCut: another character-based MT evaluation metric by @BramVanroy in https://github.com/huggingface/evaluate/pull/290\r\n* asr model evaluator addition + doc by @bayartsogt-ya in https://github.com/huggingface/evaluate/pull/378\r\n* Docs for EvaluationSuite by @mathemakitten in https://github.com/huggingface/evaluate/pull/340\r\n* Update the documentation of Mauve by @krishnap25 in https://github.com/huggingface/evaluate/pull/377\r\n* fix-ci-badge by @lvwerra in https://github.com/huggingface/evaluate/pull/385\r\n\r\n## New Contributors\r\n* @Raibows made their first contribution in https://github.com/huggingface/evaluate/pull/335\r\n* @kashif made their first contribution in https://github.com/huggingface/evaluate/pull/330\r\n* @clefourrier made their first contribution in https://github.com/huggingface/evaluate/pull/359\r\n* @davebulaval made their first contribution in https://github.com/huggingface/evaluate/pull/364\r\n* @awinml made their first contribution in https://github.com/huggingface/evaluate/pull/368\r\n* @arjunpatel7 made their first contribution in https://github.com/huggingface/evaluate/pull/370\r\n* @bayartsogt-ya made their first contribution in https://github.com/huggingface/evaluate/pull/378\r\n* @krishnap25 made their first contribution in https://github.com/huggingface/evaluate/pull/377\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.3.0...v0.4.0","reactions":{"url":"https://api.github.com/repos/huggingface/evaluate/releases/85867573/reactions","total_count":1,"+1":0,"-1":0,"laugh":0,"hooray":1,"confused":0,"heart":0,"rocket":0,"eyes":0},"mentions_count":17},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/79777170","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/79777170/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/79777170/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.3.0","id":79777170,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4EwU2S","tag_name":"v0.3.0","target_commitish":"main","name":"v0.3.0","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-10-13T12:59:31Z","updated_at":"2022-10-13T13:04:14Z","published_at":"2022-10-13T13:04:14Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.3.0","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.3.0","body":"## What's Changed\r\n* add multilabel f1 eval usage by @fcakyon in https://github.com/huggingface/evaluate/pull/221\r\n* Force get_supported_tasks() to return a list instead of dict keys by @mathemakitten in https://github.com/huggingface/evaluate/pull/227\r\n* Unpin rouge_score by @albertvillanova in https://github.com/huggingface/evaluate/pull/220\r\n* Remove import statement in Measurement Card by @meg-huggingface in https://github.com/huggingface/evaluate/pull/231\r\n* make rouge support multi-ref by @lvwerra in https://github.com/huggingface/evaluate/pull/229\r\n* Fix enforce string by @lvwerra in https://github.com/huggingface/evaluate/pull/230\r\n* Fix examples in perplexity measurement docs by @mathemakitten in https://github.com/huggingface/evaluate/pull/238\r\n* Add Wilcoxon's signed rank test by @douwekiela in https://github.com/huggingface/evaluate/pull/237\r\n* Add support for two input columns for TextClassificationEvaluator by @fxmarty in https://github.com/huggingface/evaluate/pull/205\r\n* fix bug in TEMPLATE_REQUIRE: add comma by @BramVanroy in https://github.com/huggingface/evaluate/pull/248\r\n* Minor quicktour doc suggestions by @stevhliu in https://github.com/huggingface/evaluate/pull/236\r\n* Clarify error message for ChrF no. references by @BramVanroy in https://github.com/huggingface/evaluate/pull/247\r\n* only track unique missing dependencies by @BramVanroy in https://github.com/huggingface/evaluate/pull/246\r\n* Update evaluate in spaces by @lvwerra in https://github.com/huggingface/evaluate/pull/228\r\n* add `commit_hash` to args by @lvwerra in https://github.com/huggingface/evaluate/pull/253\r\n* Change perplexity to be calculated with base e by @mathemakitten in https://github.com/huggingface/evaluate/pull/242\r\n* Rebase for previous PR by @mathemakitten in https://github.com/huggingface/evaluate/pull/254\r\n* Fix docstrings with new perplexities with base e by @mathemakitten in https://github.com/huggingface/evaluate/pull/255\r\n* add a tokenizer option to rouge by @lvwerra in https://github.com/huggingface/evaluate/pull/258\r\n* Adding  list_duplicates=True to example. by @meg-huggingface in https://github.com/huggingface/evaluate/pull/263\r\n* Minor change in describing what this does. by @meg-huggingface in https://github.com/huggingface/evaluate/pull/267\r\n* Mapping example output to returned output. by @meg-huggingface in https://github.com/huggingface/evaluate/pull/268\r\n* Changes \"duplicates_list\" to \"duplicates_dict\" (since it's dict) by @meg-huggingface in https://github.com/huggingface/evaluate/pull/265\r\n* Changes \"duplicates_list\" to \"duplicates_dict\" in the example. by @meg-huggingface in https://github.com/huggingface/evaluate/pull/264\r\n* Add slow flag to two column parity test by @lvwerra in https://github.com/huggingface/evaluate/pull/273\r\n* Remove `handle_impossible_answer` from the default `PIPELINE_KWARGS` in the question answering evaluator by @fxmarty in https://github.com/huggingface/evaluate/pull/272\r\n* Toxicity Measurement by @sashavor in https://github.com/huggingface/evaluate/pull/262\r\n* Automatically choose dataset split if none provided by @mathemakitten in https://github.com/huggingface/evaluate/pull/232\r\n* Fix YAML in Toxicity by @lvwerra in https://github.com/huggingface/evaluate/pull/278\r\n* Added metric Brier Score by @kadirnar in https://github.com/huggingface/evaluate/pull/275\r\n* Check for mismatch in device setup in evaluator by @mathemakitten in https://github.com/huggingface/evaluate/pull/287\r\n* Fix transfomers import in the evaluator by @mathemakitten in https://github.com/huggingface/evaluate/pull/291\r\n* Add support for name field when loading data by @mathemakitten in https://github.com/huggingface/evaluate/pull/283\r\n* Adding regard measurement by @sashavor in https://github.com/huggingface/evaluate/pull/271\r\n* Raise exception instead of assert in BertScore by @BramVanroy in https://github.com/huggingface/evaluate/pull/292\r\n* fix regard yaml by @lvwerra in https://github.com/huggingface/evaluate/pull/295\r\n* Add CONTRIBUTING.md by @mathemakitten in https://github.com/huggingface/evaluate/pull/293\r\n* Refactor kwargs and configs by @lvwerra in https://github.com/huggingface/evaluate/pull/188\r\n* Revert \"Refactor kwargs and configs\" by @lvwerra in https://github.com/huggingface/evaluate/pull/299\r\n* Add missing `split` and `subset` kwarg into other evaluators  by @mathemakitten in https://github.com/huggingface/evaluate/pull/301\r\n* Adding HONEST score by @sashavor in https://github.com/huggingface/evaluate/pull/279\r\n* fix wrong sorting in check by @sanderland in https://github.com/huggingface/evaluate/pull/305\r\n* Fix HONEST yaml by @lvwerra in https://github.com/huggingface/evaluate/pull/303\r\n* Refactor current_features to selected_feature_format by @mathemakitten in https://github.com/huggingface/evaluate/pull/306\r\n* replace datasets list with local list of tasks by @lvwerra in https://github.com/huggingface/evaluate/pull/309\r\n* Adding torch to the requirements by @sashavor in https://github.com/huggingface/evaluate/pull/311\r\n* Honest space fix by @sashavor in https://github.com/huggingface/evaluate/pull/312\r\n* Use HTML relative paths for tiles by @lewtun in https://github.com/huggingface/evaluate/pull/318\r\n* Test for valid YAML files by @mathemakitten in https://github.com/huggingface/evaluate/pull/308\r\n* add versioning the `HubEvaluationModuleFactory` by @lvwerra in https://github.com/huggingface/evaluate/pull/314\r\n* Add text2text evaluator by @lvwerra in https://github.com/huggingface/evaluate/pull/261\r\n* try main if tag does not work by @lvwerra in https://github.com/huggingface/evaluate/pull/322\r\n\r\n## New Contributors\r\n* @fcakyon made their first contribution in https://github.com/huggingface/evaluate/pull/221\r\n* @meg-huggingface made their first contribution in https://github.com/huggingface/evaluate/pull/231\r\n* @stevhliu made their first contribution in https://github.com/huggingface/evaluate/pull/236\r\n* @kadirnar made their first contribution in https://github.com/huggingface/evaluate/pull/275\r\n* @sanderland made their first contribution in https://github.com/huggingface/evaluate/pull/305\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.2.2...v0.3.0","reactions":{"url":"https://api.github.com/repos/huggingface/evaluate/releases/79777170/reactions","total_count":3,"+1":0,"-1":0,"laugh":2,"hooray":1,"confused":0,"heart":0,"rocket":0,"eyes":0},"mentions_count":13},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/73176303","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/73176303/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/73176303/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.2.2","id":73176303,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4EXJTv","tag_name":"v0.2.2","target_commitish":"main","name":"v0.2.2","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-07-29T14:17:10Z","updated_at":"2022-07-29T14:58:33Z","published_at":"2022-07-29T14:58:33Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.2.2","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.2.2","body":"## What's Changed\r\n* Update CLI docs by @lvwerra in https://github.com/huggingface/evaluate/pull/218\r\n* Add a fingerprint for each EvaluationModule by @mathemakitten in https://github.com/huggingface/evaluate/pull/206\r\n* Fix loading error by @lvwerra in https://github.com/huggingface/evaluate/pull/222\r\n\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.2.1...v0.2.2","reactions":{"url":"https://api.github.com/repos/huggingface/evaluate/releases/73176303/reactions","total_count":2,"+1":0,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":2,"eyes":0},"mentions_count":2},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/73061525","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/73061525/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/73061525/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.2.1","id":73061525,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4EWtSV","tag_name":"v0.2.1","target_commitish":"main","name":"v0.2.1","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-07-28T12:32:16Z","updated_at":"2022-07-28T13:13:50Z","published_at":"2022-07-28T13:13:50Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.2.1","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.2.1","body":"## What's Changed\r\n* Add measurements to quality and style checks by @lvwerra in https://github.com/huggingface/evaluate/pull/203\r\n* Add comparisons and measurements to code quality tests by @lvwerra in https://github.com/huggingface/evaluate/pull/204\r\n* Remove mention to datasets from docs by @albertvillanova in https://github.com/huggingface/evaluate/pull/207\r\n* Adding label distribution measurement by @sashavor in https://github.com/huggingface/evaluate/pull/202\r\n* Fix spaces tagging by @lvwerra in https://github.com/huggingface/evaluate/pull/217\r\n* set datasets to >=2.0.0 by @lvwerra in https://github.com/huggingface/evaluate/pull/216\r\n\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.2.0...v0.2.1","mentions_count":3},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/72763534","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/72763534/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/72763534/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.2.0","id":72763534,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4EVkiO","tag_name":"v0.2.0","target_commitish":"main","name":"v0.2.0","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-07-25T13:41:54Z","updated_at":"2022-07-25T14:34:09Z","published_at":"2022-07-25T14:34:09Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.2.0","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.2.0","body":"## What's New\r\n\r\n### `evaluator`\r\nThe `evaluator` has been extended to three new tasks:\r\n- `\"image-classification\"`\r\n- `\"token-classification\"`\r\n- `\"question-answering\"`\r\n\r\n### `combine`\r\nWith `combine` one can bundle several metrics into a single object that can be evaluated in one call and also used in combination with the `evalutor`.\r\n\r\n## What's Changed\r\n* Fix typo in WER docs by @pn11 in https://github.com/huggingface/evaluate/pull/147\r\n* Fix rouge outputs by @lvwerra in https://github.com/huggingface/evaluate/pull/158\r\n* add tutorial for custom pipeline by @lvwerra in https://github.com/huggingface/evaluate/pull/154\r\n* refactor `evaluator` tests by @lvwerra in https://github.com/huggingface/evaluate/pull/155\r\n* rename `input_texts` to `predictions` in perplexity by @lvwerra in https://github.com/huggingface/evaluate/pull/157\r\n* Add link to GitHub author by @lewtun in https://github.com/huggingface/evaluate/pull/166\r\n* Add `combine` to compose multiple evaluations by @lvwerra in https://github.com/huggingface/evaluate/pull/150\r\n* test string casting only on first element by @lvwerra in https://github.com/huggingface/evaluate/pull/159\r\n* remove unused fixtures from unittests by @lvwerra in https://github.com/huggingface/evaluate/pull/170\r\n* Add a test to check that Evaluator evaluations match transformers examples by @fxmarty in https://github.com/huggingface/evaluate/pull/163\r\n* Add smaller model for `TextClassificationEvaluator` test by @fxmarty in https://github.com/huggingface/evaluate/pull/172\r\n* Add tags to spaces by @lvwerra in https://github.com/huggingface/evaluate/pull/162\r\n* Rename evaluation modules by @lvwerra in https://github.com/huggingface/evaluate/pull/160\r\n* Update push_evaluations_to_hub.py by @lvwerra in https://github.com/huggingface/evaluate/pull/174\r\n* update evaluate dependency for spaces by @lvwerra in https://github.com/huggingface/evaluate/pull/175\r\n* Add `ImageClassificationEvaluator` by @fxmarty in https://github.com/huggingface/evaluate/pull/173\r\n* attempting to let meteor handle multiple references per prediction by @sashavor in https://github.com/huggingface/evaluate/pull/164\r\n* fixed duplicate calculation of spearmanr function in metrics wrapper. by @benlipkin in https://github.com/huggingface/evaluate/pull/176\r\n* forbid hyphens in template for module names by @lvwerra in https://github.com/huggingface/evaluate/pull/177\r\n* switch from Github to Hub module factory for canonical modules by @lvwerra in https://github.com/huggingface/evaluate/pull/180\r\n* Fix bertscore idf by @lvwerra in https://github.com/huggingface/evaluate/pull/183\r\n* refactor evaluator base and task classes by @lvwerra in https://github.com/huggingface/evaluate/pull/185\r\n* Avoid importing tensorflow when importing evaluate by @NouamaneTazi in https://github.com/huggingface/evaluate/pull/135\r\n* Add QuestionAnsweringEvaluator by @fxmarty in https://github.com/huggingface/evaluate/pull/179\r\n* Evaluator perf by @ola13 in https://github.com/huggingface/evaluate/pull/178\r\n* Fix QuestionAnsweringEvaluator for squad v2, fix examples by @fxmarty in https://github.com/huggingface/evaluate/pull/190\r\n* Rename perf metric evaluator by @lvwerra in https://github.com/huggingface/evaluate/pull/191\r\n* Fix typos in QA Evaluator by @lewtun in https://github.com/huggingface/evaluate/pull/192\r\n* Evaluator device placement by @lvwerra in https://github.com/huggingface/evaluate/pull/193\r\n* Change test command in installation.mdx to use exact_match by @mathemakitten in https://github.com/huggingface/evaluate/pull/194\r\n* Add `TokenClassificationEvaluator` by @fxmarty in https://github.com/huggingface/evaluate/pull/167\r\n* Pin rouge_score by @albertvillanova in https://github.com/huggingface/evaluate/pull/197\r\n* add poseval by @lvwerra in https://github.com/huggingface/evaluate/pull/195\r\n* Combine docs by @lvwerra in https://github.com/huggingface/evaluate/pull/201\r\n* Evaluator column loading by @lvwerra in https://github.com/huggingface/evaluate/pull/200\r\n* Evaluator documentation by @lvwerra in https://github.com/huggingface/evaluate/pull/199\r\n\r\n## New Contributors\r\n* @pn11 made their first contribution in https://github.com/huggingface/evaluate/pull/147\r\n* @fxmarty made their first contribution in https://github.com/huggingface/evaluate/pull/163\r\n* @benlipkin made their first contribution in https://github.com/huggingface/evaluate/pull/176\r\n* @NouamaneTazi made their first contribution in https://github.com/huggingface/evaluate/pull/135\r\n* @mathemakitten made their first contribution in https://github.com/huggingface/evaluate/pull/194\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.1.2...v0.2.0","mentions_count":10},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/69601663","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/69601663/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/69601663/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.1.2","id":69601663,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4EJgl_","tag_name":"v0.1.2","target_commitish":"main","name":"v0.1.2","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-06-16T09:47:26Z","updated_at":"2022-06-16T10:01:40Z","published_at":"2022-06-16T10:01:40Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.1.2","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.1.2","body":"## What's Changed\r\n* Fix trec sacrebleu by @lvwerra in https://github.com/huggingface/evaluate/pull/130\r\n* Add distilled version Cometihno by @BramVanroy in https://github.com/huggingface/evaluate/pull/131\r\n* fix: add yaml extension to github action for release by @lvwerra in https://github.com/huggingface/evaluate/pull/133\r\n* fix docs badge by @lvwerra in https://github.com/huggingface/evaluate/pull/134\r\n* fix cookiecutter path to repository by @lvwerra in https://github.com/huggingface/evaluate/pull/139\r\n* docs: make metric cards more prominent by @lvwerra in https://github.com/huggingface/evaluate/pull/132\r\n* Update README.md by @sashavor in https://github.com/huggingface/evaluate/pull/145\r\n* Fix datasets download imports by @albertvillanova in https://github.com/huggingface/evaluate/pull/143\r\n\r\n## New Contributors\r\n* @BramVanroy made their first contribution in https://github.com/huggingface/evaluate/pull/131\r\n* @albertvillanova made their first contribution in https://github.com/huggingface/evaluate/pull/143\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.1.1...v0.1.2","reactions":{"url":"https://api.github.com/repos/huggingface/evaluate/releases/69601663/reactions","total_count":1,"+1":1,"-1":0,"laugh":0,"hooray":0,"confused":0,"heart":0,"rocket":0,"eyes":0},"mentions_count":4},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/68945766","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/68945766/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/68945766/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.1.1","id":68945766,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4EHAdm","tag_name":"v0.1.1","target_commitish":"main","name":"v0.1.1","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-06-08T12:28:23Z","updated_at":"2022-06-08T12:38:39Z","published_at":"2022-06-08T12:38:39Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.1.1","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.1.1","body":"## What's Changed\r\n* Fix broken links by @mishig25 in https://github.com/huggingface/evaluate/pull/92\r\n* Fix readme by @lvwerra in https://github.com/huggingface/evaluate/pull/98\r\n* Fixing broken evaluate-measurement hub link by @panwarnaveen9 in https://github.com/huggingface/evaluate/pull/102\r\n* fix typo in autodoc by @manueldeprada in https://github.com/huggingface/evaluate/pull/101\r\n* fix typo by @manueldeprada in https://github.com/huggingface/evaluate/pull/100\r\n* FIX `pip install evaluate[evaluator]` by @philschmid in https://github.com/huggingface/evaluate/pull/103\r\n* fix description field in metric template readme by @lvwerra in https://github.com/huggingface/evaluate/pull/122\r\n* Add automatic pypi release for evaluate by @osanseviero in https://github.com/huggingface/evaluate/pull/121\r\n* Fix typos in Evaluator docstrings by @lewtun in https://github.com/huggingface/evaluate/pull/124\r\n* Fix spaces description in metadata by @lvwerra in https://github.com/huggingface/evaluate/pull/123\r\n* fix revision string if it is a python version by @lvwerra in https://github.com/huggingface/evaluate/pull/129\r\n* Use accuracy as default metric for text classification Evaluator by @lewtun in https://github.com/huggingface/evaluate/pull/128\r\n* bump `evaluate` dependency in spaces by @lvwerra in https://github.com/huggingface/evaluate/pull/88\r\n\r\n## New Contributors\r\n* @panwarnaveen9 made their first contribution in https://github.com/huggingface/evaluate/pull/102\r\n* @manueldeprada made their first contribution in https://github.com/huggingface/evaluate/pull/101\r\n* @philschmid made their first contribution in https://github.com/huggingface/evaluate/pull/103\r\n* @osanseviero made their first contribution in https://github.com/huggingface/evaluate/pull/121\r\n* @lewtun made their first contribution in https://github.com/huggingface/evaluate/pull/124\r\n\r\n**Full Changelog**: https://github.com/huggingface/evaluate/compare/v0.1.0...v0.1.1","mentions_count":7},{"url":"https://api.github.com/repos/huggingface/evaluate/releases/68247112","assets_url":"https://api.github.com/repos/huggingface/evaluate/releases/68247112/assets","upload_url":"https://uploads.github.com/repos/huggingface/evaluate/releases/68247112/assets{?name,label}","html_url":"https://github.com/huggingface/evaluate/releases/tag/v0.1.0","id":68247112,"author":{"login":"lvwerra","id":8264887,"node_id":"MDQ6VXNlcjgyNjQ4ODc=","avatar_url":"https://avatars.githubusercontent.com/u/8264887?v=4","gravatar_id":"","url":"https://api.github.com/users/lvwerra","html_url":"https://github.com/lvwerra","followers_url":"https://api.github.com/users/lvwerra/followers","following_url":"https://api.github.com/users/lvwerra/following{/other_user}","gists_url":"https://api.github.com/users/lvwerra/gists{/gist_id}","starred_url":"https://api.github.com/users/lvwerra/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/lvwerra/subscriptions","organizations_url":"https://api.github.com/users/lvwerra/orgs","repos_url":"https://api.github.com/users/lvwerra/repos","events_url":"https://api.github.com/users/lvwerra/events{/privacy}","received_events_url":"https://api.github.com/users/lvwerra/received_events","type":"User","user_view_type":"public","site_admin":false},"node_id":"RE_kwDOHF4oAM4EEV5I","tag_name":"v0.1.0","target_commitish":"main","name":"Initial relase of `evaluate`","draft":false,"immutable":false,"prerelease":false,"created_at":"2022-05-31T13:52:22Z","updated_at":"2022-05-31T13:57:20Z","published_at":"2022-05-31T13:57:20Z","assets":[],"tarball_url":"https://api.github.com/repos/huggingface/evaluate/tarball/v0.1.0","zipball_url":"https://api.github.com/repos/huggingface/evaluate/zipball/v0.1.0","body":"# Release notes\r\n​\r\nThese are the release notes of the initial release of the Evaluate library.\r\n​\r\n## Goals\r\n​\r\nGoals of the Evaluate library:\r\n​\r\n- reproducibility: reporting and reproducing results is easy\r\n- ease-of-use: access to a wide range of evaluation tools with a unified interface\r\n- diversity: provide wide range of evaluation tools with metrics, comparisons, and measurements\r\n- multimodal: models and datasets of many modalities can be evaluated\r\n- community-driven: anybody can add custom evaluations by hosting them on the Hugging Face Hub\r\n​\r\n## Release overview:\r\n​\r\n- `evaluate.load()`: The `load()` function is the main entry point into evaluate and allows to load evaluation modules from a local folder, the evaluate repository, or the Hugging Face Hub. It downloads, caches, and loads the evaluation modules and returns an `evaluate.EvaluationModule`.\r\n- `evaluate.save()`: With `save()` a user can save evaluation results in a JSON file. In addition to the results from `evaluate.EvaluationModule` it can save additional parameters and automatically saves the timestamp, git commit hash, library version as well as Python path. One can either provide a directory for the results, in which case file names are automatically created, or an explicit file name for the result.\r\n- `evaluate.push_to_hub()`: The `push_to_hub` function allows to push the results of a model evaluation to the model card on the Hugging Face Hub. The model, dataset, and metric are specified such that they can be linked on the hub.\r\n- `evaluate.EvaluationModule`: The `EvaluationModule` class is the baseclass for all evaluation modules. There are three module types: metrics (to evaluate models), comparisons (to compare models), and measurements (to analyze datasets). The inputs can be either added with `add` (single input) and `add_batch` (batch of inputs) followed by a final `compute` call to compute the scores or all inputs can be passed to `compute` directly. Under the hood, Apache Arrow stores and loads the input data to compute the scores.\r\n- `evaluate.EvaluationModuleInfo`: The `EvaluationModule` class is used to store attributes:\r\n    - `description`: A short description of the evaluation module.\r\n    - `citation`: A BibTex string for citation when available.\r\n    - `features`: A `Features` object defining the input format. The inputs provided to `add`, `add_batch`, and `compute` are tested against these types and an error is thrown in case of a mismatch.\r\n    - `inputs_description`: This is equivalent to the modules docstring.\r\n    - `homepage`: The homepage of the module.\r\n    - `license`: The license of the module.\r\n    - `codebase_urls`: Link to the code behind the module.\r\n    - `reference_urls`: Additional reference URLs.\r\n- `evaluate.evaluator`: The `evaluator` provides automated evaluation and only requires a model, dataset, metric, in contrast to the metrics in the `EvaluationModule` which require  model predictions. It has three main components: a model wrapped in a pipeline, a dataset, and a metric, and it returns the computed evaluation scores. Besides the three main components, it may also require two mappings to align the columns in the dataset and the pipeline labels with the datasets labels. This is an experimental feature -- currently, only text classification is supported.\r\n- `evaluate-cli`: The community can add custom metrics by adding the necessary module script to a Space on the Hugging Face Hub. The `evaluate-cli` is a tool that simplifies this process by creating the Space, populating a template, and pushing it to the Hub. It also provides instructions to customize the template and integrate custom logic.\r\n​\r\n## Main contributors:\r\n​\r\n@lvwerra , @sashavor , @NimaBoscarino , @ola13 , @osanseviero , @lhoestq , @lewtun , @douwekiela  ","reactions":{"url":"https://api.github.com/repos/huggingface/evaluate/releases/68247112/reactions","total_count":4,"+1":0,"-1":0,"laugh":0,"hooray":4,"confused":0,"heart":0,"rocket":0,"eyes":0},"mentions_count":8}]