{"url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028","forks_url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/forks","commits_url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/commits","id":"34b83f02d36e8c865060ed9d899ba028","node_id":"MDQ6R2lzdDM0YjgzZjAyZDM2ZThjODY1MDYwZWQ5ZDg5OWJhMDI4","git_pull_url":"https://gist.github.com/34b83f02d36e8c865060ed9d899ba028.git","git_push_url":"https://gist.github.com/34b83f02d36e8c865060ed9d899ba028.git","html_url":"https://gist.github.com/michaellihs/34b83f02d36e8c865060ed9d899ba028","files":{"python-learning.md":{"filename":"python-learning.md","type":"text/markdown","language":"Markdown","raw_url":"https://gist.githubusercontent.com/michaellihs/34b83f02d36e8c865060ed9d899ba028/raw/9c61a1c0ee3e7cc89e5f9c739e924def711df9e6/python-learning.md","size":6048,"truncated":false,"content":"# Python Learnings\n\n## General Learnings\n\n* `__init__.py` tells Python that the directory is a package\n\n## Standard Project Setup\n\n```text\nproject/\n├── MANIFEST.in\n├── README.md\n├── setup.py                   important for building distribution out of a package\n├── src\n│    └── tasks\n│        ├── __init__.py\n│        ├── api.py\n│        ├── cli.py\n│        ├── config.py\n│        ├── tasksdb_pymongo.py\n│        └── tasksdb_tinydb.py\n└── tests\n    ├── conftest.py\n    ├── pytest.ini\n    ├── func\n    │    ├── __init__.py      empty to make pytest look for pytest.ini one level up\n    │    ├── test_add.py\n    │    └── ...\n    └── unit\n        ├── __init__.py       empty to make pytest look for pytest.ini one level up\n        ├── test_task.py\n        └── ...\n```\n\n## Package / Dependency Management\n\n* see https://stefanoborini.com/current-status-of-python-packaging/\n* python *packaging* is not really about python packages. It’s about python *distributions*\n* A *distribution* of Python is a bundle that contains an implementation of Python along with a bunch of libraries or tools.\n\n### \"New\" way\n\n* use Poetry\n* create a `pyproject.toml` and specify dependencies with a minimum version\n\n### \"Old\" way\n\n* install `virtualenv`\n\n   ```bash\n   python3 -m pip install --user virtualenv\n   ```\n\n* create a new virtual environment in `venv` via\n\n   ```bash\n   python3 -m venv venv\n   ```\n\n* activate virtual environment via\n\n   ```bash\n   source venv/bin/activate\n   ```\n  \n* create a `requirements.txt` file with your dependencies\n\n   ```\n   pytest >= 6.2.4\n   ```\n\n* install dependencies into your `venv` via\n\n   ```bash\n   pip install -r requirements.txt\n   ```\n\n\n### Creating virtual environments\n\n```bash\n$ pip3 install -U virtualenv\n$ python3 -m virtualenv venv\n$ source venv/bin/activate\n$ pip install pytest\n```\n\n\n## Testing\n\n### Pytest setup\n\n* see https://docs.pytest.org/en/6.2.x/goodpractices.html\n\n### Naming conventions of files\n\n`pytest` auto-discovers files with the following naming conventions:\n\n* Test files `test_...py` or `..._test.py`\n* Test methods `test_<something>`\n* Test classes `Test<Something>`\n\n### Filtering tests\n\nIf you want to run only a single test, you can use\n\n```bash\n$ pytest -v tasks/test_four.py::test_asdict\n```\n\n### Organising Tests\n\n#### Markers\n\n1. Add a marker to your test file:\n\n   ```python\n   import pytest\n   \n   @pytest.mark.run_these_please\n   def test_asdict():\n   ```      \n\n2. Add a `pytest.ini` to your tests directory\n\n   ```ini\n   [pytest]\n   markers =\n       run_these_please: mark a test as please to be run\n   ```\n   \n3. run your tests via\n\n   ```bash\n   $ pytest -v -m run_these_please\n   ```\n   \nYou can specify the location of the `pytest.ini` file with the `-c` flag:\n\n```bash\n$ pytest -c ../pytest.ini -v -m please_run_me\n```\n\n### Resources\n\n* [The Pytesting Book](https://learning.oreilly.com/library/view/python-testing-with/9781680502848/f_0011.xhtml)\n* [Pytest Home](https://docs.pytest.org)\n\n### tox\n\nOnce you are done with your work and want to make sure that your actual package passes all tests you may want to look into [tox](https://docs.pytest.org/en/6.2.x/goodpractices.html#tox), the virtualenv test automation tool and its pytest support. tox helps you to setup virtualenv environments with pre-defined dependencies and then executing a pre-configured test command with options. It will run tests against the installed package and not against your source code checkout, helping to detect packaging glitches.\n\n\n## Dockerizing\n\n### Building Python Docker Images\n\n```Dockerfile\nFROM python:3.9-slim-bullseye\n\nWORKDIR /usr/src/app\n\nUSER root\n\nRUN addgroup --gid 1001 --system mbdrna\nRUN adduser --uid 1001 --gid 1001 mbdrna\nRUN apt-get update\nRUN apt-get install -y gcc libffi-dev musl-dev make g++\n\n# see https://pythonspeed.com/articles/activate-virtualenv-dockerfile/\nENV VIRTUAL_ENV=/opt/venv\nRUN python3 -m venv $VIRTUAL_ENV\nENV PATH=\"$VIRTUAL_ENV/bin:$PATH\"\n\nRUN pip install --upgrade pip\nRUN pip install --upgrade azure-cli\nRUN pip install --upgrade setuptools\n\n# everything above can be cached, so split pip install here\n\nCOPY . .\n\nRUN . venv/bin/activate && pip install --upgrade -r sdp_flask/requirements.txt \\\n    && chown -R mbdrna:mbdrna /usr/src/app \n\nWORKDIR /usr/src/app/sdp_flask\n\nUSER mbdrna\nENV FLASK_RUN_PORT=1337 \nENV FLASK_ENV=development\nCMD [ \"../venv/bin/flask\", \"run\", \"--host=0.0.0.0\" ]\n```\n\n### Docker Compose for dev environments\n\n```yaml\nversion: '3.5'\n\nnetworks:\n  default:\n    name: default\n    driver: bridge\n\nservices:\n  flask:\n    image: 'flask-app:local-dev-version'\n    depends_on:\n      - rabbit\n      - redis\n    ports:\n      - \"1337:1337\"\n    volumes:\n      - ./app_flask:/usr/src/app/app_flask\n    container_name: flask-app\n    networks:\n      - default\n    environment:\n      CELERY_BROKER_URL: amqp://guest:guest@rabbit/\n      REDIS_HOST: redis\n\n  # see https://hub.docker.com/_/rabbitmq\n  rabbit:\n    image: \"rabbitmq:3.9.5\"\n    networks:\n      - default\n    ports:\n      - \"5672:5672\"\n    container_name: rabbit\n\n  # see https://hub.docker.com/_/redis\n  redis:\n    image: \"redis:6.2.5\"\n    networks:\n      - default\n    ports:\n      - \"6379:6379\"\n    container_name: redis\n```\n\n## Celery\n\n### Celery Testing\n\n* https://www.distributedpython.com/2018/05/01/unit-testing-celery-tasks/\n\n## Tools\n\n* [Poetry - Python Packaging and Dependency Management](https://python-poetry.org/)\n\n\n## References\n\n* [Activating venv in Dockerfile](https://pythonspeed.com/articles/activate-virtualenv-dockerfile/)\n* [Docker Base Images for Python](https://pythonspeed.com/articles/alpine-docker-python/)\n* [Flask Project Layout](https://flask.palletsprojects.com/en/2.0.x/tutorial/layout/)\n* [Avoid re-installing packages in Python Docker images](https://stackoverflow.com/questions/25305788/how-to-avoid-reinstalling-packages-when-building-docker-image-for-python-project/25307587#25307587)","encoding":"utf-8"}},"public":true,"created_at":"2021-09-01T08:48:44Z","updated_at":"2025-05-03T02:42:51Z","description":"Python","comments":0,"user":null,"comments_enabled":true,"comments_url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/comments","owner":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"forks":[{"url":"https://api.github.com/gists/881ebb8aaac4a5d532d78ed2dc72624c","user":{"login":"jinghuayao","id":17486729,"node_id":"MDQ6VXNlcjE3NDg2NzI5","avatar_url":"https://avatars.githubusercontent.com/u/17486729?v=4","gravatar_id":"","url":"https://api.github.com/users/jinghuayao","html_url":"https://github.com/jinghuayao","followers_url":"https://api.github.com/users/jinghuayao/followers","following_url":"https://api.github.com/users/jinghuayao/following{/other_user}","gists_url":"https://api.github.com/users/jinghuayao/gists{/gist_id}","starred_url":"https://api.github.com/users/jinghuayao/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/jinghuayao/subscriptions","organizations_url":"https://api.github.com/users/jinghuayao/orgs","repos_url":"https://api.github.com/users/jinghuayao/repos","events_url":"https://api.github.com/users/jinghuayao/events{/privacy}","received_events_url":"https://api.github.com/users/jinghuayao/received_events","type":"User","user_view_type":"public","site_admin":false,"name":"rooibios_tea","company":null,"blog":"","location":null,"email":"yaojhyao@gmail.com","hireable":null,"bio":"Ph.D. in Mathematics","twitter_username":null,"public_repos":143,"public_gists":3,"followers":5,"following":47,"created_at":"2016-02-26T02:09:12Z","updated_at":"2026-07-22T00:50:00Z"},"id":"881ebb8aaac4a5d532d78ed2dc72624c","created_at":"2023-03-15T22:28:24Z","updated_at":"2023-03-15T22:28:24Z"},{"url":"https://api.github.com/gists/7853c32eab95a3d7c57eb52963cc74af","user":{"login":"awsvpc","id":88736238,"node_id":"MDQ6VXNlcjg4NzM2MjM4","avatar_url":"https://avatars.githubusercontent.com/u/88736238?v=4","gravatar_id":"","url":"https://api.github.com/users/awsvpc","html_url":"https://github.com/awsvpc","followers_url":"https://api.github.com/users/awsvpc/followers","following_url":"https://api.github.com/users/awsvpc/following{/other_user}","gists_url":"https://api.github.com/users/awsvpc/gists{/gist_id}","starred_url":"https://api.github.com/users/awsvpc/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/awsvpc/subscriptions","organizations_url":"https://api.github.com/users/awsvpc/orgs","repos_url":"https://api.github.com/users/awsvpc/repos","events_url":"https://api.github.com/users/awsvpc/events{/privacy}","received_events_url":"https://api.github.com/users/awsvpc/received_events","type":"User","user_view_type":"public","site_admin":false,"name":null,"company":null,"blog":"","location":null,"email":null,"hireable":null,"bio":null,"twitter_username":null,"public_repos":1269,"public_gists":697,"followers":1,"following":0,"created_at":"2021-08-10T17:33:59Z","updated_at":"2026-06-23T01:44:25Z"},"id":"7853c32eab95a3d7c57eb52963cc74af","created_at":"2025-05-03T02:42:51Z","updated_at":"2025-05-03T02:42:51Z"}],"history":[{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"5cb023083f176786c656e5d1fa5a8fede928967c","committed_at":"2021-09-21T11:55:20Z","change_status":{"total":6,"additions":6,"deletions":0},"url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/5cb023083f176786c656e5d1fa5a8fede928967c"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"10974700a2f3b41974910170393017c4315185df","committed_at":"2021-09-02T07:29:48Z","change_status":{"total":2,"additions":0,"deletions":2},"url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/10974700a2f3b41974910170393017c4315185df"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"b9a120a551bce7f6b77bdd2836ee3cf8b312ccf1","committed_at":"2021-09-01T14:08:08Z","change_status":{"total":199,"additions":199,"deletions":0},"url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/b9a120a551bce7f6b77bdd2836ee3cf8b312ccf1"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"f77cb632d97737894be2dea54681490fa34d12bc","committed_at":"2021-09-01T08:51:01Z","change_status":{"total":9,"additions":7,"deletions":2},"url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/f77cb632d97737894be2dea54681490fa34d12bc"},{"user":{"login":"michaellihs","id":575011,"node_id":"MDQ6VXNlcjU3NTAxMQ==","avatar_url":"https://avatars.githubusercontent.com/u/575011?v=4","gravatar_id":"","url":"https://api.github.com/users/michaellihs","html_url":"https://github.com/michaellihs","followers_url":"https://api.github.com/users/michaellihs/followers","following_url":"https://api.github.com/users/michaellihs/following{/other_user}","gists_url":"https://api.github.com/users/michaellihs/gists{/gist_id}","starred_url":"https://api.github.com/users/michaellihs/starred{/owner}{/repo}","subscriptions_url":"https://api.github.com/users/michaellihs/subscriptions","organizations_url":"https://api.github.com/users/michaellihs/orgs","repos_url":"https://api.github.com/users/michaellihs/repos","events_url":"https://api.github.com/users/michaellihs/events{/privacy}","received_events_url":"https://api.github.com/users/michaellihs/received_events","type":"User","user_view_type":"public","site_admin":false},"version":"504d36d39cec3de107c593e0a534ee9644d638f5","committed_at":"2021-09-01T08:48:44Z","change_status":{"total":48,"additions":48,"deletions":0},"url":"https://api.github.com/gists/34b83f02d36e8c865060ed9d899ba028/504d36d39cec3de107c593e0a534ee9644d638f5"}],"truncated":false}