Python Bytes

Python Bytes is a weekly podcast hosted by Michael Kennedy and Brian Okken. The show is a short discussion on the headlines and noteworthy news in the Python, developer, and data science space.

https://pythonbytes.fm/

subscribe
share





episode 220: What, why, and where of friendly errors in Python


Sponsored by Datadog: pythonbytes.fm/datadog

Special guest: Hannah Stepanek

Watch on YouTube

Michael #1:We Downloaded 10,000,000 Jupyter Notebooks From Github – This Is What We Learned

  • by Alena Guzharina from JetBrains
  • Used the hundreds of thousands of publicly accessible repos on GitHub to learn more about the current state of data science. I think it’s inspired by work showcased here on Talk Python.
  • 2 years ago there were 1,230,000 Jupyter Notebooks published on GitHub. By October 2020 this number had grown 8 times, and we were able to download 9,720,000 notebooks. 8x growth.
  • Despite the rapid growth in popularity of R and Julia in recent years, Python still remains the most commonly used language for writing code in Jupyter Notebooks by an enormous margin.
  • Python 2 went from 53% → 11% in the last two years.
  • Interesting graphs about package usage
  • Not all notebooks are story telling with code: 50% of notebooks contain fewer than 4 Markdown cells and more than 66 code cells.
  • Although there are some outliers, like notebooks with more than 25,000 code lines, 95% of the notebooks contain less than 465 lines of code.

Brian #2:pytest-pythonpath

  • plugin for adding to the PYTHONPATH from the pytests.ini file before tests run
  • Mentioned briefly in episode 62 as a temporary stopgap until you set up a proper package install for your code. (cringing at my arrogance).
  • Lots of projects are NOT packages. For example, applications.
  • I’ve been working with more and more people to get started on testing and the first thing that often comes up is “My tests can’t see my code. Please fix.”
  • Example
    • proj/src/stuff_you_want_to_test.py
    • proj/tests/test_code.py
    • You can’t import stuff_you_want_to_test.py from the proj/tests directory by default.
  • The more I look at the problem, the more I appreciate the simplicity of pytest-pythonpath
  • pytest-pythonpath does one thing I really care about:
    • Add this to a pytest.ini file at the proj level:
[pytest] python_paths = src
  • That’s it. That’s all you have to do to fix the above problem.
  • Paths relative to the directory that pytest.ini is in. Which should be a parent or grandparent of the tests directory.
  • I really can’t think of a simpler way for people to get around this problem.

Hannah #3:Thinking in Pandas

  • Pandas dependency hierarchy (simplified):
    • Pandas -> NumPy -> BLAS (Basic Linear Algebra Subprograms)
  • Languages:
  • -
- Python -> C -> Assembly df["C"] = df["A"] + df["B"] A = [ 1 4 2 0 ] B = [ 3 2 5 1 ] C = [ 1 + 3 4 + 2 2 + 5 0 + 1 ]
  • Pandas tries to get the best performance by running operations in parallel.
  • You might think we could speed this problem up by doing something like this:
Thread 1: 1 + 3 Thread 2: 4 + 2 Thread 3: 2 + 5 Thread 4: 0 + 1
  • However, the GIL (Global Interpreter Lock) prevents us from achieving the performance improvement we are hoping for.
  • Below is an example of a common threading problem and how a lock solves that problem.
  • -
Thread 1 total Thread 2 1 + 3 + 4 + 2 0 0 + 5 10 0 + 6 + 2 total += 10 0 13 total =10 0 total += 13 10 total = 13 13 Thread 1 total Thread 2 1 + 3 + 4 + 2 0 unlocked 0 + 5 10 0 unlocked + 6 + 2 total += 10 0 locked 13 total =10 0 locked 10 unlocked 10 locked total += 13 10 locked total = 13 23 unlocked
  • As it turns out, because Python manages memory for you every object in Python would be subject to these kinds of threading issues:
a = 1 # reference count = 1 b = a # reference count = 2 del(b) # reference count = 1 del(a) # reference count = 0
  • So, the GIL was invented to avoid this headache which only lets one thread run at a time.
  • Certain parts of the Pandas dependency hierarchy are not subject to the GIL (simplified):
    • Pandas -> NumPy -> BLAS (Basic Linear Algebra Subprograms)
    • GIL -> no GIL -> hardware optimizations
  • So we can get around the GIL in C land but what kind of optimizations does BLAS provide us with?
    • Parallel operations inside the CPU via Vector registers
  • A vector register is like a regular register but instead of holding one value it can hold multiple values.
| 1 | 4 | 2 | 0 | + + + + | 3 | 2 | 5 | 1 | = = = = | 4 | 6 | 7 | 1 |
  • Vector registers are only so large though, so the Dataframe is broken up into chunks and the vector operations are performed on each chunk.

Michael #4:Quickle

  • Fast. Benchmarks show it’s among the fastest serialization methods for Python.
  • Safe. Unlike pickle, deserializing a user provided message doesn’t allow for arbitrary code execution.
  • Flexible. Unlike msgpack or json, Quickle natively supports a wide range of Python builtin types.
  • Versioning. Quickle supports “schema evolution”. Messages can be sent between clients with different schemas without error.
  • Example
>>> import quickle >>> data = quickle.dumps({"hello": "world"}) >>> quickle.loads(data) {'hello': 'world'}

Brian #5:what(), why(), where(), explain(), more() from friendly-traceback console

  • Do this:
$ pip install friendly-friendly_traceback.install() $ python -i >>> import friendly_traceback >>> friendly_traceback.start_console() >>>
  • Now, after an exception happens, you can ask questions about it.
>>> pass = 1 Traceback (most recent call last): File "[HTML_REMOVED]", line 1 pass = 1 ^ SyntaxError: invalid syntax >>> what() SyntaxError: invalid syntax A `SyntaxError` occurs when Python cannot understand your code. >>> why() You were trying to assign a value to the Python keyword `pass`. This is not allowed. >>> where() Python could not understand the code in the file '[HTML_REMOVED]' beyond the location indicated by --> and ^. -->1: pass = 1 ^
  • Cool for teaching or learning.

Hannah #6:Bandit

  • Bandit is a static analysis security tool.
  • It’s like a linter but for security issues.
pip install bandit bandit -r .
  • I prefer to run it in a git pre-commit hook:
# .pre-commit-config.yaml repos: repo: https://github.com/PyCQA/bandit rev: '1.7.6' hooks: - id: bandit
  • It finds issues like:
    • flask_debug_true
    • request_with_no_cert_validation
  • You can ignore certain issues just like any other linter:
assert len(foo) == 1 # nosec

Extras:

Brian:

  • Meetups this week 2/3 done.
    • NOAA Tuesday, Aberdeen this morning - “pytest Fixtures”
    • PDX West tomorrow - Michael Presenting “Python Memory Deep Dive”
  • Updated my training page, testandcode.com/training
    • Feedback welcome.
    • I really like working directly with teams and now that trainings can be virtual, a couple half days is super easy to do.

Michael:

  • PEP 634 -- Structural Pattern Matching: Specification accepted in 3.10
  • PyCon registration open
  • Python Web Conf reg open
  • Hour of code - minecraft

Joke:

Sent in via Michel Rogers-Vallée, Dan Bader, and Allan Mcelroy. :)

PEP 8 Song

Watch on YouTube

  • By Leon Sandoy and team at Python Discord


share







 2021-02-11  47m