Ben Clifford @benc@mastodon.hawaga.org.uk

Did a talk for @ParslProject the other month about serialisation that turned into "projecting the ghosts of objects into other processes" and then just did another talk on why we can't use multiprocessing and that also turned into "projecting the ghosts of objects into other processes" so I wonder if I can make some composite blog post out of that

#Python

I'm still wondering about the best way to test various dependency versions for @ParslProject

I want to test a bunch rather than just "the latest" because we work with some distributions that pin earlier versions of stuff...

#Python

Another day, another Python 3.12 removed `distutils` and broke my thing.

At least in this case, I control the environment and did a pin to 3.11 along with "hello to you reading this comment in 2027"

#Python

did something change about += becoming more threadsafe between Python 3.7 and Python 3.10? I have some vague memory that it did but can't find any documentation, and running a few test programs seems to suggest so?

#Python #Threads #Atomics

another day, another "thou shalt not catch an exception"

#Python #exceptions

i made a class called DontSerializeMeBro and if you instantiate it inside structures (modules, functions, classes...) you think shouldn't be serialized, you find out...

#Python

#Python people, I'm looking for a linter that will pick up suspiciously mutable default values for function arguments

(think: `whatever=[]`)

For the codebase I'm working on, probably doing it the other way and whitelisting a small set of known-valid types would be fine too...

Any pointers?

This bugs the shit out of me.

was trying to think of a hashtag that described two small projects I've made (today's #Python picklerun, and years ago, #Haskell acme-smuggler) that give you "things" that look like values in your programming language, but actually they do weird stuff...

settled on #SatansValues

https://github.com/topics/satans-values

I made a horrible thing, `picklerun`: it's a decorator which helps you run a function invocation remotely inside the standard Python deserializer (pickle):

https://github.com/benclifford/picklerun

#Python

copying objects between different python processes really is incredible fuckery disguised by the fact that most of the time it works.

#Python #Serialization

realised I can read the "what's new" for Python 3.8 to discover the features that will be unlocked by @ParslProject dropping Python 3.7 support... rather than regarding that as old news...

#Python

horrific package namespace discovery:

import a.b imports the module `b` that is defined in `a.py` and a.b.q refers to the q defined in that module b.

from a.b import q imports the q defined in a/b.py

which is not the same thing, if a.py defines its own symbol b (for example, by importing a package b from somewhere else that is then implicitly re-exported)

#Python

In addition to my unix process exit traumas this week, another problem is some code that tries to impose a timeout on an already running Python function, and the comedy that ensures from that.

Interruptible computing is apparently hard.

#Python

When your Python decorator makes it so that = assignment can raise a runtime exception...

That only took me a week to diagnose.

#Python

today I am discovering Interestingness in cache shortcuts in Python's lru_cache implementation.

For a cached function s, 3 and 3.0 are treated as different:

>>> s((3.0))
'3.0'
>>> s((3))
'3'

but single element tuples containing those two values are treated as the same:

>>> s((3.0,))
'(3.0,)'
>>> s((3,))
'(3.0,)'

(mostly I discovered this because there's an option to force distinctions between types, but it didn't seem to be necessary in my first simple test above...)

#Python #Caching

dill vs ParamSpec - a battle I wasn't expecting, but apparently here I am.

#mypy #dill #serialization #python

made some notes on "equality" and "values" in Python, as relates to caching/checkpointing and serialisation in a couple of project's I working on/with: @ParslProject and the related funcX/Globus Compute

They're pretty rough, but wondering if I can make a short talk out of them or something blog-posty.

http://www.hawaga.org.uk/ben/tech/beyond-values/beyond-values.html

#Python #ExtensionalEquality #Serialization #HPC

repeated bug I'm finding hard to protect against in Python is creating a future, launching some code elsewhere to eventually populate that future, adding a callback to that future, then the immediately following code assumes that the callback will fire *later* rather than having already fired.

this leads to race conditions that I am struggling to code defensively around

#python

i am once again wrestling with decorator typing in python

#mypy #python

getting @ParslProject working with Python 3.11 (yes yes, i know its almost time for Python 3.12...)

It's always interesting to me to see what minor weirdnesses appear when adding in a new python version.

so far, `IntEnum` now returns its int instead of its textual name...

out version bound on pandas is now so old it has to build from source so that adds 15 minutes onto our CI runs...

and a type checking error that was fixed by a `mypy` upgrade...

#Python