Did a talk for @ParslProject the other month about serialisation that turned into "projecting the ghosts of objects into other processes" and then just did another talk on why we can't use multiprocessing and that also turned into "projecting the ghosts of objects into other processes" so I wonder if I can make some composite blog post out of that
I'm still wondering about the best way to test various dependency versions for @ParslProject
I want to test a bunch rather than just "the latest" because we work with some distributions that pin earlier versions of stuff...
Another day, another Python 3.12 removed `distutils` and broke my thing.
At least in this case, I control the environment and did a pin to 3.11 along with "hello to you reading this comment in 2027"
another day, another "thou shalt not catch an exception"
i made a class called DontSerializeMeBro and if you instantiate it inside structures (modules, functions, classes...) you think shouldn't be serialized, you find out...
#Python people, I'm looking for a linter that will pick up suspiciously mutable default values for function arguments
(think: `whatever=[]`)
For the codebase I'm working on, probably doing it the other way and whitelisting a small set of known-valid types would be fine too...
Any pointers?
This bugs the shit out of me.
was trying to think of a hashtag that described two small projects I've made (today's #Python picklerun, and years ago, #Haskell acme-smuggler) that give you "things" that look like values in your programming language, but actually they do weird stuff...
settled on #SatansValues
I made a horrible thing, `picklerun`: it's a decorator which helps you run a function invocation remotely inside the standard Python deserializer (pickle):
copying objects between different python processes really is incredible fuckery disguised by the fact that most of the time it works.
realised I can read the "what's new" for Python 3.8 to discover the features that will be unlocked by @ParslProject dropping Python 3.7 support... rather than regarding that as old news...
horrific package namespace discovery:
import a.b imports the module `b` that is defined in `a.py` and a.b.q refers to the q defined in that module b.
from a.b import q imports the q defined in a/b.py
which is not the same thing, if a.py defines its own symbol b (for example, by importing a package b from somewhere else that is then implicitly re-exported)
In addition to my unix process exit traumas this week, another problem is some code that tries to impose a timeout on an already running Python function, and the comedy that ensures from that.
Interruptible computing is apparently hard.
When your Python decorator makes it so that = assignment can raise a runtime exception...
That only took me a week to diagnose.
today I am discovering Interestingness in cache shortcuts in Python's lru_cache implementation.
For a cached function s, 3 and 3.0 are treated as different:
>>> s((3.0))
'3.0'
>>> s((3))
'3'
but single element tuples containing those two values are treated as the same:
>>> s((3.0,))
'(3.0,)'
>>> s((3,))
'(3.0,)'
(mostly I discovered this because there's an option to force distinctions between types, but it didn't seem to be necessary in my first simple test above...)
dill vs ParamSpec - a battle I wasn't expecting, but apparently here I am.
made some notes on "equality" and "values" in Python, as relates to caching/checkpointing and serialisation in a couple of project's I working on/with: @ParslProject and the related funcX/Globus Compute
They're pretty rough, but wondering if I can make a short talk out of them or something blog-posty.
http://www.hawaga.org.uk/ben/tech/beyond-values/beyond-values.html
repeated bug I'm finding hard to protect against in Python is creating a future, launching some code elsewhere to eventually populate that future, adding a callback to that future, then the immediately following code assumes that the callback will fire *later* rather than having already fired.
this leads to race conditions that I am struggling to code defensively around
getting @ParslProject working with Python 3.11 (yes yes, i know its almost time for Python 3.12...)
It's always interesting to me to see what minor weirdnesses appear when adding in a new python version.
so far, `IntEnum` now returns its int instead of its textual name...
out version bound on pandas is now so old it has to build from source so that adds 15 minutes onto our CI runs...
and a type checking error that was fixed by a `mypy` upgrade...
Hi.
I'm Ben.
For fun I ride my bike around Berlin and build pretty things with LEDs and microcontrollers.
I'm also interested in programming language design/theory stuff.
I help with a refugee reception volunteer group in Berlin for people arriving from Ukraine.
I like dogs.
http://www.hawaga.org.uk/ben/
https://github.com/benclifford
https://twitter.com/benclifford