Pablo is a great speaker and I love Tachyon! However, this is a different type of/target for profiling. Tachyon profiles the entire CPython interpreter, this blog post is just type profiling/recording and interpreter path profiling for the CPython core interpreter loop itself.
Typically, the inner loop of an interpreter is all about keeping the branch predictor happy and trying to fit in L1 cache as much as possible. So, falling through makes sense. I’m also curious whether a simple check of a flag and a conditional branch (which is highly predictable for a mode flag like whether you’re tracing or not) wouldn’t be faster than one of the indirect branches. That would keep the tracing code out of L1 when not in use (you can locate it in a remote function).
This is an intuitive step and I kind of did something similar initially (I simplified the diagram a lot, but it was basically what you are suggesting). However, it slowed down the computed goto interpreter by 6% (though not the tail calling interpreter).
The other comment about interpreters being greatly affected by instruction cache is right. The other problem is that PGO in CPython's test suite could be improved and when you have the full thing as a switch-case and trigger the profiling mode, PGO then mistakenly thinks that the RECORD_INST_X section is hot (when it is clearly not). This causes a pessimization in the interpreter loop leading to the 6% slowdown.
The centralized approach they're using does this without exploding in size. I think what you want would be thread local tables to avoid locks/syncing across threads? Would only help for mt, though.
Writing to a ring buffer and processing in batches should be a relatively big W for everything. You could then filter for repeated instructions to compress, which would lend itself to prefetching the next instruction and buffer with hints. Mixed precision record storage might help too, but you're plucking hairs at that point. Making the sampling adaptive based on the hw profiling counters might eliminate some 'useless' work.
The overhead is already so low that it really should not matter, though. Used to spend a lot of time trying to find wins like this but caches have gotten so large it basically doesn't matter. Even in pathologically memory or io bandwidth bound cases I've found it's usually faster to just run 2 or 4 smaller instances over trying to coordinate all of the threads in a single big one.
I'm not clear on why there is `ip++` in the RECORD_INST handler. We've already moved to the next instruction by running `ip++` at the end of the handler in the normal dispatch table. In the RECORD_INST handler we do the work to record the instruction but don't do the actual work of that instruction. We can't, because we only have the one handler that has to work for all instructions. Shouldn't we jump to DISPATCHER_TABLE_NORMAL without incrementing ip again?
A `bool profile` branch stays expensive even when it predicts perfectly, because it bloats every opcode handler, and on a computed goto interpreter that extra code messes with the branch predictor history each dispatch site builds up, which is the whole reason dispatch is fast. Swapping the entire table dodges that. The part I like is fanning every opcode into one recording instruction and then back out through the real table, which is what keeps the second table from turning into a whole second interpreter like the earlier approach did.
Here are explanation on how to use it from one of the people working on it: https://www.youtube.com/watch?v=f1x4X83CDSA and the chunky docs that go with it in the beta of 3.15: https://docs.python.org/3.15/library/profiling.sampling.html
Pablo is a great speaker and I love Tachyon! However, this is a different type of/target for profiling. Tachyon profiles the entire CPython interpreter, this blog post is just type profiling/recording and interpreter path profiling for the CPython core interpreter loop itself.
Uh sorry I mistook this for the 3.15 profiler announcement, thanks for the correction!
Ah, Pablo gave that talk (I guess that a previous version) at PyConES some months ago and it was interesting and the profiling looked very nice ...
Could you duplicate `RECORD_INST` before each `INSTRUCTION_N`, so that:
```
```becomes: ```
```This way you could directly swap `dispatch_var` with an array populated with the `RECORD_INST_*` labels, and remove one step at runtime.
Or maybe this is what you are trying to avoid to reduce the binary size ?
Typically, the inner loop of an interpreter is all about keeping the branch predictor happy and trying to fit in L1 cache as much as possible. So, falling through makes sense. I’m also curious whether a simple check of a flag and a conditional branch (which is highly predictable for a mode flag like whether you’re tracing or not) wouldn’t be faster than one of the indirect branches. That would keep the tracing code out of L1 when not in use (you can locate it in a remote function).
This is an intuitive step and I kind of did something similar initially (I simplified the diagram a lot, but it was basically what you are suggesting). However, it slowed down the computed goto interpreter by 6% (though not the tail calling interpreter).
The other comment about interpreters being greatly affected by instruction cache is right. The other problem is that PGO in CPython's test suite could be improved and when you have the full thing as a switch-case and trigger the profiling mode, PGO then mistakenly thinks that the RECORD_INST_X section is hot (when it is clearly not). This causes a pessimization in the interpreter loop leading to the 6% slowdown.
The centralized approach they're using does this without exploding in size. I think what you want would be thread local tables to avoid locks/syncing across threads? Would only help for mt, though.
Writing to a ring buffer and processing in batches should be a relatively big W for everything. You could then filter for repeated instructions to compress, which would lend itself to prefetching the next instruction and buffer with hints. Mixed precision record storage might help too, but you're plucking hairs at that point. Making the sampling adaptive based on the hw profiling counters might eliminate some 'useless' work.
The overhead is already so low that it really should not matter, though. Used to spend a lot of time trying to find wins like this but caches have gotten so large it basically doesn't matter. Even in pathologically memory or io bandwidth bound cases I've found it's usually faster to just run 2 or 4 smaller instances over trying to coordinate all of the threads in a single big one.
That sounds smart, maybe worth testing, but I think it bloats the instruction cache even when not tracing.
I'm not clear on why there is `ip++` in the RECORD_INST handler. We've already moved to the next instruction by running `ip++` at the end of the handler in the normal dispatch table. In the RECORD_INST handler we do the work to record the instruction but don't do the actual work of that instruction. We can't, because we only have the one handler that has to work for all instructions. Shouldn't we jump to DISPATCHER_TABLE_NORMAL without incrementing ip again?
A `bool profile` branch stays expensive even when it predicts perfectly, because it bloats every opcode handler, and on a computed goto interpreter that extra code messes with the branch predictor history each dispatch site builds up, which is the whole reason dispatch is fast. Swapping the entire table dodges that. The part I like is fanning every opcode into one recording instruction and then back out through the real table, which is what keeps the second table from turning into a whole second interpreter like the earlier approach did.
Reading through this accounts other comments, it feels rather Claude-like. I wouldn't mention this if it weren't a consistent pattern