<qyliss>
I think there are two ways of looking at what I need to do
<qyliss>
Either I need a relatively small total fixed amount of growable buffers, or a large amount of allocs/frees, thousands of times per second
<qyliss>
i.e., I know up front that there will always be 2^n buffers in total, for some n <= 16
<qyliss>
But I do not know in advance what size those buffers will be
<qyliss>
They will probably always be kilobytes, as opposed to bytes or megabytes, though.
<qyliss>
That's not guaranteed but should be extremely common.
<qyliss>
I do have to be prepared to handle a 1GB buffer if it comes up, though.
<qyliss>
The absolute maximum buffer size is 2^32 bytes
<qyliss>
What I'd really like, though, is to not have to write an allocator, at least not now.
<qyliss>
I'd much prefer to use something off-the-shelf, and come back to tune it later
<qyliss>
But I haven't really found much in the way of off-the-shelf here
<qyliss>
cc edef for allocator discussion
<qyliss>
edef has cleverly pointed out elsenet that I don't actually need an allocator
<qyliss>
I can allocate n buffers of the maximum possible size up front, and then use madvise/fallocate to tell the kernel when I'm done with a buffer.
<qyliss>
And then it can clean up those pages until I next use that buffer
<IdleBot_2e4f9b4b>
If you consider a risk of needing 1 GiB sized buffers, sounds like an overcommit-dependent strategy