Package org.apache.drill.exec.memory

Interface BufferAllocator

All Superinterfaces:
AutoCloseable
All Known Implementing Classes:
BaseAllocator, RootAllocator
public interface BufferAllocator extends AutoCloseable
Wrapper class to deal with byte buffer allocation. Ensures users only use designated methods.

  • Method Summary

    Modifier and Type
    Method
    Description
    void
    assertOpen()
    Asserts (using java assertions) that the provided allocator is currently open.
    DrillBuf
    buffer(int size)
    Allocate a new or reused buffer of the provided size.
    DrillBuf
    buffer(int size, BufferManager manager)
    Allocate a new or reused buffer of the provided size.
    void
    close()
    Close and release all buffers generated from this buffer pool.
    long
    getAllocatedMemory()
    Returns the amount of memory currently allocated from this allocator.
    io.netty.buffer.ByteBufAllocator
    getAsByteBufAllocator()
    Returns the allocator this allocator falls back to when it needs more memory.
    DrillBuf
    getEmpty()
    Get a reference to the empty buffer associated with this allocator.
    long
    getLimit()
    Return the current maximum limit this allocator imposes.
    String
    getName()
    Return the name of this allocator.
    long
    getPeakMemoryAllocation()
    Returns the peak amount of memory allocated from this allocator.
    boolean
    isOverLimit()
    Return whether or not this allocator (or one if its parents) is over its limits.
    BufferAllocator
    newChildAllocator(String name, long initReservation, long maxAllocation)
    Create a new child allocator.
    AllocationReservation
    newReservation()
    Create an allocation reservation.
    DrillBuf
    read(int length, InputStream in)
    Reads the specified number of bytes into a new Drillbuf.
    void
    read(DrillBuf buf, int length, InputStream in)
    Read the contents of a DrillBuf from a stream.
    boolean
    setLenient()
    Request lenient enforcement of the allocation limits.
    void
    setLimit(long newLimit)
    Set the maximum amount of memory this allocator is allowed to allocate.
    String
    toVerboseString()
    Return a verbose string describing this allocator.
    void
    write(DrillBuf buf, int length, OutputStream out)
    Write the contents of a DrillBuf to a stream.
    void
    write(DrillBuf buf, OutputStream out)
    Write the contents of a DrillBuf to a stream.

  • Method Details

    • buffer

      DrillBuf buffer(int size)
      Allocate a new or reused buffer of the provided size. Note that the buffer may technically be larger than the requested size for rounding purposes. However, the buffer's capacity will be set to the configured size.
      Parameters:
      size - The size in bytes.
      Returns:
      a new DrillBuf, or null if the request can't be satisfied
      Throws:
      OutOfMemoryException - if buffer cannot be allocated

    • buffer

      DrillBuf buffer(int size, BufferManager manager)
      Allocate a new or reused buffer of the provided size. Note that the buffer may technically be larger than the requested size for rounding purposes. However, the buffer's capacity will be set to the configured size.
      Parameters:
      size - The size in bytes.
      manager - A buffer manager to manage reallocation.
      Returns:
      a new DrillBuf, or null if the request can't be satisfied
      Throws:
      OutOfMemoryException - if buffer cannot be allocated

    • getAsByteBufAllocator

      io.netty.buffer.ByteBufAllocator getAsByteBufAllocator()
      Returns the allocator this allocator falls back to when it needs more memory.
      Returns:
      the underlying allocator used by this allocator

    • newChildAllocator

      BufferAllocator newChildAllocator(String name, long initReservation, long maxAllocation)
      Create a new child allocator.
      Parameters:
      name - the name of the allocator.
      initReservation - the initial space reservation (obtained from this allocator)
      maxAllocation - maximum amount of space the new allocator can allocate
      Returns:
      the new allocator, or null if it can't be created

    • close

      void close()
      Close and release all buffers generated from this buffer pool.

      When assertions are on, complains if there are any outstanding buffers; to avoid that, release all buffers before the allocator is closed.

      Specified by:
      close in interface AutoCloseable

    • getAllocatedMemory

      long getAllocatedMemory()
      Returns the amount of memory currently allocated from this allocator.
      Returns:
      the amount of memory currently allocated

    • setLimit

      void setLimit(long newLimit)
      Set the maximum amount of memory this allocator is allowed to allocate.
      Parameters:
      newLimit - The new Limit to apply to allocations

    • setLenient

      boolean setLenient()
      Request lenient enforcement of the allocation limits. Use for memory-managed operators to prevent minor math errors from killing queries. This is temporary until Drill manages memory better. Leniency is allowed only in production code (no assertions), not in debug mode (assertions enabled).
      Returns:
      true if leniency was granted, false if the allocator will enforce strict limits despite the request

    • getLimit

      long getLimit()
      Return the current maximum limit this allocator imposes.
      Returns:
      Limit in number of bytes.

    • getPeakMemoryAllocation

      long getPeakMemoryAllocation()
      Returns the peak amount of memory allocated from this allocator.
      Returns:
      the peak amount of memory allocated

    • newReservation

      AllocationReservation newReservation()
      Create an allocation reservation. A reservation is a way of building up a request for a buffer whose size is not known in advance. See .
      Returns:
      the newly created reservation

    • getEmpty

      DrillBuf getEmpty()
      Get a reference to the empty buffer associated with this allocator. Empty buffers are special because we don't worry about them leaking or managing reference counts on them since they don't actually point to any memory.

    • getName

      String getName()
      Return the name of this allocator. This is a human readable name that can help debugging. Typically provides coordinates about where this allocator was created

    • isOverLimit

      boolean isOverLimit()
      Return whether or not this allocator (or one if its parents) is over its limits. In the case that an allocator is over its limit, all consumers of that allocator should aggressively try to addrss the overlimit situation.

    • toVerboseString

      String toVerboseString()
      Return a verbose string describing this allocator. If in DEBUG mode, this will also include relevant stacktraces and historical logs for underlying objects
      Returns:
      A very verbose description of the allocator hierarchy.

    • assertOpen

      void assertOpen()
      Asserts (using java assertions) that the provided allocator is currently open. If assertions are disabled, this is a no-op.

    • write

      void write(DrillBuf buf, OutputStream out) throws IOException
      Write the contents of a DrillBuf to a stream. Use this method, rather than calling the DrillBuf.getBytes() method, because this method avoids repeated heap allocation for the intermediate heap buffer. Uses the reader and writer indexes to determine the number of bytes to write. Useful only for bufs created using those indexes.
      Parameters:
      buf - the Drillbuf to write
      out - the output stream
      Throws:
      IOException - if a write error occurs

    • write

      void write(DrillBuf buf, int length, OutputStream out) throws IOException
      Write the contents of a DrillBuf to a stream. Use this method, rather than calling the DrillBuf.getBytes() method, because this method avoids repeated heap allocation for the intermediate heap buffer. Writes the specified number of bytes starting from the head of the given Drillbuf.
      Parameters:
      buf - the Drillbuf to write
      length - the number of bytes to read. Must be less than or equal to number of bytes allocated in the buffer.
      out - the output stream
      Throws:
      IOException - if a write error occurs

    • read

      void read(DrillBuf buf, int length, InputStream in) throws IOException
      Read the contents of a DrillBuf from a stream. Use this method, rather than calling the DrillBuf.writeBytes() method, because this method avoids repeated heap allocation for the intermediate heap buffer. The buffer must have already been allocated.
      Parameters:
      buf - the buffer to read with space already allocated
      length - number of bytes to read
      in - input stream from which to read data
      Throws:
      IOException - if a read error occurs

    • read

      DrillBuf read(int length, InputStream in) throws IOException
      Reads the specified number of bytes into a new Drillbuf.
      Parameters:
      length - number of bytes to read
      in - input stream from which to read data
      Returns:
      the buffer holding the data read from the stream
      Throws:
      IOException - if a read error occurs