Package org.apache.drill.exec.server.rest.stream

Class StreamingHttpConnection

java.lang.Object
org.apache.drill.exec.rpc.AbstractDisposableUserClientConnection
org.apache.drill.exec.server.rest.BaseWebUserConnection
org.apache.drill.exec.server.rest.stream.StreamingHttpConnection
All Implemented Interfaces:
ConnectionThrottle, UserClientConnection
public class StreamingHttpConnection extends BaseWebUserConnection
Stream the results of a query to a REST client as JSON, following the schema defined by QueryResult to maintain backward compatibility. That schema is not ideal for streaming, but it is what it is.

Streaming requires that we use a low-level JSON writer and handle serialization ourselves. While we could build up, then serialize objects, there is little advantage of doing so, and it would simply increase cost -- the very thing that this design tries to decrease.

Does minimal pretty-printing: just inserts newlines in a few places.

Enforces the row limit, discarding all rows after the requested limit. Trusts that the query will enforce the limit at the batch level to avoid wasting resources.

  • Constructor Details

    • StreamingHttpConnection

      public StreamingHttpConnection(WebSessionResources webSessionResources)

  • Method Details

    • onStart

      public void onStart(UserBitShared.QueryId queryId, int rowLimit)
      Provide query info once the query starts. Sent from the REST request thread.

    • outputAvailable

      public void outputAvailable(OutputStream out) throws IOException
      Set the output stream. Sent from the REST request thread in the StreamingOutput callback once the output stream is available. Unblocks the Screen thread.
      Throws:
      IOException

    • sendData

      public void sendData(RpcOutcomeListener<GeneralRPCProtos.Ack> listener, QueryDataPackage data)
      Called from query thread, specifically from the Screen operator, for each batch.
      Parameters:
      listener - The listener
      data - The data to be sent

    • emitBatch

      public void emitBatch(RowSetReader batchReader) throws IOException
      Throws:
      IOException

    • writeNewline

      public void writeNewline(JsonOutput gen) throws IOException
      Throws:
      IOException

    • finish

      public void finish() throws IOException
      Called from the REST request, after the query completes, to emit the end of the JSON payload: 
      
   } ],
   "queryState" : "COMPLETED"
 }

      Admittedly the tail is pretty lame, but we need it to maintain backward compatibility.

      Note that, under the original design, there is no good way to report an error that occurs once the query starts running. Here, we report the query state, which can indicate an error. But, the design of the API does not currently provide the error message itself.

      Throws:
      IOException