org.apache.drill.exec.server.rest.stream

Class StreamingHttpConnection

java.lang.Object

org.apache.drill.exec.rpc.AbstractDisposableUserClientConnection

org.apache.drill.exec.server.rest.BaseWebUserConnection

org.apache.drill.exec.server.rest.stream.StreamingHttpConnection

All Implemented Interfaces:

ConnectionThrottle, UserClientConnection

public class StreamingHttpConnection
extends BaseWebUserConnection

Stream the results of a query to a REST client as JSON, following the schema defined by QueryResult to maintain backward compatibility. That schema is not ideal for streaming, but it is what it is.

Streaming requires that we use a low-level JSON writer and handle serialization ourselves. While we could build up, then serialize objects, there is little advantage of doing so, and it would simply increase cost -- the very thing that this design tries to decrease.

Does minimal pretty-printing: just inserts newlines in a few places.

Enforces the row limit, discarding all rows after the requested limit. Trusts that the query will enforce the limit at the batch level to avoid wasting resources.

Field Summary

Fields inherited from class org.apache.drill.exec.server.rest.BaseWebUserConnection

webSessionResources

Fields inherited from class org.apache.drill.exec.rpc.AbstractDisposableUserClientConnection

error, latch, queryState

Constructor Summary

Constructors

Constructor and Description
StreamingHttpConnection(WebSessionResources webSessionResources)

Method Summary

Modifier and Type	Method and Description
void	emitBatch(RowSetReader batchReader)
void	finish() Called from the REST request, after the query completes, to emit the end of the JSON payload:
void	onStart(UserBitShared.QueryId queryId, int rowLimit) Provide query info once the query starts.
void	outputAvailable(OutputStream out) Set the output stream.
void	sendData(RpcOutcomeListener<GeneralRPCProtos.Ack> listener, QueryDataPackage data) Called from query thread, specifically from the Screen operator, for each batch.
void	writeNewline(JsonOutput gen)

Methods inherited from class org.apache.drill.exec.server.rest.BaseWebUserConnection

getClosureFuture, getRemoteAddress, getSession, resources, setAutoRead, webDataType

Methods inherited from class org.apache.drill.exec.rpc.AbstractDisposableUserClientConnection

await, await, getError, getQueryState, sendResult

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

StreamingHttpConnection

public StreamingHttpConnection(WebSessionResources webSessionResources)

Method Detail

onStart

public void onStart(UserBitShared.QueryId queryId,
                    int rowLimit)

Provide query info once the query starts. Sent from the REST request thread.

outputAvailable

public void outputAvailable(OutputStream out)
                     throws IOException

Set the output stream. Sent from the REST request thread in the StreamingOutput callback once the output stream is available. Unblocks the Screen thread.

Throws:

IOException

sendData

public void sendData(RpcOutcomeListener<GeneralRPCProtos.Ack> listener,
                     QueryDataPackage data)

Called from query thread, specifically from the Screen operator, for each batch.

Parameters:

listener - The listener

data - The data to be sent

emitBatch

public void emitBatch(RowSetReader batchReader)
               throws IOException

Throws:

IOException

writeNewline

public void writeNewline(JsonOutput gen)
                  throws IOException

Throws:

IOException

finish

public void finish()
            throws IOException

Called from the REST request, after the query completes, to emit the end of the JSON payload:

   } ],
   "queryState" : "COMPLETED"
 }
 

Admittedly the tail is pretty lame, but we need it to maintain backward compatibility.

Note that, under the original design, there is no good way to report an error that occurs once the query starts running. Here, we report the query state, which can indicate an error. But, the design of the API does not currently provide the error message itself.

Throws:

IOException