Most efficient conversion of ResultSet to JSON?
The following code converts a ResultSet to a JSON string using JSONArray and JSONObject.
import org.json.JSONArray;
import org.json.JSONObject;
import org.json.JSONException;
import java.sql.SQLException;
import java.sql.ResultSet;
import java.sql.ResultSetMetaData;
public class ResultSetConverter {
public static JSONArray convert( ResultSet rs )
throws SQLException, JSONException
{
JSONArray json = new JSONArray();
ResultSetMetaData rsmd = rs.getMetaData();
while(rs.next()) {
int numColumns = rsmd.getColumnCount();
JSONObject obj = new JSONObject();
for (int i=1; i<numColumns+1; i++) {
String column_name = rsmd.getColumnName(i);
if(rsmd.getColumnType(i)==java.sql.Types.ARRAY){
obj.put(column_name, rs.getArray(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.BIGINT){
obj.put(column_name, rs.getInt(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.BOOLEAN){
obj.put(column_name, rs.getBoolean(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.BLOB){
obj.put(column_name, rs.getBlob(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.DOUBLE){
obj.put(column_name, rs.getDouble(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.FLOAT){
obj.put(column_name, rs.getFloat(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.INTEGER){
obj.put(column_name, rs.getInt(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.NVARCHAR){
obj.put(column_name, rs.getNString(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.VARCHAR){
obj.put(column_name, rs.getString(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.TINYINT){
obj.put(column_name, rs.getInt(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.SMALLINT){
obj.put(column_name, rs.getInt(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.DATE){
obj.put(column_name, rs.getDate(column_name));
}
else if(rsmd.getColumnType(i)==java.sql.Types.TIMESTAMP){
obj.put(column_name, rs.getTimestamp(column_name));
}
else{
obj.put(column_name, rs.getObject(column_name));
}
}
json.put(obj);
}
return json;
}
}
- Is there a faster way?
- Is there a way that uses less memory?
Answer
I think there's a way to use less memory (a fixed and not linear amount depending on data cardinality) but this imply to change the method signature. In fact we may print the Json data directly on an output stream as soon as we fetch them from the ResultSet: the already written data will be garbage collected since we don't need an array that keeps them in memory.
I use GSON that accepts type adapters. I wrote a type adapter to convert ResultSet to JsonArray and it looks very like to your code. I'm waiting the "Gson 2.1: Targeted Dec 31, 2011" release which will have the "Support for user-defined streaming type adapters". Then I'll modify my adapter to be a streaming adapter.
Update
As promised I'm back but not with Gson, instead with Jackson 2. Sorry to be late (of 2 years).
Preface: The key to use less memory of the result itsef is in the "server side" cursor. With this kind of cursors (a.k.a. resultset to Java devs) the DBMS sends data incrementally to client (a.k.a. driver) as the client goes forward with the reading. I think Oracle cursor are server side by default. For MySQL > 5.0.2 look for useCursorFetch at connection url paramenter. Check about your favourite DBMS.
1: So to use less memory we must:
- use server side cursor behind the scene
- use resultset open as read only and, of course, forward only;
- avoid to load all the cursor in a list (or a
JSONArray) but write each row directly on an output line, where for output line I mean an output stream or a writer or also a json generator that wraps an output stream or a writer.
2: As Jackson Documentation says:
Streaming API is best performing (lowest overhead, fastest read/write; other 2 methods build on it)
3: I see you in your code use getInt, getBoolean. getFloat... of ResultSet without wasNull. I expect this can yield problems.
4: I used arrays to cache thinks and to avoid to call getters each iteration. Although not a fan of the switch/case construct, I used it for that int SQL Types.
The answer: Not yet fully tested, it's based on Jackson 2.2:
<dependency>
<groupId>com.fasterxml.jackson.core</groupId>
<artifactId>jackson-databind</artifactId>
<version>2.2.2</version>
</dependency>
The ResultSetSerializer object instructs Jackson on how to serialize (tranform the object to JSON) a ResultSet. It uses the Jackson Streaming API inside. Here the code of a test:
SimpleModule module = new SimpleModule();
module.addSerializer(new ResultSetSerializer());
ObjectMapper objectMapper = new ObjectMapper();
objectMapper.registerModule(module);
[ . . . do the query . . . ]
ResultSet resultset = statement.executeQuery(query);
// Use the DataBind Api here
ObjectNode objectNode = objectMapper.createObjectNode();
// put the resultset in a containing structure
objectNode.putPOJO("results", resultset);
// generate all
objectMapper.writeValue(stringWriter, objectNode);
And, of course, the code of the ResultSetSerializer class:
public class ResultSetSerializer extends JsonSerializer<ResultSet> {
public static class ResultSetSerializerException extends JsonProcessingException{
private static final long serialVersionUID = -914957626413580734L;
public ResultSetSerializerException(Throwable cause){
super(cause);
}
}
@Override
public Class<ResultSet> handledType() {
return ResultSet.class;
}
@Override
public void serialize(ResultSet rs, JsonGenerator jgen, SerializerProvider provider) throws IOException, JsonProcessingException {
try {
ResultSetMetaData rsmd = rs.getMetaData();
int numColumns = rsmd.getColumnCount();
String[] columnNames = new String[numColumns];
int[] columnTypes = new int[numColumns];
for (int i = 0; i < columnNames.length; i++) {
columnNames[i] = rsmd.getColumnLabel(i + 1);
columnTypes[i] = rsmd.getColumnType(i + 1);
}
jgen.writeStartArray();
while (rs.next()) {
boolean b;
long l;
double d;
jgen.writeStartObject();
for (int i = 0; i < columnNames.length; i++) {
jgen.writeFieldName(columnNames[i]);
switch (columnTypes[i]) {
case Types.INTEGER:
l = rs.getInt(i + 1);
if (rs.wasNull()) {
jgen.writeNull();
} else {
jgen.writeNumber(l);
}
break;
case Types.BIGINT:
l = rs.getLong(i + 1);
if (rs.wasNull()) {
jgen.writeNull();
} else {
jgen.writeNumber(l);
}
break;
case Types.DECIMAL:
case Types.NUMERIC:
jgen.writeNumber(rs.getBigDecimal(i + 1));
break;
case Types.FLOAT:
case Types.REAL:
case Types.DOUBLE:
d = rs.getDouble(i + 1);
if (rs.wasNull()) {
jgen.writeNull();
} else {
jgen.writeNumber(d);
}
break;
case Types.NVARCHAR:
case Types.VARCHAR:
case Types.LONGNVARCHAR:
case Types.LONGVARCHAR:
jgen.writeString(rs.getString(i + 1));
break;
case Types.BOOLEAN:
case Types.BIT:
b = rs.getBoolean(i + 1);
if (rs.wasNull()) {
jgen.writeNull();
} else {
jgen.writeBoolean(b);
}
break;
case Types.BINARY:
case Types.VARBINARY:
case Types.LONGVARBINARY:
jgen.writeBinary(rs.getBytes(i + 1));
break;
case Types.TINYINT:
case Types.SMALLINT:
l = rs.getShort(i + 1);
if (rs.wasNull()) {
jgen.writeNull();
} else {
jgen.writeNumber(l);
}
break;
case Types.DATE:
provider.defaultSerializeDateValue(rs.getDate(i + 1), jgen);
break;
case Types.TIMESTAMP:
provider.defaultSerializeDateValue(rs.getTime(i + 1), jgen);
break;
case Types.BLOB:
Blob blob = rs.getBlob(i);
provider.defaultSerializeValue(blob.getBinaryStream(), jgen);
blob.free();
break;
case Types.CLOB:
Clob clob = rs.getClob(i);
provider.defaultSerializeValue(clob.getCharacterStream(), jgen);
clob.free();
break;
case Types.ARRAY:
throw new RuntimeException("ResultSetSerializer not yet implemented for SQL type ARRAY");
case Types.STRUCT:
throw new RuntimeException("ResultSetSerializer not yet implemented for SQL type STRUCT");
case Types.DISTINCT:
throw new RuntimeException("ResultSetSerializer not yet implemented for SQL type DISTINCT");
case Types.REF:
throw new RuntimeException("ResultSetSerializer not yet implemented for SQL type REF");
case Types.JAVA_OBJECT:
default:
provider.defaultSerializeValue(rs.getObject(i + 1), jgen);
break;
}
}
jgen.writeEndObject();
}
jgen.writeEndArray();
} catch (SQLException e) {
throw new ResultSetSerializerException(e);
}
}
}