Repository navigation
GH-1406: Read Float4 values as double and BigDecimal with correct precision - #1407
scruz-denodo wants to merge 1 commit into
Conversation
ArrowFlightJdbcFloat4VectorAccessor widened the float to double in getDouble() and passed it to BigDecimal.valueOf(double) in getBigDecimal(), which exposed the binary approximation of the float: 0.1f was returned as 0.10000000149011612. Convert through the float's decimal representation instead, so getDouble(), getObject(Double.class) and getBigDecimal() return the value the float stands for.
|
Thank you for opening a pull request! Please label the PR with one or more of:
Also, add the 'breaking-change' label if appropriate. See CONTRIBUTING.md for details. |
lidavidm
left a comment
There was a problem hiding this comment.
The issue states:
For a finite float, getDouble(), getBigDecimal() and getObject(Double.class) should return the decimal value the float represents. That is the shortest decimal that uniquely identifies the float, the same one Float.toString yields. For 0.1f that is 0.1, which is what getString() already returns. Other JDBC drivers show this behaviour for REAL columns.
Where is it documented that this is expected? What other JDBC drivers show this behavior? IMO, converting float to double via the string representation is very unexpected.
|
Hi, Here is a testing class showing the root cause and results returned by Postgresq, SQLServer, Oracle, Gizmo (for testing a flight-sql database) and Duckdb. package scruz.test;
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
public class FloatTests {
public static void main(String args[]) throws Exception {
TestConfig[] testConfigs = new TestConfig[]{
new TestConfig("jdbc:postgresql://localhost:5432/vdp", "org.postgresql.Driver", "xxxx", "xxxx", "SELECT CAST(0.1 AS real)"),
new TestConfig("jdbc:sqlserver://<host>:1433;databaseName=master;encrypt=false", "com.microsoft.sqlserver.jdbc.SQLServerDriver", "xxxx", "xxxx", "SELECT CAST(0.1 AS real);"),
new TestConfig("jdbc:oracle:thin:@<host>:1521/ORCLPDB1", "oracle.jdbc.OracleDriver", "xxxx", "xxxx", "SELECT CAST(0.1 AS real) FROM dual"),
new TestConfig("jdbc:arrow-flight-sql://localhost:31337?useEncryption=0&disableCertificateVerification=true", "org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver", "admin", "admin", "SELECT CAST(0.1 AS real)"),
new TestConfig("jdbc:duckdb:", "org.duckdb.DuckDBDriver", "", "", "SELECT CAST(0.1 AS real)"),
};
for (TestConfig testConfig : testConfigs) {
Class.forName(testConfig.DRIVER);
System.out.println("\nTesting " + testConfig.URI);
try (Connection connection = DriverManager.getConnection(testConfig.URI, testConfig.USER, testConfig.PASSWORD);
PreparedStatement pst = connection.prepareStatement(testConfig.QUERY);
ResultSet rs = pst.executeQuery()) {
rs.next();
System.out.println("getString() " + rs.getString(1));
System.out.println("getObject() " + rs.getObject(1));
System.out.println("getFloat() " + rs.getFloat(1));
System.out.println("getDouble() " + rs.getDouble(1));
System.out.println("getBigDecimal() " + rs.getBigDecimal(1));
}
}
//Java isolated test
System.out.println("\nJava isolated");
final float value = 0.1f;
System.out.println("value: " + value);
System.out.println("value (with cast double): " + (double)value);
System.out.println("value (with double + toString): " + Double.parseDouble(Float.toString(value)));
}
private static class TestConfig {
final String URI;
final String DRIVER;
final String USER;
final String PASSWORD;
final String QUERY;
TestConfig(String URI, String DRIVER, String USER, String PASSWORD, String QUERY) {
this.URI = URI;
this.DRIVER = DRIVER;
this.USER = USER;
this.PASSWORD = PASSWORD;
this.QUERY = QUERY;
}
}
}The output shows that:
The isolated java test shows the root cause and the proposed solution. This is the entire output. Testing jdbc:postgresql://localhost:5432/vdp
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.1
getBigDecimal() 0.1
Testing jdbc:sqlserver://<host>:1433;databaseName=master;encrypt=false
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.10000000149011612
getBigDecimal() 0.1
Testing jdbc:oracle:thin:@<host>:1521/ORCLPDB1
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.1
getBigDecimal() 0.1
Testing jdbc:arrow-flight-sql://localhost:31337?useEncryption=0&disableCertificateVerification=true
getString() 0.1
getObject() 0.10000000149011612
getFloat() 0.1
getDouble() 0.10000000149011612
getBigDecimal() 0.10000000149011612
Testing jdbc:duckdb:
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.10000000149011612
getBigDecimal() 0.1
Java isolated
value: 0.1
value (with cast double): 0.10000000149011612
value (with double + toString): 0.1About the proposed solution, we are already using that solution for avoiding the issue on other sites for a long time. |
|
Given that the drivers generally don't agree, then I don't think that shows the behavior for Double is actually a problem. I can be convinced that BigDecimal should be changed. |
|
Hi, In a sense, the float type is a subset of the double type.
About this, I understand that you agree to the change but only affecting to I would like to argue that at least The reason is that method public static Object get(Cursor.Accessor accessor, int targetSqlType, boolean signed, Calendar localCalendar) throws SQLException {
....
case 6:
case 8:
double aDouble = accessor.getDouble();
if (aDouble == (double)0.0F && accessor.wasNull()) {
return null;
}
return aDouble;That said, fixing the |
|
But a float can't actually represent Also, for PostgreSQL, it appears what you're seeing is actually an artifact of text vs binary format; when the binary format is used, you instead get Detailsimport java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.Properties;
public class PgjdbcFloatReproducer {
public static void main(String[] args) throws Exception {
if (args.length != 3) {
throw new IllegalArgumentException(
"Usage: PgjdbcFloatReproducer <jdbc-url> <username> <password>");
}
for (int threshold : new int[] {0, -1}) {
Properties properties = new Properties();
properties.setProperty("user", args[1]);
properties.setProperty("password", args[2]);
properties.setProperty("prepareThreshold", Integer.toString(threshold));
properties.setProperty("binaryTransfer", "true");
properties.setProperty("preferQueryMode", "extended");
try (Connection connection = DriverManager.getConnection(args[0], properties);
PreparedStatement statement = connection.prepareStatement("SELECT CAST(0.1 AS real)");
ResultSet rs = statement.executeQuery()) {
rs.next();
System.out.println("\nprepareThreshold=" + threshold);
System.out.println("getString() " + rs.getString(1));
System.out.println("getObject() " + rs.getObject(1));
System.out.println("getFloat() " + rs.getFloat(1));
System.out.println("getDouble() " + rs.getDouble(1));
System.out.println("getBigDecimal() " + rs.getBigDecimal(1));
}
}
}
} |
|
Also for Oracle, we should be testing BINARY_FLOAT and not REAL. |
|
Hi, I modified the attached class for accessing to a table instead of passing the number as argument with the I tested also with Oracle. Both of them (Postgresql and Oracle) return I paste the output and the modified code: PostgresqlTable create table test_float (
c_decimal decimal(4, 2),
c_real real,
c_double double precision
);
insert into test_float (c_decimal, c_real, c_double) values (0.1, 0.1, 0.1);Output OracleTable create table test_float (
c_number number(4, 2),
c_float float,
c_binary_float binary_float,
c_binary_double binary_double,
c_double double precision,
c_real real
);
insert into test_float (c_number, c_float, c_binary_float, c_binary_double, c_double, c_real) values (0.1, 0.1, 0.1, 0.1, 0.1, 0.1);And the modified Java code: import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.Properties;
public class PgjdbcFloatReproducer {
public static void main(String[] args) throws Exception {
if (args.length != 3) {
throw new IllegalArgumentException(
"Usage: PgjdbcFloatReproducer <jdbc-url> <username> <password>");
}
Properties properties = new Properties();
properties.setProperty("user", args[1]);
properties.setProperty("password", args[2]);
try (Connection connection = DriverManager.getConnection(args[0], properties);
PreparedStatement statement = connection.prepareStatement("SELECT * from test_float");
ResultSet rs = statement.executeQuery()) {
while (rs.next()) {
for (int i=1;i<=rs.getMetaData().getColumnCount();i++) {
System.out.println(rs.getMetaData().getColumnName(i));
System.out.println("....................");
System.out.println("\tgetString() " + rs.getString(1));
System.out.println("\tgetObject() " + rs.getObject(1));
System.out.println("\tgetFloat() " + rs.getFloat(1));
System.out.println("\tgetDouble() " + rs.getDouble(1));
System.out.println("\tgetBigDecimal() " + rs.getBigDecimal(1) + "\n");
}
}
}
}
} |
What's Changed
ArrowFlightJdbcFloat4VectorAccessorno longer exposes float-widening artifacts when aFloat4Vectoris read as adoubleor adecimal.getDouble()(and thereforegetObject(Double.class)) now returnsDouble.parseDouble(Float.toString(value))instead of(double) value. A0.1fcell now gives0.1,not0.10000000149011612.getBigDecimal()andgetBigDecimal(int scale)now usenew BigDecimal(Float.toString(value))instead ofBigDecimal.valueOf(value), which widened the float to a double first.NaNand±Infinityare still returned bygetDouble().getBigDecimal()still throwsSQLExceptionfor them, andnullcells still return0.0/null.Tests (ArrowFlightJdbcFloat4VectorAccessorTest):
getDoubleandgetBigDecimalexpectations now use Float.toString.(float) getDouble() == getFloat().getDouble,getObject(Double.class),getBigDecimal()andgetBigDecimal(9).NaN getDoubletest is added.Closes #1406.