Skip to content

GH-1406: Read Float4 values as double and BigDecimal with correct precision - #1407

Open
scruz-denodo wants to merge 1 commit into
apache:mainfrom
scruz-denodo:float4-precision
Open

scruz-denodo wants to merge 1 commit into
apache:mainfrom
scruz-denodo:float4-precision

Conversation

@scruz-denodo

Copy link
Copy Markdown

What's Changed

ArrowFlightJdbcFloat4VectorAccessor no longer exposes float-widening artifacts when a Float4Vector is read as a double or a decimal.

  • getDouble() (and therefore getObject(Double.class)) now returns Double.parseDouble(Float.toString(value)) instead of (double) value. A 0.1f cell now gives 0.1, not 0.10000000149011612.
  • getBigDecimal() and getBigDecimal(int scale) now use new BigDecimal(Float.toString(value)) instead of BigDecimal.valueOf(value), which widened the float to a double first.
  • NaN and ±Infinity are still returned by getDouble(). getBigDecimal() still throws SQLException for them, and null cells still return 0.0 / null.

Tests (ArrowFlightJdbcFloat4VectorAccessorTest):

  • getDouble and getBigDecimal expectations now use Float.toString.
  • A new test checks that (float) getDouble() == getFloat().
  • A new test covers 0.1f, -0.1f, 0.2f, 0.3f, 1.1f, 123.456f, 3.4e-5f and -0.0f through getDouble, getObject(Double.class), getBigDecimal() and getBigDecimal(9).
  • A 0.1f test checks 0.1, 0.1 and 0.100000000.
  • A NaN getDouble test is added.

Closes #1406.

ArrowFlightJdbcFloat4VectorAccessor widened the float to double in getDouble() and passed it to BigDecimal.valueOf(double) in getBigDecimal(), which exposed the binary approximation of the float: 0.1f was returned as 0.10000000149011612. Convert through the float's decimal representation instead, so getDouble(), getObject(Double.class) and getBigDecimal() return the value the float stands for.
@github-actions

Copy link
Copy Markdown

Thank you for opening a pull request!

Please label the PR with one or more of:

  • bug-fix
  • chore
  • dependencies
  • documentation
  • enhancement

Also, add the 'breaking-change' label if appropriate.

See CONTRIBUTING.md for details.

@lidavidm lidavidm left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue states:

For a finite float, getDouble(), getBigDecimal() and getObject(Double.class) should return the decimal value the float represents. That is the shortest decimal that uniquely identifies the float, the same one Float.toString yields. For 0.1f that is 0.1, which is what getString() already returns. Other JDBC drivers show this behaviour for REAL columns.

Where is it documented that this is expected? What other JDBC drivers show this behavior? IMO, converting float to double via the string representation is very unexpected.

@scruz-denodo

Copy link
Copy Markdown
Author

Hi,
the root cause is with how Java does the cast from the float. In that process is where the precision of the initial number is lost.

Here is a testing class showing the root cause and results returned by Postgresq, SQLServer, Oracle, Gizmo (for testing a flight-sql database) and Duckdb.

package scruz.test;

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;

public class FloatTests {

    public static void main(String args[]) throws Exception {
        TestConfig[] testConfigs = new TestConfig[]{
            new TestConfig("jdbc:postgresql://localhost:5432/vdp", "org.postgresql.Driver", "xxxx", "xxxx", "SELECT CAST(0.1 AS real)"),
            new TestConfig("jdbc:sqlserver://<host>:1433;databaseName=master;encrypt=false", "com.microsoft.sqlserver.jdbc.SQLServerDriver", "xxxx", "xxxx", "SELECT CAST(0.1 AS real);"),
            new TestConfig("jdbc:oracle:thin:@<host>:1521/ORCLPDB1", "oracle.jdbc.OracleDriver", "xxxx", "xxxx", "SELECT CAST(0.1 AS real) FROM dual"),
            new TestConfig("jdbc:arrow-flight-sql://localhost:31337?useEncryption=0&disableCertificateVerification=true", "org.apache.arrow.driver.jdbc.ArrowFlightJdbcDriver", "admin", "admin", "SELECT CAST(0.1 AS real)"),
            new TestConfig("jdbc:duckdb:", "org.duckdb.DuckDBDriver", "", "", "SELECT CAST(0.1 AS real)"),
        };

        for (TestConfig testConfig : testConfigs) {
            Class.forName(testConfig.DRIVER);
            System.out.println("\nTesting " + testConfig.URI);
            try (Connection connection = DriverManager.getConnection(testConfig.URI, testConfig.USER, testConfig.PASSWORD);
                 PreparedStatement pst = connection.prepareStatement(testConfig.QUERY);
                 ResultSet rs = pst.executeQuery()) {
                rs.next();
                System.out.println("getString() " + rs.getString(1));
                System.out.println("getObject() " + rs.getObject(1));
                System.out.println("getFloat() " + rs.getFloat(1));
                System.out.println("getDouble() " + rs.getDouble(1));
                System.out.println("getBigDecimal() " + rs.getBigDecimal(1));
            }
        }

        //Java isolated test
        System.out.println("\nJava isolated");
        final float value = 0.1f;
        System.out.println("value: " + value);
        System.out.println("value (with cast double): " + (double)value);
        System.out.println("value (with double + toString): " + Double.parseDouble(Float.toString(value)));
    }

    private static class TestConfig {
        final String URI;
        final String DRIVER;
        final String USER;
        final String PASSWORD;
        final String QUERY;

        TestConfig(String URI, String DRIVER, String USER, String PASSWORD, String QUERY) {
            this.URI = URI;
            this.DRIVER = DRIVER;
            this.USER = USER;
            this.PASSWORD = PASSWORD;
            this.QUERY = QUERY;
        }
    }
}

The output shows that:

  • Oracle, and Postgresql do not have problems with the conversion.
  • SQLServer and Duckdb have problem with the getDouble method returning 0.10000000149011612
  • Gizmo through flight-sql JDBC driver have problem with getObject, getDouble and getBigDecimal

The isolated java test shows the root cause and the proposed solution.

This is the entire output.

Testing jdbc:postgresql://localhost:5432/vdp
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.1
getBigDecimal() 0.1

Testing jdbc:sqlserver://<host>:1433;databaseName=master;encrypt=false
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.10000000149011612
getBigDecimal() 0.1

Testing jdbc:oracle:thin:@<host>:1521/ORCLPDB1
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.1
getBigDecimal() 0.1

Testing jdbc:arrow-flight-sql://localhost:31337?useEncryption=0&disableCertificateVerification=true
getString() 0.1
getObject() 0.10000000149011612
getFloat() 0.1
getDouble() 0.10000000149011612
getBigDecimal() 0.10000000149011612

Testing jdbc:duckdb:
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.10000000149011612
getBigDecimal() 0.1

Java isolated
value: 0.1
value (with cast double): 0.10000000149011612
value (with double + toString): 0.1

About the proposed solution, we are already using that solution for avoiding the issue on other sites for a long time.
I was searching for alternatives but I do not find any other fix which can deal with the 100% of scenarios.

@lidavidm

Copy link
Copy Markdown
Member

Given that the drivers generally don't agree, then I don't think that shows the behavior for Double is actually a problem. I can be convinced that BigDecimal should be changed.

@scruz-denodo

Copy link
Copy Markdown
Author

Hi,
one last try to convince you that PostgreSQL and Oracle behavior are better than SQLServer and Duckdb.

In a sense, the float type is a subset of the double type.
You have a value such 0.1 in a float. So it is expected, in my opinion, that you can get that exact number on a double variable, because the double type is able to represent the same number.

Given that the drivers generally don't agree, then I don't think that shows the behavior for Double is actually a problem. I can be convinced that BigDecimal should be changed.

About this, I understand that you agree to the change but only affecting to getBigDecimal method.

I would like to argue that at least getObject should be also modified. The flight-sql driver is the unique from the tested drivers at previous comment which is returning the number with extra decimals in that case.

The reason is that method org.apache.calcite.avatica.AvaticaSite#get is relying on the accessor.getDouble() method for FLOAT and DOUBLE sql types. So the returned value in this case is 0.10000000149011612

    public static Object get(Cursor.Accessor accessor, int targetSqlType, boolean signed, Calendar localCalendar) throws SQLException {
     ....
            case 6:
            case 8:
                double aDouble = accessor.getDouble();
                if (aDouble == (double)0.0F && accessor.wasNull()) {
                    return null;
                }

                return aDouble;

That said, fixing the getObject method means to fix the getDouble, due the Avatica implementation. So, in the end, the five methods would be modified (same initial PR objective)

@lidavidm

Copy link
Copy Markdown
Member

But a float can't actually represent 0.1 exactly. We just write and print that as shorthand for the closest value.

Also, for PostgreSQL, it appears what you're seeing is actually an artifact of text vs binary format; when the binary format is used, you instead get 0.1000....

prepareThreshold=0
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.1
getBigDecimal() 0.1

prepareThreshold=-1
getString() 0.1
getObject() 0.1
getFloat() 0.1
getDouble() 0.10000000149011612
getBigDecimal() 0.1
Details
import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.Properties;

public class PgjdbcFloatReproducer {
  public static void main(String[] args) throws Exception {
    if (args.length != 3) {
      throw new IllegalArgumentException(
          "Usage: PgjdbcFloatReproducer <jdbc-url> <username> <password>");
    }

    for (int threshold : new int[] {0, -1}) {
      Properties properties = new Properties();
      properties.setProperty("user", args[1]);
      properties.setProperty("password", args[2]);
      properties.setProperty("prepareThreshold", Integer.toString(threshold));
      properties.setProperty("binaryTransfer", "true");
      properties.setProperty("preferQueryMode", "extended");

      try (Connection connection = DriverManager.getConnection(args[0], properties);
          PreparedStatement statement = connection.prepareStatement("SELECT CAST(0.1 AS real)");
          ResultSet rs = statement.executeQuery()) {
        rs.next();
        System.out.println("\nprepareThreshold=" + threshold);
        System.out.println("getString() " + rs.getString(1));
        System.out.println("getObject() " + rs.getObject(1));
        System.out.println("getFloat() " + rs.getFloat(1));
        System.out.println("getDouble() " + rs.getDouble(1));
        System.out.println("getBigDecimal() " + rs.getBigDecimal(1));
      }
    }
  }
}

@lidavidm

Copy link
Copy Markdown
Member

Also for Oracle, we should be testing BINARY_FLOAT and not REAL.

@scruz-denodo

Copy link
Copy Markdown
Author

Hi, I modified the attached class for accessing to a table instead of passing the number as argument with the cast. For removing noise with the prepared statement.

I tested also with Oracle. Both of them (Postgresql and Oracle) return 0.1 values with the different number types and methods.

I paste the output and the modified code:

Postgresql

Table

create table test_float (
    c_decimal 	decimal(4, 2),
    c_real		real,
    c_double	double precision
);

insert into test_float (c_decimal, c_real, c_double) values (0.1, 0.1, 0.1);

Output

c_decimal
....................
	getString() 0.10
	getObject() 0.10
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.10

c_real
....................
	getString() 0.10
	getObject() 0.10
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.10

c_double
....................
	getString() 0.10
	getObject() 0.10
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.10

Oracle

Table

create table test_float (
    c_number 	number(4, 2),
    c_float		float,
    c_binary_float		binary_float,
    c_binary_double	binary_double,
    c_double	double precision,
    c_real	real
);

insert into test_float (c_number, c_float, c_binary_float, c_binary_double, c_double, c_real) values (0.1, 0.1, 0.1, 0.1, 0.1, 0.1);
C_NUMBER
....................
	getString() 0.1
	getObject() 0.1
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.1

C_FLOAT
....................
	getString() 0.1
	getObject() 0.1
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.1

C_BINARY_FLOAT
....................
	getString() 0.1
	getObject() 0.1
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.1

C_BINARY_DOUBLE
....................
	getString() 0.1
	getObject() 0.1
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.1

C_DOUBLE
....................
	getString() 0.1
	getObject() 0.1
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.1

C_REAL
....................
	getString() 0.1
	getObject() 0.1
	getFloat() 0.1
	getDouble() 0.1
	getBigDecimal() 0.1

And the modified Java code:

import java.sql.Connection;
import java.sql.DriverManager;
import java.sql.PreparedStatement;
import java.sql.ResultSet;
import java.util.Properties;

public class PgjdbcFloatReproducer {
    public static void main(String[] args) throws Exception {
        if (args.length != 3) {
            throw new IllegalArgumentException(
                "Usage: PgjdbcFloatReproducer <jdbc-url> <username> <password>");
        }

        Properties properties = new Properties();
        properties.setProperty("user", args[1]);
        properties.setProperty("password", args[2]);

        try (Connection connection = DriverManager.getConnection(args[0], properties);
             PreparedStatement statement = connection.prepareStatement("SELECT * from test_float");
             ResultSet rs = statement.executeQuery()) {
            while (rs.next()) {
                for (int i=1;i<=rs.getMetaData().getColumnCount();i++) {
                    System.out.println(rs.getMetaData().getColumnName(i));
                    System.out.println("....................");
                    System.out.println("\tgetString() " + rs.getString(1));
                    System.out.println("\tgetObject() " + rs.getObject(1));
                    System.out.println("\tgetFloat() " + rs.getFloat(1));
                    System.out.println("\tgetDouble() " + rs.getDouble(1));
                    System.out.println("\tgetBigDecimal() " + rs.getBigDecimal(1) + "\n");
                }
            }
        }
    }
}

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Flight SQL JDBC] Read Float4 values as double and BigDecimal with correct precision

2 participants