Uploaded image for project: 'Spark'
  1. Spark
  2. SPARK-22273

Fix key/value schema field names in HashMapGenerators.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Minor
    • Resolution: Fixed
    • 2.1.1, 2.2.0
    • 2.1.3, 2.2.1, 2.3.0
    • SQL
    • None

    Description

      When fixing schema field names using escape characters with addReferenceMinorObj() at SPARK-18952, double-quotes around the names were remained and the names become something like "((java.lang.String) references[1])".

      /* 055 */     private int maxSteps = 2;
      /* 056 */     private int numRows = 0;
      /* 057 */     private org.apache.spark.sql.types.StructType keySchema = new org.apache.spark.sql.types.StructType().add("((java.lang.String) references[1])", org.apache.spark.sql.types.DataTypes.StringType);
      /* 058 */     private org.apache.spark.sql.types.StructType valueSchema = new org.apache.spark.sql.types.StructType().add("((java.lang.String) references[2])", org.apache.spark.sql.types.DataTypes.LongType);
      /* 059 */     private Object emptyVBase;
      

      We should remove the double-quotes to refer the values in references properly:

      /* 055 */     private int maxSteps = 2;
      /* 056 */     private int numRows = 0;
      /* 057 */     private org.apache.spark.sql.types.StructType keySchema = new org.apache.spark.sql.types.StructType().add(((java.lang.String) references[1]), org.apache.spark.sql.types.DataTypes.StringType);
      /* 058 */     private org.apache.spark.sql.types.StructType valueSchema = new org.apache.spark.sql.types.StructType().add(((java.lang.String) references[2]), org.apache.spark.sql.types.DataTypes.LongType);
      /* 059 */     private Object emptyVBase;
      

      Attachments

        Activity

          People

            ueshin Takuya Ueshin
            ueshin Takuya Ueshin
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: