Uploaded image for project: 'Hive'
  1. Hive
  2. HIVE-13203

STRUCT type not allowing more than 266 fields. Table get created with more fields but throws error while querying against the table. SHOW CREATE TABLE skips having some fields in STRUCT.

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Trivial
    • Resolution: Unresolved
    • 1.2.0
    • None
    • Database/Schema
    • None

    Description

      I am trying to create a table with multiple STRUCT column types, one of which having more than 300 fields.

      I get the table created with no problem, but when i load i face few errors in loading up the data.when i check SHOW CREATE TABLE TABLENAME, i get fields only till 266th column and rest of them do not appear in the table DDL.(check the issue at StateOrProvinc, search for StateOrProvinc in the second DDL).

      Would like to know, if there is any restriction on the number of fields to be using in STRUCT datatype.

      EXAMPLE:

      BELOW IS THE DDL i run in hive and the second below DLL is the TABLE DDL i get from SHOW CREATE TABLE.

      CREATE TABLE property_feature.property_feature_new(
      `_id` STRUCT<`mls_sysid`:String,`type`:String>,
      all_attrs STRUCT<`1`: String ,`10`: String ,`100`: String ,`105`: String ,`113`: String ,`118`: String ,`126`: String ,`1299`: String ,`1301`: String ,`1305`: String ,`1307`: String ,`1319`: String ,`1325`: String ,`1327`: String ,`1329`: String ,`1339`: String ,`1341`: String ,`1345`: String ,`1351`: String ,`1353`: String ,`1359`: String ,`1363`: String ,`1365`: String ,`1367`: String ,`1373`: String ,`1377`: String ,`1397`: String ,`1401`: String ,`1403`: String ,`1405`: String ,`1409`: String ,`1411`: String ,`1413`: String ,`1417`: String ,`1419`: String ,`1421`: String ,`1423`: String ,`1431`: String ,`1437`: String ,`1439`: String ,`144`: String ,`1441`: String ,`145`: String ,`1451`: String ,`1453`: String ,`146`: String ,`1467`: String ,`1469`: String ,`1483`: String ,`1485`: String ,`1491`: String ,`1493`: String ,`15`: String ,`1505`: String ,`1519`: String ,`1521`: String ,`1523`: String ,`155`: String ,`156`: String ,`158`: String ,`1586`: String ,`1678`: String ,`169`: String ,`17`: String ,`177`: String ,`1783`: String ,`1785`: String ,`1787`: String ,`1789`: String ,`1793`: String ,`1795`: String ,`1797`: String ,`1799`: String ,`1801`: String ,`1803`: String ,`1805`: String ,`1807`: String ,`1809`: String ,`1811`: String ,`1815`: String ,`1818`: String ,`1826`: String ,`1829`: String ,`1874`: String ,`1876`: String ,`1878`: String ,`1880`: String ,`1882`: String ,`207`: String ,`215`: String ,`226`: String ,`2293`: String ,`232`: String ,`239`: String ,`241`: String ,`245`: String ,`2456`: String ,`248`: String ,`2482`: String ,`2490`: String ,`2491`: String ,`2492`: String ,`2493`: String ,`2495`: String ,`25`: String ,`2502`: String ,`2503`: String ,`2517`: String ,`2522`: String ,`255`: String ,`256`: String ,`2573`: String ,`2574`: String ,`260`: String ,`263`: String ,`265`: String ,`276`: String ,`278`: String ,`286`: String ,`293`: String ,`296`: String ,`299`: String ,`3`: String ,`30`: String ,`307`: String ,`312`: String ,`320`: String ,`328`: String ,`340`: String ,`348`: String ,`354`: String ,`363`: String ,`364`: String ,`369`: String ,`375`: String ,`376`: String ,`383`: String ,`401`: String ,`405`: String ,`409`: String ,`412`: String ,`42`: String ,`423`: String ,`444`: String ,`445`: String ,`446`: String ,`447`: String ,`450`: String ,`451`: String ,`452`: String ,`453`: String ,`455`: String ,`458`: String ,`461`: String ,`463`: String ,`464`: String ,`51`: String ,`543472980`: String ,`57`: String ,`75`: String ,`90`: String ,`92`: String ,`AVMYN`: String ,`AccessibilityAmenities`: String ,`AddressPublic`: String ,`Amenities`: String ,`AmountofWaterFront`: String ,`AppliancesIncluded`: String ,`AppraiserInitials`: String ,`AppraiserName`: String ,`Basement`: String ,`BathsFull`: String ,`BathsHalf`: String ,`BathsTotal`: String ,`BedsTotal`: String ,`Block`: String ,`Book`: String ,`BuildingNumber`: String ,`City`: String ,`CloseDate`: String ,`ClosePrice`: String ,
      `CoListAgentDirectWorkPhone`: String ,`CoListAgentEmail`: String ,`CoListAgentFullName`: String ,`CoListAgentMLSID`: String ,`CoListAgent_MUI`: String ,`CoListOfficeMLSID`: String ,`CoListOfficeName`: String ,`CoListOfficePhone`: String ,`CoListOffice_MUI`: String ,`Color`: String ,`CommentsYN`: String ,`Construction`: String ,`Cooling`: String ,`CountyOrParish`: String ,`DOM`: String ,`Deed`: String ,`Directions`: String ,`Driveway`: String ,`Electric`: String ,`Equipment`: String ,`Exterior`: String ,`Fireplace`: String ,`FloorNumber`: String ,`Floors`: String ,`FoundationMaterials`: String ,`FullTaxAmount`: String ,`Gas`: String ,`HeatFuel`: String ,`HeatSystem`: String ,`Internet`: String ,`ListAgentDirectWorkPhone`: String ,`ListAgentEmail`: String ,`ListAgentFullName`: String ,`ListAgentMLSID`: String ,`ListAgent_MUI`: String ,`ListOfficeMLSID`: String ,`ListOfficeName`: String ,`ListOfficePhone`: String ,`ListOffice_MUI`: String ,`ListPrice`: String ,`ListingContractDate`: String ,`Location`: String ,`Lot`: String ,`LotSizeArea`: String ,`MLSNumber`: String ,`Make`: String ,`MapNumber`: String ,`MatrixModifiedDT`: String ,`MobileHomeIncludes`: String ,`Model`: String ,`MonthlyAssociationFee`: String ,`NeighborhoodAssoc`: String ,`NumberOfRooms`: String ,`OneBedroomUnits`: String ,`OriginalListPrice`: String ,`Page`: String ,`Parking`: String ,`PhotoCount`: String ,`PhotoModificationTimestamp`: String ,`PostalCode`: String ,`PropertySubType`: String ,`PropertyType`: String ,`PublicAddress`: String ,`RecreationalWater`: String ,`Remarks`: String ,`Restrictions`: String ,`RoadFrontage`: String ,`Roads`: String ,`Roof`: String ,`SchoolDistrict`: String ,`Seasonal`: String ,`SellingAgentDirectWorkPhone`: String ,`SellingAgentEmail`: String ,`SellingAgentFullName`: String ,`SellingAgentMLSID`: String ,`SellingAgent_MUI`: String ,`SellingOfficeMLSID`: String ,`SellingOfficeName`: String ,`SellingOfficePhone`: String ,`SellingOffice_MUI`: String ,`SerialNumber`: String ,`Site`: String ,`SoldTerms`: String ,`SourceofSquareFootage`: String ,`SqFtFinishedTotal`: String ,`StateOrProvince`: String ,`Status`: String ,`Style`: String ,`SupplementPublicCount`: String ,`Surveyed`: String ,`TaxYear`: String ,`ThreeBedroomUnits`: String ,`Transportation`: String ,`TwoBedroomUnits`: String ,`UnitType`: String ,`UnitsInBuilding`: String ,`VehicleStorage`: String ,`View`: String ,`VirtualTour`: String ,`WasteWaterDisposal`: String ,`Water`: String ,`WaterBodyName`: String ,`WaterBodyType`: String ,`WaterFrontage`: String ,`WaterFrontageOwned`: String ,`WaterFrontageShared`: String ,`WaterHeater`: String ,`YearBuilt`: String ,`Zoning`: String ,`matrix_unique_id`: String ,`removed_created_at`: String ,`sysid`: String>
      basement` string
      ) STORED AS parquet;

      CREATE TABLE `property_feature`(
      `_id` struct<mls_sysid:string,type:string>,
      `all_attrs` struct<1:string,10:string,100:string,105:string,113:string,118:string,126:string,1299:string,
      1301:string,1305:string,1307:string,1319:string,1325:string,1327:string,1329:string,
      1339:string,1341:string,1345:string,1351:string,1353:string,1359:string,1363:string,
      1365:string,1367:string,1373:string,1377:string,1397:string,1401:string,1403:string,
      1405:string,1409:string,1411:string,1413:string,1417:string,1419:string,1421:string,
      1423:string,1431:string,1437:string,1439:string,144:string,1441:string,145:string,1451:string,
      1453:string,146:string,1467:string,1469:string,1483:string,1485:string,1491:string,1493:string,
      15:string,1505:string,1519:string,1521:string,1523:string,155:string,156:string,158:string,
      1586:string,1678:string,169:string,17:string,177:string,1783:string,1785:string,1787:string,
      1789:string,1793:string,1795:string,1797:string,1799:string,1801:string,1803:string,1805:string,
      1807:string,1809:string,1811:string,1815:string,1818:string,1826:string,1829:string,1874:string,
      1876:string,1878:string,1880:string,1882:string,207:string,215:string,226:string,2293:string,
      232:string,239:string,241:string,245:string,2456:string,248:string,2482:string,2490:string,
      2491:string,2492:string,2493:string,2495:string,25:string,2502:string,2503:string,2517:string,
      2522:string,255:string,256:string,2573:string,2574:string,260:string,263:string,265:string,
      276:string,278:string,286:string,293:string,296:string,299:string,3:string,30:string,307:string,
      312:string,320:string,328:string,340:string,348:string,354:string,363:string,364:string,369:string,
      375:string,376:string,383:string,401:string,405:string,409:string,412:string,42:string,423:string,444:string,445:string,446:string,447:string,450:string,451:string,452:string,453:string,455:string,458:string,461:string,463:string,464:string,51:string,543472980:string,57:string,75:string,90:string,92:string,AVMYN:string,AccessibilityAmenities:string,
      AddressPublic:string,Amenities:string,AmountofWaterFront:string,AppliancesIncluded:string,AppraiserInitials:string,AppraiserName:string,Basement:string,BathsFull:string,BathsHalf:string,BathsTotal:string,BedsTotal:string,Block:string,Book:string,BuildingNumber:string,City:string,CloseDate:string,ClosePrice:string,CoListAgentDirectWorkPhone:string,CoListAgentEmail:string,CoListAgentFullName:string,CoListAgentMLSID:string,CoListAgent_MUI:string,CoListOfficeMLSID:string,CoListOfficeName:string,CoListOfficePhone:string,
      CoListOffice_MUI:string,Color:string,CommentsYN:string,Construction:string,Cooling:string,CountyOrParish:string,DOM:string,Deed:string,Directions:string,Driveway:string,Electric:string,Equipment:string,Exterior:string,Fireplace:string,FloorNumber:string,Floors:string,FoundationMaterials:string,FullTaxAmount:string,Gas:string,HeatFuel:string,HeatSystem:string,Internet:string,ListAgentDirectWorkPhone:string,ListAgentEmail:string,ListAgentFullName:string,ListAgentMLSID:string,ListAgent_MUI:string,ListOfficeMLSID:string,ListOfficeName:string,ListOfficePhone:string,ListOffice_MUI:string,ListPrice:string,ListingContractDate:string,Location:string,Lot:string,LotSizeArea:string,MLSNumber:string,Make:string,MapNumber:string,MatrixModifiedDT:string,MobileHomeIncludes:string,Model:string,MonthlyAssociationFee:string,NeighborhoodAssoc:string,NumberOfRooms:string,OneBedroomUnits:string,OriginalListPrice:string,Page:string,Parking:string,PhotoCount:string,PhotoModificationTimestamp:string,PostalCode:string,PropertySubType:string,PropertyType:string,PublicAddress:string,RecreationalWater:string,Remarks:string,Restrictions:string,RoadFrontage:string,Roads:string,Roof:string,SchoolDistrict:string,Seasonal:string,SellingAgentDirectWorkPhone:string,SellingAgentEmail:string,SellingAgentFullName:string,SellingAgentMLSID:string,SellingAgent_MUI:string,SellingOfficeMLSID:string,SellingOfficeName:string,SellingOfficePhone:string,SellingOffice_MUI:string,SerialNumber:string,Site:string,SoldTerms:string,
      SourceofSquareFootage:string,SqFtFinishedTotal:string,StateOrProvinc,
      `basement` string
      ) STORED AS parquet;

      Please help out in case if there is any workaround for it or any other restriction details.

      Attachments

        Activity

          People

            Satyajit Satyajit varma
            Satyajit Satyajit varma
            Votes:
            0 Vote for this issue
            Watchers:
            0 Start watching this issue

            Dates

              Created:
              Updated: