Uploaded image for project: 'Ambari'
  1. Ambari
  2. AMBARI-25880

For hadoop_logs collection of Infra-solr, indexed log_message field is too large to copy to key_log_message field

    XMLWordPrintableJSON

Details

    Description

      The hadoop_logs collection of Infra-solr is used to store logs generated by services, such as hadoop、spark, etc. 
      However, Infra-solr will report an error when the length of the single-line log generated by services exceeds 2^15-2, Because Infra-solr cannot copy the data of field log_message to field key_log_messag

      //代码占位符
      org.apache.solr.common.SolrException: Exception writing document id 54d786cd-f6a7-4746-9b17-e9b2b0fe389a to the index; possible analysis error: Document contains at least one immense term in field="action" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98, 98]...', original message: bytes can be at most 32766 in length; got 32767. Perhaps the document has an indexed string field (solr.StrField) which is too large 

      Reproduce Step:

      1 Create a json File named data.json
      [
         

      {         "log_message": "********",         "id": "54d786cd-f6a7-4746-9b17-e9b2b0fe3efa"     }

      ]
      Note: the content of field log_message is a randomly generated string with a length exceeding 2^15-2

      2 Update the hadoop_logs
      curl http://ip:port/solr/hadoop_logs/update?commit=true -H 'Content-type:application/json' --data-binary @data.json

      3 Get the error message mentioned above

       

      similar issue:

      AMBARI-24067: Log Search: indexed string field could be too large for audit_logs

       

      Attachments

        Issue Links

          Activity

            People

              guluo guluo
              guluo guluo
              Votes:
              0 Vote for this issue
              Watchers:
              1 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: