Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-4866

Streaming from Kafka with nested timestamp column

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • v3.1.0
    • None
    • Real-time Streaming
    • None
    • Docker setup

    Description

      We have an IOT Data Platform which saves in Kafka Json messages with the following format: 

      {
       "SysProperties": {
       "EventId": "1610120564321723276",
       "EnqueueTime": "2021-01-08T15:42:44.045Z",
       "UserName": "dummyuser"
       },
       "AppProperties": {
       "EventDefinition": "EquipmentData",
       "EventTime": "2021-01-08T15:42:43.26+00:00",
       "ApplicationName": "ConnectIoTSimulator",
       "ApplicationContext": null
       },
       "Data": {
       "Equipment": "DUMMY-01",
       "Material": "DUMMYZA9983884",
       "MeasurementData": {
       "Temp": 57,
       "Pressure": 48.379,
       "Energy": 47,
       "Humidity": 31.727
       }
      }
      
      

      From these messages, we are trying to build a cube in which the "AppProperties.EventTime" shall be the timestamp column (using format yyyy-MM-dd'T'HH:mm:ss.SSSZZ), "Data.MeasurementData.Temp", "Data.MeasurementData.Pressure", "Data.MeasurementData.Energy" and "Data.MeasurementData.Humidity" shall be the metrics and all other fields dimensions.

      We can setup everything fine in the kylin portal (like in the tutorial), however on runtime we have the following exceptions logged by the streaming receiver component:

      After some investigation, we figured out it only works if the timestamp field within the Json messages is not nested inside inner objects, like in the following example:

      {
       "timestamp": 1610038228492,
       "nestedData": {
       "Equipment": "EPFTES-01",
       "Material": "ZA9983884",
       "Temp": 10,
       "Pressure": 20.459,
       "Energy": 30,
       "Humidity": 80.967
       }
      }

      Therefore, we believe there is a bug handling the timestamp column when it is nested within Json messages.  

      Attachments

        1. image-2021-01-08-16-16-51-615.png
          43 kB
          Hélder Hugo Ferreira

        Activity

          People

            Unassigned Unassigned
            hhferreira Hélder Hugo Ferreira
            Votes:
            1 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated: