Uploaded image for project: 'Atlas'
  1. Atlas
  2. ATLAS-346

Atlas server loses messages sent from Hive hook if restarted after unclean shutdown

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Critical
    • Resolution: Fixed
    • None
    • 0.6-incubating
    • None
    • None

    Description

      • Start Atlas server pointed to an external Kafka instance
      • Configure HiveServer2 with Atlas hook pointing to the same Kafka instance.
      • Run a hive DDL script like
        for i in `seq 1 25`; do ./bin/beeline -u jdbc:hive2://localhost:10000 -n ${user} -p ${pass} -e "create table tbl${i} (col${i}1 int, col${i}2 string);"; done
        
      • While the script is executing, kill -9 the Atlas server
      • Let the script complete.
      • Verify that all the events are added to the ATLAS_HOOK topic.
      • Verify if the tables are added to Atlas.

      The observation is that the Kafka topic has all relevant messages, but the tables aren't added to Atlas.

      Attachments

        1. ATLAS-346.patch
          15 kB
          Hemanth Yamijala
        2. ATLAS-346.patch
          15 kB
          Hemanth Yamijala
        3. ATLAS-346.patch
          14 kB
          Hemanth Yamijala
        4. hive_server2_import_bug_snip.txt
          24 kB
          Hemanth Yamijala

        Activity

          People

            yhemanth Hemanth Yamijala
            yhemanth Hemanth Yamijala
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: