Uploaded image for project: 'Kylin'
  1. Kylin
  2. KYLIN-3335

Add project & cube related info to the job id for better filtering

    XMLWordPrintableJSON

Details

    • Improvement
    • Status: Open
    • Major
    • Resolution: Unresolved
    • None
    • Backlog
    • None
    • None

    Description

      Currently it's painful to search cube or project related jobs, since those infos are hidden in values. Especially, when users want to list job in a period under one project, by current design, all of the job output info have to be read into memory. If this kind of operation is done very often within a short period, it's easy to get OOM.

      If the job id is prefixed with project and cube names, then we can push down prefix filters, which is efficient and safe.

      This kind of change will cause backward compatibility issue. There're two ways to deal with this:

      • Set a milestone with time tag, if a search relates to data earlier than this time, then just need to do one scan with prefix filter. Otherwise, two scans are needed. One with prefix filter and the other use current strategy. As time goes on, old job infos will be deleted. Once there's no data older than the time, only one scan is needed.
      • Do migration for the old data once.

      Attachments

        Activity

          People

            Unassigned Unassigned
            yaho Zhong Yanghong
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated: