Uploaded image for project: 'Apache Rat'
  1. Apache Rat
  2. RAT-323 Harmonize UIs
  3. RAT-240

CLI: Rat does not support a .ratignore file that lives in the source tree.

Details

    • Sub-task
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 0.12
    • None
    • Client - cli, scan
    • None

    Description

      it is not possible to exclude file (or folder) by providing full path

      eg
      foo/foo2/main.c
      bar/bar2/main.c

      it is not possible to match only one of those file from project top directory .rat-excludes file.

       

      The initial proble

      Attachments

        1. rat.txt
          648 kB
          Szymon Janc

        Activity

          janc not sure what you mean or what kind of exclusions you want to configure -
          is https://github.com/ottlinger/creadur-rat-examples/tree/master/RAT-240 what you try to achieve?

          If you want to exclude a full folder, you may configure it like that

            <exclude>foo/</exclude>
            <exclude>foo/**</exclude>
          

          Do you have a link to the project that shows the behaviour you reported?

          Thanks

          pottlinger Philipp Ottlinger added a comment - janc not sure what you mean or what kind of exclusions you want to configure - is https://github.com/ottlinger/creadur-rat-examples/tree/master/RAT-240 what you try to achieve? If you want to exclude a full folder, you may configure it like that <exclude>foo/</exclude> <exclude>foo/**</exclude> Do you have a link to the project that shows the behaviour you reported? Thanks

          ping - any updates on the issue?
          Since I cannot reproduce it I will close the issue as won't fix.

          Looking forward to your feedback.
          Thanks

          pottlinger Philipp Ottlinger added a comment - ping - any updates on the issue? Since I cannot reproduce it I will close the issue as won't fix. Looking forward to your feedback. Thanks
          janc Szymon Janc added a comment -

          Hi,

          We use RAT in Apache Mynewt project for license checks.
          here is the link for our exclude https://github.com/apache/mynewt-core/blob/master/.rat-excludes

          our project code structure if that includes are always in include/ folders, and sources are in src/
          this is 'embedded' project so we have a lot of imported code from SDK/BSP for board support,
          some libs imported for crypt, protocols etc

          result in structure like this:
          eg
          hw/bsp/boardA/src/bsp.c
          hw/bsp/boardA/include/bsp.h
          hw/bsp/boardB/src/bsp.c
          hw/bsp/boardB/include/bsp.h
          net/lora/src/foo.c
          net/lora/include/foo.h

          both can be on different licenses which we would like to exclude from RAT check (with proper comment in exclude file)
          unfortunately currently only single depth path are accepted by RAT (eg src/bsp.c) but not full path.
          If I provide hw/bsp/boardA/src/bsp.c it is not excluded by RAT.

          So, for us, the best solution would be option to exclude by providing full path (either folder or explicitly files would work) eg
          hw/bsp/boardA/*
          hw/bsp/boardA/src/foo.c

          If this is no possible (due to the way RAT works) then maybe RAT could have an option for implicitly using .rat-exclude file from folder where checked files is located?
          That would also work for us without risk of ambiguous names matching.

          Hope this clarifies our needs

          janc Szymon Janc added a comment - Hi, We use RAT in Apache Mynewt project for license checks. here is the link for our exclude https://github.com/apache/mynewt-core/blob/master/.rat-excludes our project code structure if that includes are always in include/ folders, and sources are in src/ this is 'embedded' project so we have a lot of imported code from SDK/BSP for board support, some libs imported for crypt, protocols etc result in structure like this: eg hw/bsp/boardA/src/bsp.c hw/bsp/boardA/include/bsp.h hw/bsp/boardB/src/bsp.c hw/bsp/boardB/include/bsp.h net/lora/src/foo.c net/lora/include/foo.h both can be on different licenses which we would like to exclude from RAT check (with proper comment in exclude file) unfortunately currently only single depth path are accepted by RAT (eg src/bsp.c) but not full path. If I provide hw/bsp/boardA/src/bsp.c it is not excluded by RAT. So, for us, the best solution would be option to exclude by providing full path (either folder or explicitly files would work) eg hw/bsp/boardA/* hw/bsp/boardA/src/foo.c If this is no possible (due to the way RAT works) then maybe RAT could have an option for implicitly using .rat-exclude file from folder where checked files is located? That would also work for us without risk of ambiguous names matching. Hope this clarifies our needs

          I'm beginning to understand your problem -
          in case you are using the Maven integration you are already able to exclude a certain tree/directory - please see my commit
          https://github.com/ottlinger/creadur-rat-examples/commit/653545b58270fc714e0e667e3049720d960e6131
          on how to configure this.

          At the moment there's no functionality to provide a global ignore file.

          In case you want to request that feel free to file an improvement since bugs relate to errors in the current implementation

          Thanks

          pottlinger Philipp Ottlinger added a comment - I'm beginning to understand your problem - in case you are using the Maven integration you are already able to exclude a certain tree/directory - please see my commit https://github.com/ottlinger/creadur-rat-examples/commit/653545b58270fc714e0e667e3049720d960e6131 on how to configure this. At the moment there's no functionality to provide a global ignore file. In case you want to request that feel free to file an improvement since bugs relate to errors in the current implementation Thanks

          Having a look at
          https://github.com/apache/mynewt-core/blob/master/.rat-excludes
          I wonder, how do you execute RAT on your project?

          pottlinger Philipp Ottlinger added a comment - Having a look at https://github.com/apache/mynewt-core/blob/master/.rat-excludes I wonder, how do you execute RAT on your project?

          Did you know that RAT uses SCM-ignores as default exclusions - maybe this can be helpful for you in case you generated stuff during build/package time?

          have a look at:
          https://github.com/apache/creadur-rat/blob/b1f2f1d470a6cfee656a06ab10ea1210a2e51b32/apache-rat-core/src/main/java/org/apache/rat/config/SourceCodeManagementSystems.java

          Apart from that I just stumbled upon the configuration parameter:
          @Parameter(property="rat.excludesFile")
          in
          https://github.com/apache/creadur-rat/blob/b1f2f1d470a6cfee656a06ab10ea1210a2e51b32/apache-rat-plugin/src/main/java/org/apache/rat/mp/AbstractRatMojo.java

          are you using this feature and it does not behave as expected?

          pottlinger Philipp Ottlinger added a comment - Did you know that RAT uses SCM-ignores as default exclusions - maybe this can be helpful for you in case you generated stuff during build/package time? have a look at: https://github.com/apache/creadur-rat/blob/b1f2f1d470a6cfee656a06ab10ea1210a2e51b32/apache-rat-core/src/main/java/org/apache/rat/config/SourceCodeManagementSystems.java Apart from that I just stumbled upon the configuration parameter: @Parameter(property="rat.excludesFile") in https://github.com/apache/creadur-rat/blob/b1f2f1d470a6cfee656a06ab10ea1210a2e51b32/apache-rat-plugin/src/main/java/org/apache/rat/mp/AbstractRatMojo.java are you using this feature and it does not behave as expected?
          janc Szymon Janc added a comment -

          oh, good to know that RAT is making use of gitignore, will cleanup our excludes a bit

          as for using RAT, I simply run "java -jar ~/Downloads/apache-rat-0.12/apache-rat-0.12.jar -E .rat-excludes -d ." from project top folder.

          janc Szymon Janc added a comment - oh, good to know that RAT is making use of gitignore, will cleanup our excludes a bit as for using RAT, I simply run "java -jar ~/Downloads/apache-rat-0.12/apache-rat-0.12.jar -E .rat-excludes -d ." from project top folder.

          hmm - what's the expected behaviour. I can run RAT successfully and get some files without licenses:

          $ java -jar ~/Downloads/apache-rat-0.12.jar -E .rat-excludes -d .
          
          *****************************************************
          Summary
          -------
          Generated at: 2017-10-05T23:25:03+02:00
          
          Notes: 4
          Binaries: 0
          Archives: 0
          Standards: 1786
          
          Apache Licensed: 1765
          Generated Documents: 0
          
          JavaDocs are generated, thus a license header is optional.
          Generated files do not require license headers.
          
          6 Unknown Licenses
          
          *****************************************************
          
          Files with unapproved licenses:
          
            ./hw/bsp/pic32mx470_6lp_clicker/src/sbrk.c
            ./hw/bsp/pic32mz2048_wi-fire/src/sbrk.c
            ./kernel/os/include/os/arch/pic32/os/cp0defs.h
            ./kernel/os/src/arch/pic32/startup/cache-err-exception.S
            ./kernel/os/src/arch/pic32/startup/crt0.S
            ./kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S
          ...
          

          So your expectation would be to exclude /hw/bsp or should the kernel also be excluded?

          It's possible that the CLI does not have the same functionality as the maven version, but at first I'd like to find out what you want to have.

          Thanks

          pottlinger Philipp Ottlinger added a comment - hmm - what's the expected behaviour. I can run RAT successfully and get some files without licenses: $ java -jar ~/Downloads/apache-rat-0.12.jar -E .rat-excludes -d . ***************************************************** Summary ------- Generated at: 2017-10-05T23:25:03+02:00 Notes: 4 Binaries: 0 Archives: 0 Standards: 1786 Apache Licensed: 1765 Generated Documents: 0 JavaDocs are generated, thus a license header is optional. Generated files do not require license headers. 6 Unknown Licenses ***************************************************** Files with unapproved licenses: ./hw/bsp/pic32mx470_6lp_clicker/src/sbrk.c ./hw/bsp/pic32mz2048_wi-fire/src/sbrk.c ./kernel/os/include/os/arch/pic32/os/cp0defs.h ./kernel/os/src/arch/pic32/startup/cache-err-exception.S ./kernel/os/src/arch/pic32/startup/crt0.S ./kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S ... So your expectation would be to exclude /hw/bsp or should the kernel also be excluded? It's possible that the CLI does not have the same functionality as the maven version, but at first I'd like to find out what you want to have. Thanks
          janc Szymon Janc added a comment -

          if you look at .rat-exclude you'll see that we are force to use very generic names for exclusion like lora, mesh, rtt etc and those can simply exclude too much

          so what this report is about is that we would like to be able to have our rat-exclude be like:
          ./kernel/os/src/arch/pic32/
          ./hw/bsp/pic32mz2048_wi-fire/
          ./net/lora/node/
          ./hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h

          instead of currently used
          pic32
          lora
          compiler_abstraction.h

          as those can simply match too much.

          janc Szymon Janc added a comment - if you look at .rat-exclude you'll see that we are force to use very generic names for exclusion like lora, mesh, rtt etc and those can simply exclude too much so what this report is about is that we would like to be able to have our rat-exclude be like: ./kernel/os/src/arch/pic32/ ./hw/bsp/pic32mz2048_wi-fire/ ./net/lora/node/ ./hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h instead of currently used pic32 lora compiler_abstraction.h as those can simply match too much.

          Commit 1811592 from hugo.hirsch in branch 'rat/trunk'
          [ https://svn.apache.org/r1811592 ]

          RAT-240: Add comments in exclusion files/change parsing

          • Added skippig of empty and comment lines.
          • Added new NameFileFilter and WildcardFileFilter to support more
            exclusions in different formats.
          jira-bot ASF subversion and git services added a comment - Commit 1811592 from hugo.hirsch in branch 'rat/trunk' [ https://svn.apache.org/r1811592 ] RAT-240 : Add comments in exclusion files/change parsing Added skippig of empty and comment lines. Added new NameFileFilter and WildcardFileFilter to support more exclusions in different formats.

          janc can you verify with the current 0.13-SNAPSHOT that your stuff is properly excluded ?!

          I checked out your project and added the following example lines to your .rat-exclude-file:

          +
          +# RAT-240
          +#.*hw/bsp*
          +/hw/bsp/pic32mz2048_wi-fire/src/sbrk.c
          +/kernel/os/src/arch/pic32/*
          +/hw/bsp/pic32mz2048_wi-fire/*
          +/net/lora/node/
          +/hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h
          +pic32mx470_6lp_clicker*
          +run.sh
          +# from the ticket itself
          +/kernel/os/src/arch/pic32/
          +/hw/bsp/pic32mz2048_wi-fire/
          +/net/lora/node/
          +/hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h
          +# New files?
          +hw/bsp/pic32mz2048_wi-fire/src/sbrk.c
          +kernel/os/include/os/arch/pic32/os/cp0defs.h
          +kernel/os/src/arch/pic32/startup/cache-err-exception.S
          +kernel/os/src/arch/pic32/startup/crt0.S
          +kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S
          +#**.***
          

          My local version runs with:

          $ cat run.sh 
          #!/bin/bash
          java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more
          

          Is that what you need?

          pottlinger Philipp Ottlinger added a comment - janc can you verify with the current 0.13-SNAPSHOT that your stuff is properly excluded ?! I checked out your project and added the following example lines to your .rat-exclude-file: + +# RAT-240 +#.*hw/bsp* +/hw/bsp/pic32mz2048_wi-fire/src/sbrk.c +/kernel/os/src/arch/pic32/* +/hw/bsp/pic32mz2048_wi-fire/* +/net/lora/node/ +/hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h +pic32mx470_6lp_clicker* +run.sh +# from the ticket itself +/kernel/os/src/arch/pic32/ +/hw/bsp/pic32mz2048_wi-fire/ +/net/lora/node/ +/hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h +# New files? +hw/bsp/pic32mz2048_wi-fire/src/sbrk.c +kernel/os/include/os/arch/pic32/os/cp0defs.h +kernel/os/src/arch/pic32/startup/cache-err-exception.S +kernel/os/src/arch/pic32/startup/crt0.S +kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S +#**.*** My local version runs with: $ cat run.sh #!/bin/bash java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more Is that what you need?

          Commit 1811618 from hugo.hirsch in branch 'rat/trunk'
          [ https://svn.apache.org/r1811618 ]

          RAT-240: Add change documentation and update tests

          • Test comment/empty line input.
          • Test input that is trimmed.
          • Add changes.
          jira-bot ASF subversion and git services added a comment - Commit 1811618 from hugo.hirsch in branch 'rat/trunk' [ https://svn.apache.org/r1811618 ] RAT-240 : Add change documentation and update tests Test comment/empty line input. Test input that is trimmed. Add changes.
          janc Szymon Janc added a comment -

          Hi,

          which version should I use? I tried apache-rat-0.13-20171009.224400-55.jar and couldn't make it work as you described

          /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx
          /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/*
          /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h

          none of those work for excluding, I get this:

          Files with unapproved licenses:
          ...
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Release_Notes.html
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xe.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f405xx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f407xx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410cx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410rx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410tx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f411xe.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412cx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412rx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412vx.h
          ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412zx.h
          .....

          janc Szymon Janc added a comment - Hi, which version should I use? I tried apache-rat-0.13-20171009.224400-55.jar and couldn't make it work as you described /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/* /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h none of those work for excluding, I get this: Files with unapproved licenses: ... ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Release_Notes.html ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xe.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f405xx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f407xx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410cx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410rx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410tx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f411xe.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412cx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412rx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412vx.h ./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412zx.h .....

          The version should be correct (latest SNAPSHOT) - are you on windows?

          I've checked out your project and added

          /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx
          /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/*
          /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h
          

          to the exclusion file and it works fine

          $ java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more
          

          Can you check out the version and try to build it for yourself just to make sure you are using the latest SNAPHOT?
          https://svn.apache.org/repos/asf/creadur/rat/trunk

          HTH

          pottlinger Philipp Ottlinger added a comment - The version should be correct (latest SNAPSHOT) - are you on windows? I've checked out your project and added /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/* /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h to the exclusion file and it works fine $ java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more Can you check out the version and try to build it for yourself just to make sure you are using the latest SNAPHOT? https://svn.apache.org/repos/asf/creadur/rat/trunk HTH
          janc Szymon Janc added a comment -

          I'm running this on Linux (Fedora 26).

          I'm bit puzzled since I'm not able to make it work as you described. I'm attaching log (rat.txt) showing change I made in .rat-excludes and output of RAT execution.
          As you can see /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx is not excluded.

          Any advice?

          janc Szymon Janc added a comment - I'm running this on Linux (Fedora 26). I'm bit puzzled since I'm not able to make it work as you described. I'm attaching log (rat.txt) showing change I made in .rat-excludes and output of RAT execution. As you can see /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx is not excluded. Any advice?
          janc Szymon Janc added a comment -

          RAT execution log

          janc Szymon Janc added a comment - RAT execution log

          Are you working on a specific branch?

          I've checked out mynewt-master and added:

          $ git diff
          diff --git a/.rat-excludes b/.rat-excludes
          index bc3eb00..bc08173 100644
          --- a/.rat-excludes
          +++ b/.rat-excludes
          @@ -344,3 +344,6 @@ split-calliope_mini.ld
           
           # Bluetooth Mesh - Apache 2.0 License
           mesh
          +
          +docs*
          +run.sh
          
          

          to .rat-excludes.

          Running RAT yields expectedly missing files:

          ✔ ~/workspace/mynewt-core [master|✚ 1…1] 
          23:30 $ ./run.sh 
          Ignored 95 lines in your exclusion files as comments or empty lines.
          
          *****************************************************
          Summary
          -------
          Generated at: 2017-11-06T23:30:04+01:00
          
          Notes: 4
          Binaries: 0
          Archives: 0
          Standards: 1841
          
          Apache Licensed: 1795
          Generated Documents: 0
          
          JavaDocs are generated, thus a license header is optional.
          Generated files do not require license headers.
          
          30 Unknown Licenses
          
          *****************************************************
          
          Files with unapproved licenses:
          
            ./hw/bsp/ci40/src/arch/mips/abiflags.S
            ./hw/bsp/ci40/src/arch/mips/gcc_startup_mips.S
            ./hw/bsp/pic32mx470_6lp_clicker/src/sbrk.c
            ./hw/bsp/pic32mz2048_wi-fire/src/sbrk.c
            ./hw/mips-hal/src/arch/mips/__exit.c
            ./hw/mips-hal/src/arch/mips/cache.S
            ./hw/mips-hal/src/arch/mips/cache.h
            ./hw/mips-hal/src/arch/mips/cache_ops.S
            ./hw/mips-hal/src/arch/mips/link.c
            ./hw/mips-hal/src/arch/mips/m32cache.S
            ./hw/mips-hal/src/arch/mips/m32cache.h
            ./hw/mips-hal/src/arch/mips/m32cache_ops.S
            ./hw/mips-hal/src/arch/mips/m32tlb_ops.S
            ./hw/mips-hal/src/arch/mips/m64tlb_ops.S
            ./hw/mips-hal/src/arch/mips/mips_cm3_l2size.S
            ./hw/mips-hal/src/arch/mips/mips_excpt_boot.S
            ./hw/mips-hal/src/arch/mips/mips_excpt_entry.S
            ./hw/mips-hal/src/arch/mips/mips_excpt_handler.c
            ./hw/mips-hal/src/arch/mips/mips_excpt_isr.S
            ./hw/mips-hal/src/arch/mips/mips_excpt_register.S
            ./hw/mips-hal/src/arch/mips/mips_fp.S
            ./hw/mips-hal/src/arch/mips/mips_l2size.S
            ./hw/mips-hal/src/arch/mips/mips_msa.S
            ./hw/mips-hal/src/arch/mips/mips_xpa.S
            ./hw/mips-hal/src/arch/mips/mxxtlb_ops.S
            ./kernel/os/include/os/arch/pic32/os/cp0defs.h
            ./kernel/os/src/arch/pic32/startup/cache-err-exception.S
            ./kernel/os/src/arch/pic32/startup/crt0.S
            ./kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S
            ./net/nimble/host/test/src/ble_hs_test_util_hci.c
          
          *****************************************************
          
          *****************************************************
            Files with Apache License headers will be marked AL
          ....
          

          but not the situation you are describing.

          run.sh looks like that to ease debugging:

          #!/bin/bash
          java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more
          

          Which Java version are you using?
          I'm on
          $ java --version
          java 9.0.1
          Java(TM) SE Runtime Environment (build 9.0.1+11)
          Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)
          same result with Oracle JDK1.8_152.

          Not sure how I can help you here anymore.

          pottlinger Philipp Ottlinger added a comment - Are you working on a specific branch? I've checked out mynewt-master and added: $ git diff diff --git a/.rat-excludes b/.rat-excludes index bc3eb00..bc08173 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -344,3 +344,6 @@ split-calliope_mini.ld # Bluetooth Mesh - Apache 2.0 License mesh + +docs* +run.sh to .rat-excludes. Running RAT yields expectedly missing files: ✔ ~/workspace/mynewt-core [master|✚ 1…1] 23:30 $ ./run.sh Ignored 95 lines in your exclusion files as comments or empty lines. ***************************************************** Summary ------- Generated at: 2017-11-06T23:30:04+01:00 Notes: 4 Binaries: 0 Archives: 0 Standards: 1841 Apache Licensed: 1795 Generated Documents: 0 JavaDocs are generated, thus a license header is optional. Generated files do not require license headers. 30 Unknown Licenses ***************************************************** Files with unapproved licenses: ./hw/bsp/ci40/src/arch/mips/abiflags.S ./hw/bsp/ci40/src/arch/mips/gcc_startup_mips.S ./hw/bsp/pic32mx470_6lp_clicker/src/sbrk.c ./hw/bsp/pic32mz2048_wi-fire/src/sbrk.c ./hw/mips-hal/src/arch/mips/__exit.c ./hw/mips-hal/src/arch/mips/cache.S ./hw/mips-hal/src/arch/mips/cache.h ./hw/mips-hal/src/arch/mips/cache_ops.S ./hw/mips-hal/src/arch/mips/link.c ./hw/mips-hal/src/arch/mips/m32cache.S ./hw/mips-hal/src/arch/mips/m32cache.h ./hw/mips-hal/src/arch/mips/m32cache_ops.S ./hw/mips-hal/src/arch/mips/m32tlb_ops.S ./hw/mips-hal/src/arch/mips/m64tlb_ops.S ./hw/mips-hal/src/arch/mips/mips_cm3_l2size.S ./hw/mips-hal/src/arch/mips/mips_excpt_boot.S ./hw/mips-hal/src/arch/mips/mips_excpt_entry.S ./hw/mips-hal/src/arch/mips/mips_excpt_handler.c ./hw/mips-hal/src/arch/mips/mips_excpt_isr.S ./hw/mips-hal/src/arch/mips/mips_excpt_register.S ./hw/mips-hal/src/arch/mips/mips_fp.S ./hw/mips-hal/src/arch/mips/mips_l2size.S ./hw/mips-hal/src/arch/mips/mips_msa.S ./hw/mips-hal/src/arch/mips/mips_xpa.S ./hw/mips-hal/src/arch/mips/mxxtlb_ops.S ./kernel/os/include/os/arch/pic32/os/cp0defs.h ./kernel/os/src/arch/pic32/startup/cache-err-exception.S ./kernel/os/src/arch/pic32/startup/crt0.S ./kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S ./net/nimble/host/test/src/ble_hs_test_util_hci.c ***************************************************** ***************************************************** Files with Apache License headers will be marked AL .... but not the situation you are describing. run.sh looks like that to ease debugging: #!/bin/bash java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more Which Java version are you using? I'm on $ java --version java 9.0.1 Java(TM) SE Runtime Environment (build 9.0.1+11) Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode) same result with Oracle JDK1.8_152. Not sure how I can help you here anymore.
          janc Szymon Janc added a comment -

          I'm using one from Fedora

          [janc@ix ~]$ java -version
          openjdk version "1.8.0_151"
          OpenJDK Runtime Environment (build 1.8.0_151-b12)
          OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)

          note that my java doesn't accept --version, only -version
          is OpenJDK supported?

          janc Szymon Janc added a comment - I'm using one from Fedora [janc@ix ~] $ java -version openjdk version "1.8.0_151" OpenJDK Runtime Environment (build 1.8.0_151-b12) OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode) note that my java doesn't accept --version, only -version is OpenJDK supported?
          utzig Fabio Utzig added a comment -

          I tested with both openjdk 8 and 9 and both didn't work. Btw, I was only able to build with openjdk8 (9 fails). I will also give sun's jdk/jre 8 a try...

          utzig Fabio Utzig added a comment - I tested with both openjdk 8 and 9 and both didn't work. Btw, I was only able to build with openjdk8 (9 fails). I will also give sun's jdk/jre 8 a try...
          utzig Fabio Utzig added a comment -

          Also tested with:

          $ java -version
          java version "1.8.0_152"
          Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
          Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)

          And it also didn't work. I took a quick look at the source code and it seems that DirectoryWalker is only applying exclusion patterns to the leaves in the path but I might be getting it completely wrong since I never spent much time programming in java.

          Would you mind providing your own build (apache-rat-0.13-SNAPSHOT.jar) for me to test here?

          utzig Fabio Utzig added a comment - Also tested with: $ java -version java version "1.8.0_152" Java(TM) SE Runtime Environment (build 1.8.0_152-b16) Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode) And it also didn't work. I took a quick look at the source code and it seems that DirectoryWalker is only applying exclusion patterns to the leaves in the path but I might be getting it completely wrong since I never spent much time programming in java. Would you mind providing your own build (apache-rat-0.13-SNAPSHOT.jar) for me to test here?

          Hi guys, utzigjanc
          I'm afraid I'm still unable to understand your problems.

          I've clone your repository
          ✔ ~/workspace/mynewt-core [master|…1]
          and added a run.sh which yields the following output:

          $ ./run.sh
          Ignored 111 lines in your exclusion files as comments or empty lines.

          *****************************************************
          Summary
          -------
          Generated at: 2017-11-23T23:40:35+01:00

          Notes: 4
          Binaries: 0
          Archives: 0
          Standards: 1830

          Apache Licensed: 1813
          Generated Documents: 0

          JavaDocs are generated, thus a license header is optional.
          Generated files do not require license headers.

          1 Unknown Licenses

          *****************************************************

          Files with unapproved licenses:

          ./run.sh

          *****************************************************

          .....

          My run.sh is:

          $ cat run.sh
          #!/bin/bash
          /home/me/jdk1.8.0_152/bin/java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more

          I've checked you rat and build it a while ago, thus the reference.

          As the build goes through successfully I don't know what I miss here.

          pottlinger Philipp Ottlinger added a comment - Hi guys, utzig janc I'm afraid I'm still unable to understand your problems. I've clone your repository ✔ ~/workspace/mynewt-core [master|…1] and added a run.sh which yields the following output: $ ./run.sh Ignored 111 lines in your exclusion files as comments or empty lines. ***************************************************** Summary ------- Generated at: 2017-11-23T23:40:35+01:00 Notes: 4 Binaries: 0 Archives: 0 Standards: 1830 Apache Licensed: 1813 Generated Documents: 0 JavaDocs are generated, thus a license header is optional. Generated files do not require license headers. 1 Unknown Licenses ***************************************************** Files with unapproved licenses: ./run.sh ***************************************************** ..... My run.sh is: $ cat run.sh #!/bin/bash /home/me/jdk1.8.0_152/bin/java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more I've checked you rat and build it a while ago, thus the reference. As the build goes through successfully I don't know what I miss here.
          janc Szymon Janc added a comment -

          ok, new release is coming so I'm back to using RAT.

          Lets say I have those new files:

          hw/bsp/arc/file1
          hw/bsp/arc/file2
          hw/bsp/arc/file3
          bsp/arc/file1
          bsp/arc/file2
          bsp/arc/file3

          How do I exclude files in bsp/arc/ without excluding files in hw/bsp/arc?
          I'd expect somtehing like this (or similar) to work but none of combination I tried works.
          ./bsp/arc/*
          ./bsp/arc

          Only working is 'arc' but then this excludes both folders.

          janc Szymon Janc added a comment - ok, new release is coming so I'm back to using RAT. Lets say I have those new files: hw/bsp/arc/file1 hw/bsp/arc/file2 hw/bsp/arc/file3 bsp/arc/file1 bsp/arc/file2 bsp/arc/file3 How do I exclude files in bsp/arc/ without excluding files in hw/bsp/arc? I'd expect somtehing like this (or similar) to work but none of combination I tried works. ./bsp/arc/* ./bsp/arc Only working is 'arc' but then this excludes both folders.

          Did you try using:

          bsp/arc/**
          

          The functionality is currently implemented to exclude as much as possible.

          The kind of filtering you want to have is not yet implemented - feel free to file a patch with a detailed sketch on how to do this.

          pottlinger Philipp Ottlinger added a comment - Did you try using: bsp/arc/** The functionality is currently implemented to exclude as much as possible. The kind of filtering you want to have is not yet implemented - feel free to file a patch with a detailed sketch on how to do this.
          claude Claude Warren added a comment - - edited

          This filtering is almost covered in PR #311

          This request reworks the exclusion code and moves the functionality that was in the Maven module into  core so everyone can use it.

          In that pull there is code in the config/exclusion/fileProcessors directory that handles reading files like .gitignore or .cvsignore where the file is in one location and only provides changes for the locations below it.

          We have the ability to specify a file on the command line that will exclude all the patterns below the directory that scanning starts in.

          It would be a simple exercize to create a ".ratignore" file parser that would process all ".ratignore" files in the tree and provided patters at various levels to ignore (or stop ignoring) them.

          If pottlinger, janc, or utzig are interested in exploring this drop me a line and I will give you pointers for how to create the processor and we can discuss how to plug it in.  I need to write developer docs anyway so this would be a good start along that path too.

           

          Also, note that this ticket is part of the Harmonization ticket and will be in 0.17

           

           

          claude Claude Warren added a comment - - edited This filtering is almost covered in PR #311 This request reworks the exclusion code and moves the functionality that was in the Maven module into  core so everyone can use it. In that pull there is code in the config/exclusion/fileProcessors directory that handles reading files like .gitignore or .cvsignore where the file is in one location and only provides changes for the locations below it. We have the ability to specify a file on the command line that will exclude all the patterns below the directory that scanning starts in. It would be a simple exercize to create a ".ratignore" file parser that would process all ".ratignore" files in the tree and provided patters at various levels to ignore (or stop ignoring) them. If pottlinger , janc , or utzig are interested in exploring this drop me a line and I will give you pointers for how to create the processor and we can discuss how to plug it in.  I need to write developer docs anyway so this would be a good start along that path too.   Also, note that this ticket is part of the Harmonization ticket and will be in 0.17    

          People

            Unassigned Unassigned
            janc Szymon Janc
            Votes:
            0 Vote for this issue
            Watchers:
            6 Start watching this issue

            Dates

              Created:
              Updated: