Details
-
Sub-task
-
Status: Open
-
Major
-
Resolution: Unresolved
-
0.12
-
None
-
None
Description
it is not possible to exclude file (or folder) by providing full path
eg
foo/foo2/main.c
bar/bar2/main.c
it is not possible to match only one of those file from project top directory .rat-excludes file.
The initial proble
Attachments
Attachments
- rat.txt
- 648 kB
- Szymon Janc
Activity
ping - any updates on the issue?
Since I cannot reproduce it I will close the issue as won't fix.
Looking forward to your feedback.
Thanks
Hi,
We use RAT in Apache Mynewt project for license checks.
here is the link for our exclude https://github.com/apache/mynewt-core/blob/master/.rat-excludes
our project code structure if that includes are always in include/ folders, and sources are in src/
this is 'embedded' project so we have a lot of imported code from SDK/BSP for board support,
some libs imported for crypt, protocols etc
result in structure like this:
eg
hw/bsp/boardA/src/bsp.c
hw/bsp/boardA/include/bsp.h
hw/bsp/boardB/src/bsp.c
hw/bsp/boardB/include/bsp.h
net/lora/src/foo.c
net/lora/include/foo.h
both can be on different licenses which we would like to exclude from RAT check (with proper comment in exclude file)
unfortunately currently only single depth path are accepted by RAT (eg src/bsp.c) but not full path.
If I provide hw/bsp/boardA/src/bsp.c it is not excluded by RAT.
So, for us, the best solution would be option to exclude by providing full path (either folder or explicitly files would work) eg
hw/bsp/boardA/*
hw/bsp/boardA/src/foo.c
If this is no possible (due to the way RAT works) then maybe RAT could have an option for implicitly using .rat-exclude file from folder where checked files is located?
That would also work for us without risk of ambiguous names matching.
Hope this clarifies our needs
I'm beginning to understand your problem -
in case you are using the Maven integration you are already able to exclude a certain tree/directory - please see my commit
https://github.com/ottlinger/creadur-rat-examples/commit/653545b58270fc714e0e667e3049720d960e6131
on how to configure this.
At the moment there's no functionality to provide a global ignore file.
In case you want to request that feel free to file an improvement since bugs relate to errors in the current implementation
Thanks
Having a look at
https://github.com/apache/mynewt-core/blob/master/.rat-excludes
I wonder, how do you execute RAT on your project?
Did you know that RAT uses SCM-ignores as default exclusions - maybe this can be helpful for you in case you generated stuff during build/package time?
Apart from that I just stumbled upon the configuration parameter:
@Parameter(property="rat.excludesFile")
in
https://github.com/apache/creadur-rat/blob/b1f2f1d470a6cfee656a06ab10ea1210a2e51b32/apache-rat-plugin/src/main/java/org/apache/rat/mp/AbstractRatMojo.java
are you using this feature and it does not behave as expected?
oh, good to know that RAT is making use of gitignore, will cleanup our excludes a bit
as for using RAT, I simply run "java -jar ~/Downloads/apache-rat-0.12/apache-rat-0.12.jar -E .rat-excludes -d ." from project top folder.
hmm - what's the expected behaviour. I can run RAT successfully and get some files without licenses:
$ java -jar ~/Downloads/apache-rat-0.12.jar -E .rat-excludes -d .
*****************************************************
Summary
-------
Generated at: 2017-10-05T23:25:03+02:00
Notes: 4
Binaries: 0
Archives: 0
Standards: 1786
Apache Licensed: 1765
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
6 Unknown Licenses
*****************************************************
Files with unapproved licenses:
./hw/bsp/pic32mx470_6lp_clicker/src/sbrk.c
./hw/bsp/pic32mz2048_wi-fire/src/sbrk.c
./kernel/os/include/os/arch/pic32/os/cp0defs.h
./kernel/os/src/arch/pic32/startup/cache-err-exception.S
./kernel/os/src/arch/pic32/startup/crt0.S
./kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S
...
So your expectation would be to exclude /hw/bsp or should the kernel also be excluded?
It's possible that the CLI does not have the same functionality as the maven version, but at first I'd like to find out what you want to have.
Thanks
if you look at .rat-exclude you'll see that we are force to use very generic names for exclusion like lora, mesh, rtt etc and those can simply exclude too much
so what this report is about is that we would like to be able to have our rat-exclude be like:
./kernel/os/src/arch/pic32/
./hw/bsp/pic32mz2048_wi-fire/
./net/lora/node/
./hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h
instead of currently used
pic32
lora
compiler_abstraction.h
as those can simply match too much.
Commit 1811592 from hugo.hirsch in branch 'rat/trunk'
[ https://svn.apache.org/r1811592 ]
RAT-240: Add comments in exclusion files/change parsing
- Added skippig of empty and comment lines.
- Added new NameFileFilter and WildcardFileFilter to support more
exclusions in different formats.
janc can you verify with the current 0.13-SNAPSHOT that your stuff is properly excluded ?!
I checked out your project and added the following example lines to your .rat-exclude-file:
+ +# RAT-240 +#.*hw/bsp* +/hw/bsp/pic32mz2048_wi-fire/src/sbrk.c +/kernel/os/src/arch/pic32/* +/hw/bsp/pic32mz2048_wi-fire/* +/net/lora/node/ +/hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h +pic32mx470_6lp_clicker* +run.sh +# from the ticket itself +/kernel/os/src/arch/pic32/ +/hw/bsp/pic32mz2048_wi-fire/ +/net/lora/node/ +/hw/mcu/nordic/src/ext/nRF5_SDK_11.0.0_89a8197/components/device/compiler_abstraction.h +# New files? +hw/bsp/pic32mz2048_wi-fire/src/sbrk.c +kernel/os/include/os/arch/pic32/os/cp0defs.h +kernel/os/src/arch/pic32/startup/cache-err-exception.S +kernel/os/src/arch/pic32/startup/crt0.S +kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S +#**.***
My local version runs with:
$ cat run.sh #!/bin/bash java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more
Is that what you need?
Commit 1811618 from hugo.hirsch in branch 'rat/trunk'
[ https://svn.apache.org/r1811618 ]
RAT-240: Add change documentation and update tests
- Test comment/empty line input.
- Test input that is trimmed.
- Add changes.
Hi,
which version should I use? I tried apache-rat-0.13-20171009.224400-55.jar and couldn't make it work as you described
/hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx
/hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/*
/hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h
none of those work for excluding, I get this:
Files with unapproved licenses:
...
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Release_Notes.html
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xe.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f405xx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f407xx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410cx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410rx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f410tx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f411xe.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412cx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412rx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412vx.h
./hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f412zx.h
.....
The version should be correct (latest SNAPSHOT) - are you on windows?
I've checked out your project and added
/hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/* /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx/Include/stm32f401xc.h
to the exclusion file and it works fine
$ java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more
Can you check out the version and try to build it for yourself just to make sure you are using the latest SNAPHOT?
https://svn.apache.org/repos/asf/creadur/rat/trunk
HTH
I'm running this on Linux (Fedora 26).
I'm bit puzzled since I'm not able to make it work as you described. I'm attaching log (rat.txt) showing change I made in .rat-excludes and output of RAT execution.
As you can see /hw/mcu/stm/stm32f4xx/src/ext/Drivers/CMSIS/Device/ST/STM32F4xx is not excluded.
Any advice?
Are you working on a specific branch?
I've checked out mynewt-master and added:
$ git diff diff --git a/.rat-excludes b/.rat-excludes index bc3eb00..bc08173 100644 --- a/.rat-excludes +++ b/.rat-excludes @@ -344,3 +344,6 @@ split-calliope_mini.ld # Bluetooth Mesh - Apache 2.0 License mesh + +docs* +run.sh
to .rat-excludes.
Running RAT yields expectedly missing files:
✔ ~/workspace/mynewt-core [master|✚ 1…1]
23:30 $ ./run.sh
Ignored 95 lines in your exclusion files as comments or empty lines.
*****************************************************
Summary
-------
Generated at: 2017-11-06T23:30:04+01:00
Notes: 4
Binaries: 0
Archives: 0
Standards: 1841
Apache Licensed: 1795
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
30 Unknown Licenses
*****************************************************
Files with unapproved licenses:
./hw/bsp/ci40/src/arch/mips/abiflags.S
./hw/bsp/ci40/src/arch/mips/gcc_startup_mips.S
./hw/bsp/pic32mx470_6lp_clicker/src/sbrk.c
./hw/bsp/pic32mz2048_wi-fire/src/sbrk.c
./hw/mips-hal/src/arch/mips/__exit.c
./hw/mips-hal/src/arch/mips/cache.S
./hw/mips-hal/src/arch/mips/cache.h
./hw/mips-hal/src/arch/mips/cache_ops.S
./hw/mips-hal/src/arch/mips/link.c
./hw/mips-hal/src/arch/mips/m32cache.S
./hw/mips-hal/src/arch/mips/m32cache.h
./hw/mips-hal/src/arch/mips/m32cache_ops.S
./hw/mips-hal/src/arch/mips/m32tlb_ops.S
./hw/mips-hal/src/arch/mips/m64tlb_ops.S
./hw/mips-hal/src/arch/mips/mips_cm3_l2size.S
./hw/mips-hal/src/arch/mips/mips_excpt_boot.S
./hw/mips-hal/src/arch/mips/mips_excpt_entry.S
./hw/mips-hal/src/arch/mips/mips_excpt_handler.c
./hw/mips-hal/src/arch/mips/mips_excpt_isr.S
./hw/mips-hal/src/arch/mips/mips_excpt_register.S
./hw/mips-hal/src/arch/mips/mips_fp.S
./hw/mips-hal/src/arch/mips/mips_l2size.S
./hw/mips-hal/src/arch/mips/mips_msa.S
./hw/mips-hal/src/arch/mips/mips_xpa.S
./hw/mips-hal/src/arch/mips/mxxtlb_ops.S
./kernel/os/include/os/arch/pic32/os/cp0defs.h
./kernel/os/src/arch/pic32/startup/cache-err-exception.S
./kernel/os/src/arch/pic32/startup/crt0.S
./kernel/os/src/arch/pic32/startup/simple-tlb-refill-exception.S
./net/nimble/host/test/src/ble_hs_test_util_hci.c
*****************************************************
*****************************************************
Files with Apache License headers will be marked AL
....
but not the situation you are describing.
run.sh looks like that to ease debugging:
#!/bin/bash java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more
Which Java version are you using?
I'm on
$ java --version
java 9.0.1
Java(TM) SE Runtime Environment (build 9.0.1+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.1+11, mixed mode)
same result with Oracle JDK1.8_152.
Not sure how I can help you here anymore.
I'm using one from Fedora
[janc@ix ~]$ java -version
openjdk version "1.8.0_151"
OpenJDK Runtime Environment (build 1.8.0_151-b12)
OpenJDK 64-Bit Server VM (build 25.151-b12, mixed mode)
note that my java doesn't accept --version, only -version
is OpenJDK supported?
I tested with both openjdk 8 and 9 and both didn't work. Btw, I was only able to build with openjdk8 (9 fails). I will also give sun's jdk/jre 8 a try...
Also tested with:
$ java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)
And it also didn't work. I took a quick look at the source code and it seems that DirectoryWalker is only applying exclusion patterns to the leaves in the path but I might be getting it completely wrong since I never spent much time programming in java.
Would you mind providing your own build (apache-rat-0.13-SNAPSHOT.jar) for me to test here?
Hi guys, utzigjanc
I'm afraid I'm still unable to understand your problems.
I've clone your repository
✔ ~/workspace/mynewt-core [master|…1]
and added a run.sh which yields the following output:
$ ./run.sh
Ignored 111 lines in your exclusion files as comments or empty lines.
*****************************************************
Summary
-------
Generated at: 2017-11-23T23:40:35+01:00
Notes: 4
Binaries: 0
Archives: 0
Standards: 1830
Apache Licensed: 1813
Generated Documents: 0
JavaDocs are generated, thus a license header is optional.
Generated files do not require license headers.
1 Unknown Licenses
*****************************************************
Files with unapproved licenses:
./run.sh
*****************************************************
.....
My run.sh is:
$ cat run.sh
#!/bin/bash
/home/me/jdk1.8.0_152/bin/java -jar ../rat-trunk/apache-rat/target/apache-rat-0.13-SNAPSHOT.jar -E .rat-excludes -d . | more
I've checked you rat and build it a while ago, thus the reference.
As the build goes through successfully I don't know what I miss here.
ok, new release is coming so I'm back to using RAT.
Lets say I have those new files:
hw/bsp/arc/file1
hw/bsp/arc/file2
hw/bsp/arc/file3
bsp/arc/file1
bsp/arc/file2
bsp/arc/file3
How do I exclude files in bsp/arc/ without excluding files in hw/bsp/arc?
I'd expect somtehing like this (or similar) to work but none of combination I tried works.
./bsp/arc/*
./bsp/arc
Only working is 'arc' but then this excludes both folders.
Did you try using:
bsp/arc/**
The functionality is currently implemented to exclude as much as possible.
The kind of filtering you want to have is not yet implemented - feel free to file a patch with a detailed sketch on how to do this.
This filtering is almost covered in PR #311
This request reworks the exclusion code and moves the functionality that was in the Maven module into core so everyone can use it.
In that pull there is code in the config/exclusion/fileProcessors directory that handles reading files like .gitignore or .cvsignore where the file is in one location and only provides changes for the locations below it.
We have the ability to specify a file on the command line that will exclude all the patterns below the directory that scanning starts in.
It would be a simple exercize to create a ".ratignore" file parser that would process all ".ratignore" files in the tree and provided patters at various levels to ignore (or stop ignoring) them.
If pottlinger, janc, or utzig are interested in exploring this drop me a line and I will give you pointers for how to create the processor and we can discuss how to plug it in. I need to write developer docs anyway so this would be a good start along that path too.
Also, note that this ticket is part of the Harmonization ticket and will be in 0.17
janc not sure what you mean or what kind of exclusions you want to configure -
is https://github.com/ottlinger/creadur-rat-examples/tree/master/RAT-240 what you try to achieve?
If you want to exclude a full folder, you may configure it like that
Do you have a link to the project that shows the behaviour you reported?
Thanks