Uploaded image for project: 'Apache NiFi'
  1. Apache NiFi
  2. NIFI-7276

PutFTP create alot of received trafic when tx small files

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Open
    • Major
    • Resolution: Unresolved
    • 1.11.4
    • None
    • Extensions
    • NIFI: Ubuntu 18.04. NIFI 1.11.4
      FTP: Ubuntu 18.04 with vsftpd

    Description

      I don't know if this is a bug, but the result of my investigation is that it seems like the PutFTP receive between 1,6 - 2,7MB of data per file it transmit. For me it seems like a lot of RX data.

      I'm using nload and nethogs to monitor the network trafic.

      I have a lot (100.000+) of small files (<1MB), which I can't combine.

      When starting the PutFTP process the nifi server starts receiving data from the ftp server and it continues with with an avg. of 50 Mbit/s, constantly.

      To increasing the throughput I've set the run duration to 2sec. and playing with the Batch size. With a batch size of 100 and over 5 min. I can see at the ftp server that nifi is logging in 5 times and start a vsftpd process. within those 5 min. nifi has tx 700 files and 65.32 MB, but the ftp server has sent 1544MB data back to nifi.

      With a batch size of 1000. It took nifi around 8 minutes for transferring 1000 files with a total sum of 64MB, but the ftp server has this time TX 2700MB data back to nifi.

      I tried with larger files and a batch size at 100. NIFI sent 249 files with at total sum of 32 GB of data within 3½ min. and this time the ftp server sent 400MB data back to nifi.

      It seems like the PutFTP receive between 1,6 - 2,7MB of data per file it transmit. For me it seems like alot of RX data.

      Attachments

        Activity

          People

            Unassigned Unassigned
            jmkofoed Jens M Kofoed
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

            Dates

              Created:
              Updated: