Uploaded image for project: 'Commons Compress'
  1. Commons Compress
  2. COMPRESS-224

Cannot uncompress very large bzip2 files

    XMLWordPrintableJSON

Details

    • Bug
    • Status: Resolved
    • Blocker
    • Resolution: Duplicate
    • 1.5
    • None
    • None
    • None
    • Java 1.7.0_03

    Description

      When extracting big files like http://download.geofabrik.de/europe/germany/bayern-latest.osm.bz2 apache-compress works nicely. But when trying the same for e.g. http://ftp5.gwdg.de/pub/misc/openstreetmap/planet.openstreetmap.org/planet/planet-latest.osm.bz2 it stops without an error after exactly 900000 bits.

      I'm using the following code:

      App.java
       public static void main(String[] args) throws IOException {
              if (args.length == 0)
                  throw new IllegalArgumentException("You need to specify the bz2 file!");
      
              String fromFile = args[0];
              if (!fromFile.endsWith(".bz2"))
                  throw new IllegalArgumentException("You need to specify a bz2 file! But was:" + fromFile);
              String toFile = pruneFileEnd(fromFile);
      
              FileInputStream in = new FileInputStream(fromFile);
              FileOutputStream out = new FileOutputStream(toFile);
              BZip2CompressorInputStream bzIn = new BZip2CompressorInputStream(in);
              try {
                  final byte[] buffer = new byte[1024 * 8];
                  int n = 0;
                  while (-1 != (n = bzIn.read(buffer))) {
                      out.write(buffer, 0, n);
                  }
              } finally {
                  out.close();
                  bzIn.close();
              }
          }
      
          public static String pruneFileEnd(String file) {
              int index = file.lastIndexOf(".");
              if (index < 0)
                  return file;
              return file.substring(0, index);
          }
      

      Attachments

        Issue Links

          Activity

            People

              Unassigned Unassigned
              peathal Peter Karich
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved: