Uploaded image for project: 'TinkerPop'
  1. TinkerPop
  2. TINKERPOP-927

bin/publish-docs.sh should only upload diffs.

Details

    • Improvement
    • Status: Closed
    • Major
    • Resolution: Done
    • 3.0.2-incubating
    • 3.1.5
    • build-release
    • None

    Description

      It takes forever for us to publish-docs. Can we just do a "diff" and see what changed (MD5?) and then svn up only those files?

      Attachments

        Activity

          I'd say it would be equally important to generate just one book. Not sure if that's a different ticket or not? I guess we'd have to do such a thing by revising maven profiles to be one profile per book. Not sure if there's a better way.

          spmallette Stephen Mallette added a comment - I'd say it would be equally important to generate just one book. Not sure if that's a different ticket or not? I guess we'd have to do such a thing by revising maven profiles to be one profile per book. Not sure if there's a better way.
          dkuppitz Daniel Kuppitz added a comment -

          I don't like the idea of touching svn. This would require that every contributor, who wants to process the docs, would have to have svn access. I think we can have an environment variable (for example TP3_DOCS_PREPROCESSOR_TMP_DIR) - if it's set, the preprocessor would store the results there, otherwise in the target/ directory (as it is now).

          This approach would allow you to do a mvn clean install and the preprocessor would still be able to determine whether it needs to process a particular file or not.

          dkuppitz Daniel Kuppitz added a comment - I don't like the idea of touching svn. This would require that every contributor, who wants to process the docs, would have to have svn access. I think we can have an environment variable (for example TP3_DOCS_PREPROCESSOR_TMP_DIR ) - if it's set, the preprocessor would store the results there, otherwise in the target/ directory (as it is now). This approach would allow you to do a mvn clean install and the preprocessor would still be able to determine whether it needs to process a particular file or not.

          dkuppitz - sorry - i don't understand your approach and it's perhaps because i don't understand the pre-processor. what does bin/process-docs.sh have to do with svn? isn't all the svn stuff in bin/publish-docs.sh?

          spmallette Stephen Mallette added a comment - dkuppitz - sorry - i don't understand your approach and it's perhaps because i don't understand the pre-processor. what does bin/process-docs.sh have to do with svn? isn't all the svn stuff in bin/publish-docs.sh?
          dkuppitz Daniel Kuppitz added a comment -

          I was referring to Marko's statement " Can we just do a "diff" and see what changed (MD5?)". The final result is only accessible through svn (or on your local machine as long as you don't do a mvn clean).

          dkuppitz Daniel Kuppitz added a comment - I was referring to Marko's statement " Can we just do a "diff" and see what changed (MD5?)". The final result is only accessible through svn (or on your local machine as long as you don't do a mvn clean ).

          I've been manually updating SVN to avoid the pain of the full data upload, especially when I just update just the developer docs or the tutorial. It seems like it still might be nicest if we could specify that there was just one book we wanted to publish.

          spmallette Stephen Mallette added a comment - I've been manually updating SVN to avoid the pain of the full data upload, especially when I just update just the developer docs or the tutorial. It seems like it still might be nicest if we could specify that there was just one book we wanted to publish.

          To build on my last comment a little, maybe we could have an option that only uploaded the book .html files only. Unless you add new images, most of the time you really only care about the "book" html files. we could also option out of javadoc which is probably just as expensive and adding back all the images.

          spmallette Stephen Mallette added a comment - To build on my last comment a little, maybe we could have an option that only uploaded the book .html files only. Unless you add new images, most of the time you really only care about the "book" html files. we could also option out of javadoc which is probably just as expensive and adding back all the images.
          spmallette Stephen Mallette added a comment - - edited

          The upload time for docs is extending longer and longer. I was thinking that a cheap way to fix this without worrying about how to do diffs would be to add a "--noJavadoc" option which would cut out about 60% of the upload time based on the relative file sizes.

          If that were implemented along with a "--noImages" options we'd get it down an to be almost instantaneous compared to where we are now (we'd only be uploading a couple of MB).

          dkuppitz does that make this issue any easier to implement?

          spmallette Stephen Mallette added a comment - - edited The upload time for docs is extending longer and longer. I was thinking that a cheap way to fix this without worrying about how to do diffs would be to add a "--noJavadoc" option which would cut out about 60% of the upload time based on the relative file sizes. If that were implemented along with a "--noImages" options we'd get it down an to be almost instantaneous compared to where we are now (we'd only be uploading a couple of MB). dkuppitz does that make this issue any easier to implement?
          dkuppitz Daniel Kuppitz added a comment - - edited

          Note to myself: The checksum reported by svn info <file> can be compared to the local checksum reported by sha1sum <file>.

          EDIT:

          Nvm, you cannot get the checksum of a remote file.

          dkuppitz Daniel Kuppitz added a comment - - edited Note to myself: The checksum reported by svn info <file> can be compared to the local checksum reported by sha1sum <file> . EDIT: Nvm, you cannot get the checksum of a remote file.
          dkuppitz Daniel Kuppitz added a comment -

          Again a note to myself: After building the docs and checking out the the current remote version:

          rm -rf target/svn
          
          mkdir -p target/svn
          ${SVN_CMD} co --depth immediates https://svn.apache.org/repos/asf/tinkerpop/site target/svn
          
          rm -f target/docs/htmlsingle/images/tinkerpop3.graffle
          
          pushd target/svn
          
          ${SVN_CMD} update "docs/${VERSION}"
          

          ...the following command can be used to get a nice overview of differences:

          # create a dummy file (only to test the "new file" scenario)
          touch ../docs/htmlsingle/new
          diff -rq docs/3.2.2-SNAPSHOT/ ../docs/htmlsingle/ | awk '/^Files / {print "U " gensub("^[^/]*/[^/]*/", "", "g", $2)} /^Only in docs/ {print "D " gensub(/:$/, "", "g", gensub("^[^/]*/[^/]*/", "", "g", $3)) "/" $4} /^Only in \.\./ {print "A " gensub(/:$/, "", "g", gensub("^[^/]*/[^/]*/[^/]*/", "", "g", $3)) "/" $4}' | sed 's/^\(.\) \//\1 /g'
          popd
          

          I've tested it by checking out the published 3.2.2-SNAPSHOT docs and building the 3.2.1 docs locally. This was the result:

          U dev/developer/index.html
          U dev/provider/index.html
          D images/betweeness-example.png
          D images/graph-cycle.png
          D images/gremlin-chef.png
          U images/gremlin-gym.png
          D images/gremlin-house-of-mirrors-cropped.png
          D images/gremlin-house-of-mirrors.png
          D images/gremlin-python-drawing.png
          D images/gremlin-variant-architecture.png
          D images/jython-logo.png
          D images/language-drivers.png
          D images/language-variants.png
          D images/nine-inch-gremlins.png
          U images/olap-traversal.png
          D images/python-logo.png
          D images/recipe-job-schema.png
          D images/remote-graph.png
          D images/shortest-path.png
          U index.html
          A new
          D recipes
          U reference/index.html
          U tutorials/getting-started/index.html
          D tutorials/gremlin-language-variants
          U tutorials/the-gremlin-console/index.html
          U upgrade/index.html
          

          Based on that publish-docs.sh could run a few commands to update the currently published docs and the whole process should be done within a few seconds.

          dkuppitz Daniel Kuppitz added a comment - Again a note to myself: After building the docs and checking out the the current remote version: rm -rf target/svn mkdir -p target/svn ${SVN_CMD} co --depth immediates https: //svn.apache.org/repos/asf/tinkerpop/site target/svn rm -f target/docs/htmlsingle/images/tinkerpop3.graffle pushd target/svn ${SVN_CMD} update "docs/${VERSION}" ...the following command can be used to get a nice overview of differences: # create a dummy file (only to test the " new file" scenario) touch ../docs/htmlsingle/ new diff -rq docs/3.2.2-SNAPSHOT/ ../docs/htmlsingle/ | awk '/^Files / {print "U " gensub( "^[^/]*/[^/]*/" , "", " g ", $2)} /^Only in docs/ {print " D " gensub(/:$/, " ", " g ", gensub(" ^[^/]*/[^/]*/ ", " ", " g ", $3)) " / " $4} /^Only in \.\./ {print " A " gensub(/:$/, " ", " g ", gensub(" ^[^/]*/[^/]*/[^/]*/ ", " ", " g ", $3)) " /" $4}' | sed 's/^\(.\) \ //\1 /g' popd I've tested it by checking out the published 3.2.2-SNAPSHOT docs and building the 3.2.1 docs locally. This was the result: U dev/developer/index.html U dev/provider/index.html D images/betweeness-example.png D images/graph-cycle.png D images/gremlin-chef.png U images/gremlin-gym.png D images/gremlin-house-of-mirrors-cropped.png D images/gremlin-house-of-mirrors.png D images/gremlin-python-drawing.png D images/gremlin-variant-architecture.png D images/jython-logo.png D images/language-drivers.png D images/language-variants.png D images/nine-inch-gremlins.png U images/olap-traversal.png D images/python-logo.png D images/recipe-job-schema.png D images/remote-graph.png D images/shortest-path.png U index.html A new D recipes U reference/index.html U tutorials/getting-started/index.html D tutorials/gremlin-language-variants U tutorials/the-gremlin-console/index.html U upgrade/index.html Based on that publish-docs.sh could run a few commands to update the currently published docs and the whole process should be done within a few seconds.
          githubbot ASF GitHub Bot added a comment -

          GitHub user dkuppitz opened a pull request:

          https://github.com/apache/tinkerpop/pull/427

          TINKERPOP-927 bin/publish-docs.sh should only upload diffs.

          https://issues.apache.org/jira/browse/TINKERPOP-927

          VOTE: +1

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/apache/tinkerpop TINKERPOP-927

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tinkerpop/pull/427.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #427


          commit 5f2d2519373e4df3550bc80d2ab6b936366a3ead
          Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
          Date: 2016-09-19T13:20:50Z

          Upload only diffs in `publish-docs.sh`.


          githubbot ASF GitHub Bot added a comment - GitHub user dkuppitz opened a pull request: https://github.com/apache/tinkerpop/pull/427 TINKERPOP-927 bin/publish-docs.sh should only upload diffs. https://issues.apache.org/jira/browse/TINKERPOP-927 VOTE: +1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/tinkerpop TINKERPOP-927 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tinkerpop/pull/427.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #427 commit 5f2d2519373e4df3550bc80d2ab6b936366a3ead Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com> Date: 2016-09-19T13:20:50Z Upload only diffs in `publish-docs.sh`.
          githubbot ASF GitHub Bot added a comment -

          GitHub user dkuppitz opened a pull request:

          https://github.com/apache/tinkerpop/pull/428

          TINKERPOP-927 bin/publish-docs.sh should only upload diffs.

          https://issues.apache.org/jira/browse/TINKERPOP-927

          VOTE: +1

          You can merge this pull request into a Git repository by running:

          $ git pull https://github.com/apache/tinkerpop TINKERPOP-927-master

          Alternatively you can review and apply these changes as the patch at:

          https://github.com/apache/tinkerpop/pull/428.patch

          To close this pull request, make a commit to your master/trunk branch
          with (at least) the following in the commit message:

          This closes #428


          commit 5f2d2519373e4df3550bc80d2ab6b936366a3ead
          Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
          Date: 2016-09-19T13:20:50Z

          Upload only diffs in `publish-docs.sh`.

          commit b20921bd4d42a08db6a182b26dd22e4808d837c2
          Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
          Date: 2016-09-19T15:23:47Z

          Merge branch 'TINKERPOP-927' into TINKERPOP-927-master

          commit 144e6f08d8bbefa3bf1810cac39a41c2f3b1a21e
          Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
          Date: 2016-09-19T15:25:05Z

          Fixed javadocs file reference.

          commit 0100f78c79460152c1e5fd7793bef215e4558311
          Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
          Date: 2016-09-19T15:26:48Z

          Merge branch 'TINKERPOP-927' into TINKERPOP-927-master

          commit 120cef66da6afac1ecbb5781fe58d3ca3e1f0129
          Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
          Date: 2016-09-19T23:12:52Z

          Fixed temporary file path.


          githubbot ASF GitHub Bot added a comment - GitHub user dkuppitz opened a pull request: https://github.com/apache/tinkerpop/pull/428 TINKERPOP-927 bin/publish-docs.sh should only upload diffs. https://issues.apache.org/jira/browse/TINKERPOP-927 VOTE: +1 You can merge this pull request into a Git repository by running: $ git pull https://github.com/apache/tinkerpop TINKERPOP-927 -master Alternatively you can review and apply these changes as the patch at: https://github.com/apache/tinkerpop/pull/428.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #428 commit 5f2d2519373e4df3550bc80d2ab6b936366a3ead Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com> Date: 2016-09-19T13:20:50Z Upload only diffs in `publish-docs.sh`. commit b20921bd4d42a08db6a182b26dd22e4808d837c2 Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com> Date: 2016-09-19T15:23:47Z Merge branch ' TINKERPOP-927 ' into TINKERPOP-927 -master commit 144e6f08d8bbefa3bf1810cac39a41c2f3b1a21e Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com> Date: 2016-09-19T15:25:05Z Fixed javadocs file reference. commit 0100f78c79460152c1e5fd7793bef215e4558311 Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com> Date: 2016-09-19T15:26:48Z Merge branch ' TINKERPOP-927 ' into TINKERPOP-927 -master commit 120cef66da6afac1ecbb5781fe58d3ca3e1f0129 Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com> Date: 2016-09-19T23:12:52Z Fixed temporary file path.
          githubbot ASF GitHub Bot added a comment - Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/428 dopest +1 ever ![]( http://migratingtaste.oldeenglish.org/iloverap/threetimesdope-funkydividendremix.jpg )
          githubbot ASF GitHub Bot added a comment -

          Github user spmallette commented on the issue:

          https://github.com/apache/tinkerpop/pull/427

          Didn't test this on tp31 - but i'm sold on #428

          VOTE +1

          githubbot ASF GitHub Bot added a comment - Github user spmallette commented on the issue: https://github.com/apache/tinkerpop/pull/427 Didn't test this on tp31 - but i'm sold on #428 VOTE +1
          githubbot ASF GitHub Bot added a comment -

          Github user okram commented on the issue:

          https://github.com/apache/tinkerpop/pull/428

          VOTE +1.

          githubbot ASF GitHub Bot added a comment - Github user okram commented on the issue: https://github.com/apache/tinkerpop/pull/428 VOTE +1.
          githubbot ASF GitHub Bot added a comment -

          Github user okram commented on the issue:

          https://github.com/apache/tinkerpop/pull/427

          VOTE +1.

          githubbot ASF GitHub Bot added a comment - Github user okram commented on the issue: https://github.com/apache/tinkerpop/pull/427 VOTE +1.
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tinkerpop/pull/427

          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tinkerpop/pull/427
          githubbot ASF GitHub Bot added a comment -

          Github user asfgit closed the pull request at:

          https://github.com/apache/tinkerpop/pull/428

          githubbot ASF GitHub Bot added a comment - Github user asfgit closed the pull request at: https://github.com/apache/tinkerpop/pull/428

          People

            dkuppitz Daniel Kuppitz
            okram Marko A. Rodriguez
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved: