Details
-
Improvement
-
Status: Closed
-
Major
-
Resolution: Done
-
3.0.2-incubating
-
None
Description
It takes forever for us to publish-docs. Can we just do a "diff" and see what changed (MD5?) and then svn up only those files?
Attachments
Issue Links
Activity
I don't like the idea of touching svn. This would require that every contributor, who wants to process the docs, would have to have svn access. I think we can have an environment variable (for example TP3_DOCS_PREPROCESSOR_TMP_DIR) - if it's set, the preprocessor would store the results there, otherwise in the target/ directory (as it is now).
This approach would allow you to do a mvn clean install and the preprocessor would still be able to determine whether it needs to process a particular file or not.
dkuppitz - sorry - i don't understand your approach and it's perhaps because i don't understand the pre-processor. what does bin/process-docs.sh have to do with svn? isn't all the svn stuff in bin/publish-docs.sh?
I was referring to Marko's statement " Can we just do a "diff" and see what changed (MD5?)". The final result is only accessible through svn (or on your local machine as long as you don't do a mvn clean).
I've been manually updating SVN to avoid the pain of the full data upload, especially when I just update just the developer docs or the tutorial. It seems like it still might be nicest if we could specify that there was just one book we wanted to publish.
To build on my last comment a little, maybe we could have an option that only uploaded the book .html files only. Unless you add new images, most of the time you really only care about the "book" html files. we could also option out of javadoc which is probably just as expensive and adding back all the images.
The upload time for docs is extending longer and longer. I was thinking that a cheap way to fix this without worrying about how to do diffs would be to add a "--noJavadoc" option which would cut out about 60% of the upload time based on the relative file sizes.
If that were implemented along with a "--noImages" options we'd get it down an to be almost instantaneous compared to where we are now (we'd only be uploading a couple of MB).
dkuppitz does that make this issue any easier to implement?
Note to myself: The checksum reported by svn info <file> can be compared to the local checksum reported by sha1sum <file>.
EDIT:
Nvm, you cannot get the checksum of a remote file.
Again a note to myself: After building the docs and checking out the the current remote version:
rm -rf target/svn mkdir -p target/svn ${SVN_CMD} co --depth immediates https://svn.apache.org/repos/asf/tinkerpop/site target/svn rm -f target/docs/htmlsingle/images/tinkerpop3.graffle pushd target/svn ${SVN_CMD} update "docs/${VERSION}"
...the following command can be used to get a nice overview of differences:
# create a dummy file (only to test the "new file" scenario) touch ../docs/htmlsingle/new diff -rq docs/3.2.2-SNAPSHOT/ ../docs/htmlsingle/ | awk '/^Files / {print "U " gensub("^[^/]*/[^/]*/", "", "g", $2)} /^Only in docs/ {print "D " gensub(/:$/, "", "g", gensub("^[^/]*/[^/]*/", "", "g", $3)) "/" $4} /^Only in \.\./ {print "A " gensub(/:$/, "", "g", gensub("^[^/]*/[^/]*/[^/]*/", "", "g", $3)) "/" $4}' | sed 's/^\(.\) \//\1 /g' popd
I've tested it by checking out the published 3.2.2-SNAPSHOT docs and building the 3.2.1 docs locally. This was the result:
U dev/developer/index.html U dev/provider/index.html D images/betweeness-example.png D images/graph-cycle.png D images/gremlin-chef.png U images/gremlin-gym.png D images/gremlin-house-of-mirrors-cropped.png D images/gremlin-house-of-mirrors.png D images/gremlin-python-drawing.png D images/gremlin-variant-architecture.png D images/jython-logo.png D images/language-drivers.png D images/language-variants.png D images/nine-inch-gremlins.png U images/olap-traversal.png D images/python-logo.png D images/recipe-job-schema.png D images/remote-graph.png D images/shortest-path.png U index.html A new D recipes U reference/index.html U tutorials/getting-started/index.html D tutorials/gremlin-language-variants U tutorials/the-gremlin-console/index.html U upgrade/index.html
Based on that publish-docs.sh could run a few commands to update the currently published docs and the whole process should be done within a few seconds.
GitHub user dkuppitz opened a pull request:
https://github.com/apache/tinkerpop/pull/427
TINKERPOP-927 bin/publish-docs.sh should only upload diffs.
https://issues.apache.org/jira/browse/TINKERPOP-927
VOTE: +1
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/tinkerpop TINKERPOP-927
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tinkerpop/pull/427.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #427
commit 5f2d2519373e4df3550bc80d2ab6b936366a3ead
Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
Date: 2016-09-19T13:20:50Z
Upload only diffs in `publish-docs.sh`.
GitHub user dkuppitz opened a pull request:
https://github.com/apache/tinkerpop/pull/428
TINKERPOP-927 bin/publish-docs.sh should only upload diffs.
https://issues.apache.org/jira/browse/TINKERPOP-927
VOTE: +1
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/apache/tinkerpop TINKERPOP-927-master
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/tinkerpop/pull/428.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #428
commit 5f2d2519373e4df3550bc80d2ab6b936366a3ead
Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
Date: 2016-09-19T13:20:50Z
Upload only diffs in `publish-docs.sh`.
commit b20921bd4d42a08db6a182b26dd22e4808d837c2
Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
Date: 2016-09-19T15:23:47Z
Merge branch 'TINKERPOP-927' into TINKERPOP-927-master
commit 144e6f08d8bbefa3bf1810cac39a41c2f3b1a21e
Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
Date: 2016-09-19T15:25:05Z
Fixed javadocs file reference.
commit 0100f78c79460152c1e5fd7793bef215e4558311
Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
Date: 2016-09-19T15:26:48Z
Merge branch 'TINKERPOP-927' into TINKERPOP-927-master
commit 120cef66da6afac1ecbb5781fe58d3ca3e1f0129
Author: Daniel Kuppitz <daniel_kuppitz@hotmail.com>
Date: 2016-09-19T23:12:52Z
Fixed temporary file path.
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/428
dopest +1 ever
![](http://migratingtaste.oldeenglish.org/iloverap/threetimesdope-funkydividendremix.jpg)
Github user spmallette commented on the issue:
https://github.com/apache/tinkerpop/pull/427
Didn't test this on tp31 - but i'm sold on #428
VOTE +1
I'd say it would be equally important to generate just one book. Not sure if that's a different ticket or not? I guess we'd have to do such a thing by revising maven profiles to be one profile per book. Not sure if there's a better way.