Docs-exporter
Documents-to-Wiki Exportation
This is a quick-and-dirty script which I am using to convert the documentation pages from the main I2P web site hosted at https://geti2p.net and http://i2p-projekt.i2p into wiki format, so that they can be put on the wiki. What it does it takes a list of urls(a file it expects called "urls.list"), loops over them, looks for sections our website template creates and removes them, tidies the result, then turns what's left into a wiki page. When using this, take care to preserve the same directory structure as on the web site, to ensure that links stay alive by being in the expected place and to preserve our ability to provide localized documentation via this wiki(However, since I am primarily an English speaker, I have chosen to start by exporting English documentation as seen in the example urls.list). So for example, when exporting the SAMv3 documentation, don't use: http://wiki.i2p-projekt.i2p/wiki/index.php/SAMv3, use http://wiki.i2p-projekt.i2p/wiki/index.php/En/docs/api/samv3. If you're careful, most of the website docs can be exported to the wiki automatically.
#! /usr/bin/env sh
for x in $(cat urls.list); do
filename=$(basename $(echo $x | sed 's|https://||g'))
if [ ! -f "wiki/$filename.md" ]; then
wget "$x" -O "wiki/$filename-pre.html"<>
grep -A 10000 '<div id="content">' "wiki/$filename-pre.html" | \
grep -B 10000 '<div id="footermenu" class="first">' | \
sed 's|<div id="footermenu" class="first">||g' | \
tidy --as-xhtml --drop-empty-elements no --input-xml --tidy-mark no -indent --indent-spaces 4 -wrap 0 --new-blocklevel-tags article,header,footer --new-inline-tags video,audio,canvas,ruby,rt,rp --break-before-br yes --sort-attributes alpha --vertical-space yes | \
tee "wiki/$filename.html"
pandoc "wiki/$filename.html" -o "wiki/$filename.wiki"
sed -i '1d' "wiki/$filename.wiki"
fi
done
This is an example urls.list for export.
https://geti2p.net/en/about/glossary https://geti2p.net/en/about/media https://geti2p.net/en/about/performance https://geti2p.net/en/comparison/tor https://geti2p.net/en/comparison/freenet https://geti2p.net/en/comparison/other-networks https://geti2p.net/en/docs/how/intro https://geti2p.net/en/docs/how/tech-intro https://geti2p.net/en/docs/how/threat-model https://geti2p.net/en/docs/how/garlic-routing https://geti2p.net/en/docs/how/network-database https://geti2p.net/en/docs/how/tunnel-routing https://geti2p.net/en/docs/how/peer-selection https://geti2p.net/en/docs/how/cryptography https://geti2p.net/en/docs/how/elgamal-aes https://geti2p.net/en/docs/api/i2ptunnel https://geti2p.net/en/docs/api/socks https://geti2p.net/en/docs/api/samv3 https://geti2p.net/en/docs/api/bob https://geti2p.net/en/docs/api/streaming https://geti2p.net/en/docs/api/datagrams https://geti2p.net/en/docs/api/i2pcontrol https://geti2p.net/en/docs/applications/supported https://geti2p.net/en/docs/applications/bittorrent https://geti2p.net/en/docs/protocol https://geti2p.net/en/docs/protocol/i2cp https://geti2p.net/en/docs/protocol/i2np https://geti2p.net/en/docs/transport https://geti2p.net/en/docs/transport/ntcp https://geti2p.net/en/docs/transport/ssu https://geti2p.net/en/docs/tunnels/implementation https://geti2p.net/en/docs/tunnels/unidirectional https://geti2p.net/en/docs/tunnels/old-implementation https://geti2p.net/en/docs/naming https://geti2p.net/en/docs/plugins https://geti2p.net/en/docs/reseed https://geti2p.net/en/docs/api/i2ptunnel https://geti2p.net/en/docs/api/socks https://geti2p.net/en/docs/api/samv3 https://geti2p.net/en/docs/api/bob https://geti2p.net/en/docs/api/streaming https://geti2p.net/en/docs/api/datagrams https://geti2p.net/en/docs/api/i2pcontrol https://geti2p.net/en/docs/api/i2pcontrol