+Confirmed: All editions of the #Wikipedia online encyclopedia blocked in #Turkey as of 8:00AM local timehttps://t.co/ybFolRmsOs pic.twitter.com/hI9tn4bHe5
— Turkey Blocks (@TurkeyBlocks) April 29, 2017
A main goal of the IPFS Project is improving humanity's access to information. We strongly oppose the censorship of history, of news, of free thought, of discourse, and of compendiums of vital information such as Wikipedia. Free access to information is key to modern human life, to a free society, and to a flourishing culture. We're alarmed by the erosion of civil liberties wherever it occurs, and we want to help people like the citizens of Turkey preserve freedom of information, even in the face of a tightening iron fist. @@ -33,7 +33,7 @@ Upon hearing the news, we revived an effort to put snapshots of Wikipedia on IPF ## Quick Background: IPFS and Content Addressing -**[IPFS](https://ipfs.io) -- the Inter-Planetary File System** -- is a new internet protocol that makes the web faster, safer, and more open. IPFS changes the _addressing_ of information, moving from _location addressing_ to _content addressing_. You can find out more about IPFS at [the IPFS Website](https://ipfs.io) or by [watching this talk](https://www.youtube.com/watch?v=2RCwZDRwk48). +[**IPFS**](https://ipfs.io) **-- the Inter-Planetary File System** -- is a new internet protocol that makes the web faster, safer, and more open. IPFS changes the _addressing_ of information, moving from _location addressing_ to _content addressing_. You can find out more about IPFS at [the IPFS Website](https://ipfs.io) or by [watching this talk](https://www.youtube.com/watch?v=2RCwZDRwk48). _Content addressing_ is a technique to reference files or data by a unique fingerprint derived from the contents of the file or data itself. Content addressing is implemented with [cryptographic hashing](https://simple.wikipedia.org/wiki/Cryptographic_hash_function), so that content addresses are secure, permanent, and derived directly from the content itself. Information systems like IPFS use content addressing to ensure files, websites, and webapps can move around the network and be distributed by any computer securely and with perfect fidelity. This means that the contents of websites like Wikipedia can be moved around and accessed in a peer-to-peer, decentralized way, much like BitTorrent or even email. This works even if access to the specific https://en.wikipedia.org servers is censored. To find out more about content addressing, you can watch [this part of a talk](https://youtu.be/2RCwZDRwk48?t=847) or [this excellent post](https://ipfs.io/ipfs/QmNhFJjGcMPqpuYfxL62VVB9528NXqDNMFXiqN5bgFYiZ1/its-time-for-the-permanent-web.html). @@ -61,7 +61,7 @@ In short, content on IPFS is **harder to attack** and easier to distribute becau **IPFS does not rely on the Certificate Authority System**, so bad or corrupt Certificate Authorities do not impact it. -**You can move content via [sneakernet](https://en.wikipedia.org/wiki/Sneakernet)!** _This is very useful in areas with poor connectivity, due to resource limitations, security reasons, or censorship._ Even if your network is physically disconnected from the rest of the internet, you can write content from IPFS onto USB drives or other external drives, physically move them to computers connected to a new network, and re-publish the content on the new network. Even though you're on a separate network, IPFS will let nodes access the content using the same identifiers in both networks as long as at least one node on the network has that content. +**You can move content via** [**sneakernet**](https://en.wikipedia.org/wiki/Sneakernet)**!** _This is very useful in areas with poor connectivity, due to resource limitations, security reasons, or censorship._ Even if your network is physically disconnected from the rest of the internet, you can write content from IPFS onto USB drives or other external drives, physically move them to computers connected to a new network, and re-publish the content on the new network. Even though you're on a separate network, IPFS will let nodes access the content using the same identifiers in both networks as long as at least one node on the network has that content. **IPFS nodes work hard to find each other on the network** and to reconnect with each other after connections get cut. @@ -93,21 +93,19 @@ Today, we are releasing the first full static snapshot on IPFS of all of https:/ The unique identifier (cryptographic hash) for the snapshot of tr.wikipedia.org from April 30th is: -- Turkish Wikipedia (30 April 2017): [/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html](https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html) +* Turkish Wikipedia (30 April 2017): [/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/](https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/) -This link points to a specific snapshot. It will always point to that specific day's snapshot. To find the most up-to-date snapshot of Wikipedia on IPFS you can use this IPNS link, which will resolve to the latest snapshot whenever we release a new one: - -- Turkish Wikipedia (most recent snapshot, resolved through IPNS): [/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html](https://ipfs.io/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html) +This link points to a specific snapshot. It will always point to that specific day's snapshot. To find the most up-to-date snapshot of Wikipedia on IPFS you can use human-readable links, as noted below. ### Human-readable links For your convenience we have set up a domain name and DNS entry at [tr.wikipedia-on-ipfs.org](http://tr.wikipedia-on-ipfs.org) that will resolve to the current IPFS snapshot. -If you are not able to access ipfs.io or wikipedia-on-ipfs.org, you can use this dnslink URL to access the content -- simply replace `ipfs.io` with the address of any IPFS gateway: +If you are not able to access `ipfs.io`, or `wikipedia-on-ipfs.org`, you can use this [dnslink](https://docs.ipfs.io/concepts/dnslink/) URL to access the content -- simply replace `ipfs.io` with the address of any [IPFS gateway](https://ipfs.github.io/public-gateway-checker/): -- Turkish Wikipedia: [https://ipfs.io/ipns/tr.wikipedia-on-ipfs.org](https://ipfs.io/ipns/tr.wikipedia-on-ipfs.org/wiki/Anasayfa.html) +* Turkish Wikipedia: [https://ipfs.io/ipns/tr.wikipedia-on-ipfs.org/](https://ipfs.io/ipns/tr.wikipedia-on-ipfs.org/wiki/) -Failing that, use the (less readable) IPFS or IPNS links from above. +Failing that, use the (less readable) IPFS links from above. ## How to access Wikipedia on IPFS @@ -117,17 +115,17 @@ We've written a number of tutorials describing the many different ways to access Your main options for accessing the snapshot of Wikipedia are: _(depending on your network, some of these won't work)_ -- **Option**: Use the ipfs.io gateway to access the 30 April 2017 snapshot: https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html -- **Option**: Use the ipfs.io gateway to access the latest version: https://ipfs.io/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html -- **Option**: Connect with the IPFS network over Tor (this is experimental). Read [this tutorial on Tor gateways](https://dweb-primer.ipfs.io/avenues-for-access/lessons/tor-gateways.html). -- **Option**: Install an IPFS node on your computer and access Wikipedia through that node (requires [using the command line](http://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything). _This is the most reliable method because it retrieves the content directly from the IPFS peer-to-peer network)_ - 1. Install IPFS [following these instructions](https://dweb-primer.ipfs.io/install-ipfs/). Use the most recent verison of IPFS: 0.4.9-rc2 or higher if possible. +* **Option**: Use any gateway to access the 30 April 2017 snapshot: https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/ +* **Option**: Use any gateway to access the latest DNSLink version: https://ipfs.io/ipns/tr.wikipedia-on-ipfs.org/wiki/ +* **Option**: Connect to gateway over Tor (this is experimental). Read [this tutorial on Tor gateways](https://dweb-primer.ipfs.io/avenues-for-access/lessons/tor-gateways.html). +* **Option**: Install an IPFS node on your computer and access Wikipedia through that node (requires [using the command line](http://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything). _This is the most reliable method because it retrieves the content directly from the IPFS peer-to-peer network)_ + 1. Install IPFS [following these instructions](https://docs.ipfs.io/install/command-line/). Use the most recent verison of IPFS: 0.4.9-rc2 or higher if possible. 2. Start your IPFS node by running `ipfs daemon` so it can connect to the network. 3. Read the content through your IPFS node's local HTTP gateway by visiting: - - 30 April 2017 snapshot: http://localhost:8080/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html - - latest snapshot: http://localhost:8080/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html + * 30 April 2017 snapshot: http://localhost:8080/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/ + * latest snapshot: http://localhost:8080/ipns/tr.wikipedia-on-ipfs.org/wiki/ -**Coming soon:** We've almost finished creating a web browser extension that will allow you to access IPFS directly from your web browser without installing any additional external tools. Watch https://github.com/ipfs/in-web-browsers for the announcement when we release that browser extension. +**Coming soon:** We've almost finished creating a [web browser extension](https://github.com/ipfs/ipfs-companion#readme) that will allow you to access IPFS directly from your web browser without installing any additional external tools. Watch https://github.com/ipfs/in-web-browsers for the announcement when we release that browser extension. ## How to Mirror Wikipedia on an IPFS Node @@ -135,22 +133,22 @@ If you want to help make this information available, you can install IPFS on you **Prerequisites**: -- Requires [using the command line](http://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything) -- You need to have enough storage space to hold the snapshot(s) you want to serve. - - Turkish snapshot requires 10 GB - - Turkish, Arabic and Kurdish snapshots combined together will require 25 GB - - English snapshot will require 250 GB -- _If possible, use a machine with a public IP address._ If you want to run an ipfs-to-http gateway that lets people access the IPFS content using their web browswer, you need to ensure that your machine can be reached with a public IP address. +* Requires [using the command line](http://lifehacker.com/5633909/who-needs-a-mouse-learn-to-use-the-command-line-for-almost-anything) +* You need to have enough storage space to hold the snapshot(s) you want to serve. + * Turkish snapshot requires 10 GB + * Turkish, Arabic and Kurdish snapshots combined together will require 25 GB + * English snapshot will require 250 GB +* _If possible, use a machine with a public IP address._ If you want to run an ipfs-to-http gateway that lets people access the IPFS content using their web browswer, you need to ensure that your machine can be reached with a public IP address. If you don't have enough storage space to hold full copies of the snapshot(s), you can still run an IPFS gateway so that people can rely on you to retrieve the content from the IPFS network on the fly. **Steps**: -1. Install IPFS [following these instructions](https://dweb-primer.ipfs.io/install-ipfs/). Use the most recent verison of IPFS -- 0.4.9-rc2 or higher if possible. -2. Start your IPFS node by running `ipfs daemon` so it can connect to the network. -3. Pin the snapshot(s) onto your machine - - Pin Turkish Wikipedia: `ipfs pin add QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX` -4. If your machine has a public IP addresses, tell people the address of your gateway. They can use that address to request IPFS content from their web browsers. +1. Install IPFS [following these instructions](https://docs.ipfs.io/install/command-line/). Use the most recent verison of IPFS -- 0.4.9-rc2 or higher if possible. +2. Start your IPFS node by running `ipfs daemon` so it can connect to the network. +3. Pin the snapshot(s) onto your machine + * Pin Turkish Wikipedia: `ipfs pin add QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX` +4. If your machine has a public IP addresses, tell people the address of your gateway. They can use that address to request IPFS content from their web browsers. **Alternative: Using a Sneakernet to Mirror data** @@ -176,10 +174,8 @@ If you would like to repeat this process, adding an updated Wikipedia snapshot o Whenever we add an updated snapshot of Wikipedia to IPFS, we will announce it in a few ways. 1. We will tweet the new snapshot hash on the [@ipfsbot](https://twitter.com/ipfsbot) account along with the hashtag [#WikipediaOnIPFS](https://twitter.com/search?f=tweets&q=WikipediaOnIPFS) -2. We will update this IPNS entry to point to the latest snapshot: - - Turkish Wikipedia: [/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html](https://ipfs.io/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html) - -The DNS entry at [tr.wikipedia-on-ipfs.org](https://tr.wikipedia-on-ipfs.org) is mapped to the IPNS value, so it will also give you the latest available snapshot (if DNS and IPNS are working properly in your area). +2. We will update this DNSLink entry to point to the latest snapshot: + * Turkish Wikipedia: [/ipns/tr.wikipedia-on-ipfs.org/](https://ipfs.io/ipns/tr.wikipedia-on-ipfs.org/) ## Who controls the information @@ -189,13 +185,12 @@ If people start relying on this information over time, it will be important to e ## Review -- You can access a snapshot of Wikipedia through any IPFS gateway - - Turkish Wikipedia (30 April 2017 snapshot): [/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html](https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/Anasayfa.html) -- These are read-only snapshots. Supporting full read-and-write would take further work. -- IPFS links use cryptographic hashes, so you don’t have to worry about spoofing -- The IPNS hashes point to the most recent snapshot - - Turkish Wikipedia (most recent snapshot): [/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html](https://ipfs.io/ipns/QmVH1VzGBydSfmNG7rmdDjAeBZ71UVeEahVbNpFQtwZK8W/wiki/Anasayfa.html) -- If people start relying on this information, we will encourage Wikimedia to take over generating these snapshots -- We are encouraging Wikimedia to publish their own IPNS hash that is always up to date AND is cryptographically signed by Wikimedia -- (if wikimedia updates DNS) if you have access to a gateway that’s outside of Turkey, you will be able to use the convenient path [/ipns/wikipedia.org](https://ipfs.io/ipns/wikipedia.org) instead of using hashes in your IPFS links -- If you want to mirror the data, run an ipfs node and pin the Wikipedia data onto your node +* You can access a snapshot of Wikipedia through any IPFS gateway + * Turkish Wikipedia (30 April 2017 snapshot): [/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/](https://ipfs.io/ipfs/QmT5NvUtoM5nWFfrQdVrFtvGfKFmG7AHE8P34isapyhCxX/wiki/) +* These are read-only snapshots. Supporting full read-and-write would take further work. +* IPFS links use cryptographic hashes, so you don’t have to worry about spoofing +* The DNSLink records point to the most recent snapshot + * Turkish Wikipedia (most recent snapshot): [/ipns/tr.wikipedia-on-ipfs.org/](https://ipfs.io/ipns/tr.wikipedia-on-ipfs.org/) +* If people start relying on this information, we will encourage Wikimedia to take over generating these snapshots +* We are encouraging Wikimedia to publish DNSLink or even IPNS record that is always up to date AND is cryptographically signed by Wikimedia +* If you want to mirror the data, run an ipfs node and pin the Wikipedia data onto your node \ No newline at end of file diff --git a/src/_blog/2021-05-31-distributed-wikipedia-mirror-update.md b/src/_blog/2021-05-31-distributed-wikipedia-mirror-update.md new file mode 100644 index 000000000..eba6c78d4 --- /dev/null +++ b/src/_blog/2021-05-31-distributed-wikipedia-mirror-update.md @@ -0,0 +1,115 @@ +--- +title: Distributed Wikipedia Mirror Update +description: Status update for 2021 Q2, usage instructions, current build process, + and open problems. +author: Marcin Rataj +date: 2021-05-31 +permalink: "/2021-05-31-distributed-wikipedia-mirror-update/" +translationKey: '' +header_image: "/wikipedia-mirrors-2021-q2-1.png" +tags: +- censorship + +--- +* [Distributed Wikipedia Mirror](https://github.com/ipfs/distributed-wikipedia-mirror#readme) and [Kiwix](https://kiwix.org/) projects are happy to the announce **general availability** of **updated** [English](https://en.wikipedia-on-ipfs.org/) and [Turkish](https://tr.wikipedia-on-ipfs.org) mirrors, along with **new languages**: [Myanmar](https://my.wikipedia-on-ipfs.org/), [Arabic](https://ar.wikipedia-on-ipfs.org), [Chinese](https://zh.wikipedia-on-ipfs.org) and [Russian](https://ru.wikipedia-on-ipfs.org). +* A handy, up-to-date list can be found at [ipfs.kiwix.org](ipfs.kiwix.org), and in the [`snapshot-hashes.yml`](https://github.com/ipfs/distributed-wikipedia-mirror/blob/main/snapshot-hashes.yml "snapshot-hashes.yml") manifest. +* The idea of a distributed Wikipedia mirror goes back to 2017, when the IPFS Project created a snapshot of English and Turkish languages and put it on IPFS. To learn why we did it, please read the original [Uncensorable Wikipedia on IPFS](https://blog.ipfs.io/24-uncensorable-wikipedia/) post. +* Below is a short status update with improved usage instructions, current build process, open problems, and future work that could be contributed to the project. + +## Improved access to Wikipedia mirrors + +### User-friendly `ipns://{dnslink}` and public gateways + +Browsers with built-in support for IPFS addresses ([Brave](https://brave.com/brave-integrates-ipfs/), [Opera](https://blog.ipfs.io/2020-03-30-ipfs-in-opera-for-android/), or a regular [Firefox](https://www.mozilla.org/en-US/firefox/new/), [Chromium](https://en.wikipedia.org/wiki/Chromium_(web_browser)) with [IPFS Companion](https://github.com/ipfs/ipfs-companion#readme)) can now load the latest snapshot using [DNSLink](https://docs.ipfs.io/concepts/dnslink/): + +* `ipns://{dnslink}` +* `ipns://en.wikipedia-on-ipfs.org` + +To ensure true P2P transport, offline storage and content integrity, you can run your own IPFS node ([command-line](https://docs.ipfs.io/install/command-line/) or [IPFS Desktop](https://docs.ipfs.io/install/ipfs-desktop/) app) combined with the [IPFS Companion](https://docs.ipfs.io/install/ipfs-companion/) browser extension. You can also use the [Brave browser, which has built-in support for IPFS](https://brave.com/brave-integrates-ipfs/): + +@[youtube](jTDkTQiKzJA) + +When it is not possible to run your own IPFS node, one of the [many public gateways](https://ipfs.github.io/public-gateway-checker/) can be used as a proxy for accessing the mirror. For example: + +* [https://dweb.link/ipns/my.wikipedia-on-ipfs.org](https://dweb.link/ipns/my.wikipedia-on-ipfs.org) +* [https://cf-ipfs.com/ipns/my.wikipedia-on-ipfs.org](https://cf-ipfs.com/ipns/my.wikipedia-on-ipfs.org) + +### Robust and immutable `ipfs://{cid}` + +If DNS resolution is blocked, or a public gateway can't be trusted, accessing the immutable snapshot using underlying cryptographic content identifier ([CID](https://docs.ipfs.io/concepts/content-addressing/)) is advised: + +* `ipfs://{cid}` + +The `{cid}` of a specific mirror can be found in [`snapshot-hashes.yml`](https://github.com/ipfs/distributed-wikipedia-mirror/blob/main/snapshot-hashes.yml), or read from its DNSLink record with`ipfs resolve -r /ipns/en.wikipedia-on-ipfs.org`. At the time of writing this post, the English mirror points at `ipfs://bafybeiaysi4s6lnjev27ln5icwm6tueaw2vdykrtjkwiphwekaywqhcjze` + +Sharing CIDs via [sneakernet](https://en.wikipedia.org/wiki/Sneakernet) is a popular way of routing around DNS issues and censorship. Turkish citizens resorted to that in 2017 when [Turkey blocked Wikipedia](https://en.wikipedia.org/wiki/Block_of_Wikipedia_in_Turkey). History does not repeat itself, but it rhymes: Myanmar started experiencing internet blackouts earlier this year: + +Confirmed: All editions of the #Wikipedia online encyclopedia blocked in #Turkey as of 8:00AM local timehttps://t.co/ybFolRmsOs pic.twitter.com/hI9tn4bHe5
— Turkey Blocks (@TurkeyBlocks) April 29, 2017
+ +To address this critical need, we [created a mirror of Myanmar Wikipedia](https://github.com/ipfs/distributed-wikipedia-mirror/pull/83) and [shared both DNSLink and CID](https://github.com/ipfs/distributed-wikipedia-mirror/pull/83#issuecomment-786021082). + +Confirmed: #Myanmar has blocked all language editions of the Wikipedia online encyclopedia, part of a widening post-coup internet censorship regime imposed by the military junta 📚
— NetBlocks (@netblocks) February 19, 2021
Network data show restriction in effect on major providers.
📰 Report: https://t.co/Jgc20OBk27 pic.twitter.com/qstGEefO4E
+ +## How to help co-hosting this? + +You can run your own IPFS node and co-host a subset of Wikipedia, store a full copy, or even follow collaborative cluster to pull in future updates automatically. + +It is also possible to donate co-hosting costs by pinning specific CID to a remote service. + +### Lazy co-hosting with your own IPFS node + +It is possible to keep a lazy-loaded copy. which does not fetch the entire Wikipedia, but keeps the browsed subset of pages around. + + $ ipfs files cp /ipfs/{cid} /my-wikipedia-snapshot + +One can convert a lazy copy to full one by recursively pinning the [DAG](https://docs.ipfs.io/concepts/glossary/#dag) behind a CID: + + $ ipfs pin add --progress {cid} + +A recursive pin will preload the entire mirror to the local datastore. +Be wary that the English one is far bigger than other ones, and pinning it requires hundreds of gigabytes and may take a very long time. + +The size of a specific mirror can be read with `ipfs files stat /ipfs/{cid}`. + +### Collaborative cluster + +This is an advanced option aimed at server administrators and power users. The `wikipedia` cluster includes all language versions and its size only grows over time. + + $ ipfs-cluster-follow wikipedia run --init wikipedia.collab.ipfscluster.io + +See _Instructions_ at [collab.ipfscluster.io](collab.ipfscluster.io). + +### Donate remote pins + +When co-hosting with your own IPFS node is not possible, one can still help by pinning snapshot CIDs to a remote pinning service. + [Learn how to _work with remote pinning services_](https://docs.ipfs.io/how-to/work-with-pinning-services/). + +## How is a mirror built? + +The current setup relies on [Wikipedia snapshots in the ZIM format](https://download.kiwix.org/zim/wikipedia/) produced by the [Kiwix](https://kiwix.org/) project. + +We don't have a web-based reader of ZIM archives (yet – more in the next section), and the way we produce a mirror is an elaborate, time-consuming process: + +1. Unpacking ZIM archive with [openzim/zim-tools](https://github.com/openzim/zim-tools) +2. Adjusting HTML/CSS/JS to fixup unpacked form +3. Import snapshot to IPFS +4. Include original ZIM inside of unpacked IPFS snapshot + +While this works, the need for unpacking and customizing the snapshot makes it difficult to reliably produce updates. And including the original ZIM for use with [Kiwix offline reader](https://www.kiwix.org/en/kiwix-reader), partially duplicates the data. + +We would love to mirror more languages, and increase the update cadence, but for that to happen we need to remove the need for unpacking ZIM archives. + +We will be looking into putting [all ZIMs from Kiwix](https://download.kiwix.org/zim/wikipedia/) on IPFS and archiving them for long term storage on [Filecoin](https://filecoin.io/) as part of [farm.openzim.org ](https://farm.openzim.org )pipeline. + +## Help Wanted and Open Problems + +If you are still reading this, there is a high chance you are interested in improving the way the distributed Wikipedia mirror works. + +Below are areas that could use a helping hand, and ideas looking for someone to explore them. + +* **Search.** There's no search function currently. Leveraging the index present in ZIM, or building a DAG-based search index optimized for use in web browsers would make existing mirrors more useful. See [distributed-wikipedia-mirror/issues/76](https://github.com/ipfs/distributed-wikipedia-mirror/issues/76). +* **Web-based ZIM reader.** The biggest impact for the project would be to create a web-based reader capable of browsing original ZIM archives without the need for unpacking them, nor installing any dedicated software. Want to help make it a reality? See [kiwix-js/issues/659](https://github.com/kiwix/kiwix-js/issues/659) +* **Improving the way ZIM is represented on IPFS.** When we store an original ZIM on IPFS, the DAG is produced by `ipfs add --cid-version 1`. This works fine, but with additional research on customizing DAG creation, we may improve deduplication and speed when doing range requests for specific bytes. There are different stages to explore here: if any of them sounds interesting to you, please comment in [distributed-wikipedia-mirror/issues/42](https://github.com/ipfs/distributed-wikipedia-mirror/issues/42). + * Stage 1: Invest some time to benchmark parameter space to see if low hanging fruits exists. + * Stage 2: Create a DAG builder that understands ZIM format and maximizes deduplication of image assets by representing them as sub-DAGs with dag-pb files. + * Stage 3: Research augmenting or replacing ZIM with [IPLD](https://ipld.io/). How can we maximize block deduplication across all snapshots and languages? How would an IPLD-based search index work? diff --git a/src/_blog/decentralizing-infrastructure-during-scaling-ethereum.md b/src/_blog/decentralizing-infrastructure-during-scaling-ethereum.md new file mode 100644 index 000000000..88be113d9 --- /dev/null +++ b/src/_blog/decentralizing-infrastructure-during-scaling-ethereum.md @@ -0,0 +1,76 @@ +--- +title: Decentralizing infrastructure during Scaling Ethereum +description: 'Announcing the winners from ETHGlobal’s Scaling Ethereum hackathon event.' +author: PL Hackathon Team +date: 2021-05-26 +permalink: "/2021-05-26-scaling-ethereum-recap/" +translationKey: 2021-05-26-scaling-ethereum-recap +header_image: "/2021-04-14-cardheader-scaling-ethereum-1459531d.png" +tags: +- recap +- hackathon + +--- +An endeavour to collaborate on creating a powerful foundation for the future of the Ethereum community, [Scaling Ethereum](https://scaling.ethglobal.co/), has come to a close, resulting in some innovative and inspiring solutions. The IPFS and Filecoin projects, alongside the ETHGlobal team, sponsored this 4-week hackathon event. + +We put up $7500 in Filecoin across various scaling areas to award to those brave and intrepid souls willing to rise to the challenge! + +## Our challenges + winners + +### 💮 NFT Metadata Updater + +Mo minting, mo problems. Best method for managing signed permissioned and timestamped updates to NFT metadata stored on IPFS via any L2 chain. + +Winner: [Nama Finance]() + +A decentralized one-stop-shop data manager for meta creators. + +@[youtube](7MdRDDK-T3M) + +### ⛵️ Sailing the Sea of CIDs + +You’re publishing everything to IPFS by default now, but… dude, where’s my stuff? Best tool, service, website or library for working with large collections of IPFS CIDs. + +Winner: [Hashmark]() + +Hashmark is like Instapaper, but for NFTs. It’s the simplest way to view and bookmark NFTs. + +@[youtube](9dccga2yQxg) + +Winner: [IPFS-CIDSail](https://hack.ethglobal.co/showcase/ipfs-cidsail-recHTKp8X5ddxI4FK) + +A command line utility that helps you find/check/keep track of your files published on IPFS. + +@[youtube](Uzt0bA6s444) + +Winner: [HydroFile](https://hack.ethglobal.co/showcase/hydrofile-recobwnZCxJ50qpoo) + +An automated way to keep track of IPFS CIDs – sail the seas of CIDs! + +@[youtube](zuRjzrqozUs) + +Winner: [DVHDB](https://hack.ethglobal.co/showcase/vhddb-video-hash-distributed-data-base-recIXMgkcMXnpSabe) - Distributed Video Hash Database + +A collaborative and distributed database of video hashes. + +@[youtube](hcHhkKUsqSk) + +Congrats to all of the projects who participated during Scaling Ethereum, and thank you to our prize winners for their great work! + +See all of the Scaling Ethereum projects in [the official showcase](https://showcase.ethglobal.co/scaling). + +## Talks + workshops + +The Protocol Labs team gave two talks at the event: + +### Pre-event talk: Deploying Web3: IPFS Architectures, Topologies and Menageries with Dietrich Ayala + +@[youtube](13rHbWCYNWA) + +### SNARKs Summit: Scaling zkSNARKSs to meet the demands of Filecoin with Friedel Ziegelmayer + +@[youtube](GRaR8N380h0) + +## Want to get involved in a future hackathon? + +Follow us on Twitter at [@IPFS](https://twitter.com/ipfs) or [@Filecoin](https://twitter.com/filecoin). You can also subscribe to the [IPFS weekly newsletter](https://ipfs.us4.list-manage.com/subscribe?u=25473244c7d18b897f5a1ff6b&id=cad54b2230), or the [Filecoin monthly](https://filecoin.us16.list-manage.com/subscribe?u=a1dfb670c4f1fb042e82a1f1d&id=cf409026a0). Or, join us in chat in the [Filecoin Slack](http://filecoin.io/slack) or the [IPFS Discord](https://discord.com/invite/vZTcrFePpt). We’ll see you there! 👋 diff --git a/src/_blog/events.md b/src/_blog/events.md index d8dd370a2..df3bf9895 100644 --- a/src/_blog/events.md +++ b/src/_blog/events.md @@ -4,6 +4,14 @@ type: Event sitemap: exclude: true data: +- title: Web3 Weekend + date: 2021-05-28 + publish_date: 2021-05-24T12:00:00+00:00 + path: https://web3.ethglobal.co/ + card_image: "/2021-05-24-cardheader-web3-weekend.png" + tags: + - community + - hackathon - title: 'DI2F: Decentralizing the Internet with IPFS and Filecoin' date: 2021-06-21 publish_date: diff --git a/src/assets/2021-05-24-cardheader-web3-weekend.png b/src/assets/2021-05-24-cardheader-web3-weekend.png new file mode 100644 index 000000000..b9dcaa5f2 Binary files /dev/null and b/src/assets/2021-05-24-cardheader-web3-weekend.png differ diff --git a/src/assets/wikipedia-mirrors-2021-q2-1.png b/src/assets/wikipedia-mirrors-2021-q2-1.png new file mode 100644 index 000000000..b3e4c0b59 Binary files /dev/null and b/src/assets/wikipedia-mirrors-2021-q2-1.png differ diff --git a/src/assets/wikipedia-mirrors-2021-q2.png b/src/assets/wikipedia-mirrors-2021-q2.png new file mode 100644 index 000000000..d49f58b88 Binary files /dev/null and b/src/assets/wikipedia-mirrors-2021-q2.png differIn response to ongoing internet restrictions / censorship in Myanmar, @Wikipedia in MY is now on @IPFS:https://t.co/trt0AbEMuW
— dietrich (@dietrich) February 25, 2021
Huge props to @playingwithsid who proposed it, & coordinated w/ native speakers.
Epic implementation effort by @lidelOrg & Kelson of @KiwixOffline!