-
Notifications
You must be signed in to change notification settings - Fork 37
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge branch 'main' into feat/companion-mv3-update
* main: (47 commits) fix: update acronym link and add dAppling links Update major-improvements-to-omnilingo.md Update 2023-12-introducing-dappling.md Update 2023-12-introducing-dappling.md Update major-improvements-to-omnilingo.md Add dAppling Intro Post Update major-improvements-to-omnilingo.md Update major-improvements-to-omnilingo.md Create major-improvements-to-omnilingo.md Update newscoverage.md Update newscoverage.md add videos (#653) Fix newsletter form (#651) Update newsletter-199.md Update newsletter-199.md Create newsletter-199.md Update README.md fix-permalink chore: add Kubo release note (#650) Update 2023-11-introducing-nabu.md ... Signed-off-by: Nishant Arora <[email protected]>
- Loading branch information
Showing
14 changed files
with
645 additions
and
58 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -5,77 +5,51 @@ | |
> | ||
<div class="flex-shrink lg:max-w-sm xl:max-w-xl mb-4 lg:mb-0"> | ||
<h2 class="type-h2">Stay informed</h2> | ||
<p class="mt-2 mr-2"> | ||
<p class="mt-2 mb-6 mr-2"> | ||
Sign up for the IPFS newsletter (<router-link | ||
:to="latestWeeklyPost ? latestWeeklyPost.path : ''" | ||
class="text-blueGreenLight hover:underline" | ||
>example</router-link | ||
>) for the latest on releases, upcoming developments, community events, | ||
and more. | ||
</p> | ||
<a target="_blank" href="https://ipfs.fyi/newsletter"> | ||
<button | ||
type="button" | ||
class=" | ||
px-3 | ||
py-2 | ||
text-white | ||
bg-blueGreen | ||
font-semibold | ||
rounded | ||
hover:bg-blueGreenScreen | ||
transition | ||
duration-300 | ||
" | ||
> | ||
Sign up | ||
</button> | ||
</a> | ||
</div> | ||
<form | ||
|
||
<div | ||
id="mc-embedded-subscribe-form" | ||
name="mc-embedded-subscribe-form" | ||
class="flex lg:justify-end max-w-lg xl:w-2/5" | ||
action="https://ipfs.us4.list-manage.com/subscribe/post?u=25473244c7d18b897f5a1ff6b&id=cad54b2230" | ||
method="post" | ||
target="_blank" | ||
@submit="subscribeClick" | ||
> | ||
<div id="mc_embed_signup_scroll" class="grid gric-col-2 w-full"> | ||
<div class="fields flex flex-col sm:flex-row col-start-1 col-span-2"> | ||
<input | ||
id="mce-EMAIL" | ||
v-model="email" | ||
required | ||
type="email" | ||
aria-label="Email Address" | ||
class="flex-grow text-black p-2 rounded" | ||
placeholder="[email protected]" | ||
name="EMAIL" | ||
/> | ||
<div class="sm:ml-4 sm:pt-0 pt-2"> | ||
<input | ||
id="mc-embedded-subscribe" | ||
type="submit" | ||
value="Subscribe" | ||
name="subscribe" | ||
class="p-2 text-white font-semibold bg-blueGreen hover:bg-blueGreenScreen transition duration-300 rounded cursor-pointer w-full" | ||
/> | ||
</div> | ||
</div> | ||
<label class="pt-2 col-start-1 col-span-2" for="gdpr_28879"> | ||
<input | ||
id="gdpr_28879" | ||
type="checkbox" | ||
class="" | ||
required | ||
name="gdpr[28879]" | ||
value="Y" | ||
/><span class="pl-2">Please send me the newsletter</span> | ||
</label> | ||
</div> | ||
<div id="mergeRow-gdpr"> | ||
<div style="position: absolute; left: -5000px" aria-hidden="true"> | ||
<input | ||
type="text" | ||
name="b_25473244c7d18b897f5a1ff6b_cad54b2230" | ||
tabindex="-1" | ||
value="" | ||
/> | ||
<div class="sm:ml-4 sm:pt-0"></div> | ||
</div> | ||
<!-- real people should not fill this in and expect good things - do not remove this or risk form bot signups--> | ||
</div> | ||
</form> | ||
</div> | ||
</div> | ||
</template> | ||
|
||
<script> | ||
import { mapState } from 'vuex' | ||
import countly from '../../util/countly' | ||
export default { | ||
name: 'NewsletterForm', | ||
props: {}, | ||
|
@@ -85,10 +59,6 @@ export default { | |
computed: { | ||
...mapState('appState', ['latestWeeklyPost']), | ||
}, | ||
methods: { | ||
subscribeClick() { | ||
countly.trackEvent(countly.events.NEWSLETTER_SUBSCRIBE) | ||
}, | ||
}, | ||
methods: {}, | ||
} | ||
</script> |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,130 @@ | ||
--- | ||
title: 'Introducing Nabu: Unleashing IPFS on the JVM' | ||
description: 'Learn about a new fast IPFS implementation in Java' | ||
author: Ian Preston | ||
date: 2023-11-07 | ||
permalink: '/2023-11-introducing-nabu/' | ||
header_image: '/nabu-banner-2023.png' | ||
tags: | ||
- 'ipfs' | ||
- 'nabu' | ||
- 'bitswap' | ||
--- | ||
|
||
Greetings from the [Peergos](https://peergos.org) team! We are thrilled to unveil what we've been working on this year: [Nabu](https://github.com/peergos/nabu) β our sleek and versatile Java implementation of IPFS. Named after the ancient Mesopotamian god of literacy, rational arts, and wisdom, Nabu makes decentralised data storage and retrieval available to the large JVM ecosystem. It's now *production ready*, as we are using it in Peeergos - a decentralised, secure file storage, sharing and social network. | ||
|
||
## Introducing Nabu: Empowering Java with IPFS Magic | ||
At its core, Nabu is a minimal IPFS implementation for storing and retrieving data blocks over the libp2p protocol. But we didn't stop there β we've also added a touch of innovation with features like authed bitswap. This addition enables the creation of private data blocks, accessible only to those with authorized permissions. Intrigued? Dive into the finer details of this innovation in our dedicated post on [authed bitswap](https://peergos.org/posts/bats). | ||
|
||
Our journey in crafting Nabu involved the implementation of additional libp2p protocols, including: | ||
|
||
* Kademlia (including IPNS): The very backbone of IPFS, aiding in the discovery of blocks and their owners. | ||
* Bitswap + Auth Extension: A protocol that facilitates the exchange of data blocks. | ||
|
||
We built upon the solid foundation of [jvm-libp2p](https://github.com/libp2p/jvm-libp2p). As we delved deeper, we realized the need to implement several crucial components. These include the [yamux muxer](https://github.com/libp2p/jvm-libp2p/tree/develop/libp2p/src/main/kotlin/io/libp2p/mux/yamux), the [TLS security provider](https://github.com/libp2p/jvm-libp2p/blob/develop/libp2p/src/main/kotlin/io/libp2p/security/tls/TLSSecureChannel.kt) (complete with ALPN early muxer negotiation), and a substantial portion of a quic transport (still a work in progress). While much of this effort started in a fork, we collaborated with [Consensys](https://consensys.io) to upstream our contributions into the main project which has now released [v1.0.0](https://github.com/libp2p/jvm-libp2p/releases/tag/1.0.0) as a result. This is used in [Teku](https://github.com/ConsenSys/teku), a Java Ethereum 2 implementation. | ||
|
||
[<img src="../assets/nabu/modules.png" width="500" height="300"/>](../assets/nabu/modules.png) | ||
|
||
Nabu's API empowers developers with the following methods: | ||
* [id](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-id) | ||
* [version](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-version) | ||
* [block/get](https://github.com/Peergos/nabu/blob/master/src/main/java/org/peergos/BlockService.java#L12) | ||
* [block/put](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-block-put) | ||
* [block/rm](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-block-rm) | ||
* [block/stat](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-block-stat) | ||
* [block/has](https://github.com/Peergos/nabu/blob/master/src/main/java/org/peergos/blockstore/Blockstore.java#L25) | ||
* [refs/local](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-refs-local) | ||
* [bloom/add](https://github.com/Peergos/nabu/blob/master/src/main/java/org/peergos/blockstore/Blockstore.java#L37) | ||
* [dht/findprovs](https://docs.ipfs.tech/reference/kubo/rpc/#api-v0-dht-findprovs) | ||
|
||
Most of these functions align with [Kubo](https://github.com/ipfs/kubo), but we've added block/has, which is a much more efficient way to ask if we have a block or not, as well as bloom/add which is useful if you are adding blocks to the blockstore externally (typically with multiple servers and S3 blockstore, and using a bloom filter). In addition we've added a few extra optional parameters to block/get, which you'll hear more about in the Performance section below. | ||
|
||
## Unique Features | ||
|
||
Nabu boasts some distinctive features that simplify building on IPFS: | ||
|
||
* P2P HTTP Proxy: This feature facilitates HTTP requests to listening peers, encrypting communication over libp2p streams. Bid farewell to the complexities of TLS certificate authorities and DNS. | ||
* Built-in S3 Blockstore: Seamlessly integrate external blockstores like S3. | ||
* [Infini-Filter](https://dl.acm.org/doi/10.1145/3589285): A bloom filter replacement that offers infinite expandability. | ||
* Peer-Specific Block Retrieval: Nabu empowers developers to fetch blocks from specific peers, streamlining data retrieval and improving privacy (See Performance section below). | ||
|
||
Let's shed some light on the first of these gems β the P2P HTTP proxy. A component we initially implemented in [Kubo in 2018](https://peergos.org/posts/dev-update#Decentralization) (behind an experimental flag), this feature introduces a new gateway endpoint with paths in the format: | ||
|
||
**/p2p/$peerid/http/** | ||
|
||
Its function is simple yet transformative: it proxies incoming HTTP requests to the specified $peerid while trimming the preceding "/p2p/$peerid/http" path. On the other end, the setup forwards incoming requests to a designated endpoint. This paradigm grants the convenience of traditional HTTP-based architecture, sans the complexities of DNS and TLS certificate authorities. By addressing the node using its public key, secure connections become effortlessly achievable. The diagram below illustrates how we use this proxy in Peergos. | ||
|
||
[<img src="../assets/nabu/p2p-http-proxy.png" width="640" height="340"/>](../assets/nabu/p2p-http-proxy.png) | ||
|
||
For a simpler example of using this, see our single file demo [chat app](https://github.com/Peergos/nabu-chat/blob/main/src/main/java/chat/Chat.java). | ||
|
||
## Performance | ||
### Faster, more private block retrieval | ||
Drawing from experience, we recognized the inefficiency of requesting every single block from the DHT or connected peers. This practice leads to excessive bandwidth consumption and sluggish content retrieval. Enter our solution: a new optional parameter,"peers" in block/get allowing retrieval from pre-specified peer IDs. In cases of unreachability, a DHT lookup through dht/findprovs api serves as a fallback option. This design of taking a set of peerids that you want to ask for blocks, encourages users to design their programs to route at a higher level than blocks, improving speed, bandwidth usage and privacy. Many apps will know which peers they want to retrieve data from in advance, and with this parameter they can massively reduce bandwidth and speed up retrieval. The motto here is "Route your peers, not your data". In Peergos, for example, given a capability to a file we can lookup the file owner's home server (specifically its peer id) and directly send bitswap requests there, so we only need to fallback to DHT lookups if their home server is unreachable. | ||
|
||
### Reduced bandwidth and CPU usage | ||
We believe that *providing* (announcing to the DHT that you have a given cid) every single block of data you have does not scale, This is because the number of DHT lookups and provide calls increases with the amount of data you are storing. The issues trying to scale this have been [documented](https://blog.ipfs.tech/2023-09-amino-refactoring/#making-reprovides-to-amino-lightning-fast). Compare this to bittorrent, which has been around much longer and has a much larger DHT, but where providing doesn't scale with the amount of data in a torrent and idle bandwidth usage is much lower. For this reason, we've made providing blocks in Nabu optional, and disabled it in Peergos (unless you are running a mirror). | ||
|
||
This leads us to the next optimisation, enabled by only sending block requests to peers we think have the data. In Kubo, bitswap will broadcast block wants to all connected peers (typically in the 100s). This is both a privacy issue and a bandwidth hog as it means joining the main IPFS DHT is very resource intensive. Nabu has an option to block such aggressive peers that flood us with requests for blocks we don't have. With this option enabled, the incoming idle bandwidth usage is reduced by 10X. | ||
|
||
### Benchmark | ||
We benchmarked Nabu against a real-world dataset β the Peergos PKI β consisting of a [CHAMP](https://blog.acolyer.org/2015/11/27/hamt/) structure with six layers, 6000 blocks, and a total size of ~2 MiB. The results speak volumes: while standard Kubo took 120 seconds to retrieve this dataset using the pin command, Nabu accomplished the task in a mere 5 seconds. And, this was achieved without any significant optimization or parallelisation, leaving much room for further enhancement. | ||
|
||
[<img src="../assets/nabu/nabu-speed.png" width="604" height="340"/>](../assets/nabu/nabu-speed.png) | ||
|
||
## Compatibility | ||
|
||
Ensuring seamless integration, we subjected Nabu to a suite of interoperability tests against all libp2p implementations, including go-libp2p, rust-libp2p, js-libp2p, and nim-libp2p across historical versions. The results of these tests are documented [here](https://github.com/libp2p/test-plans/actions/runs/5671451848/attempts/1#summary-15368587233). Some of the results are below. | ||
|
||
[<img src="../assets/nabu/nabu-interop.png" width="808" height="340"/>](../assets/nabu/nabu-interop.png) | ||
|
||
## Bringing Nabu to Life: Integration and Usage | ||
Getting started with Nabu is simple. Choose between utilizing it through the HTTP API or embedding it directly into your process. Here's a compilable example of the embedding process in Java: | ||
```java | ||
List<MultiAddress> swarmAddresses = List.of(new MultiAddress("/ip6/::/tcp/4001")); | ||
List<MultiAddress> bootstrapAddresses = List.of(new MultiAddress("/dnsaddr/bootstrap.libp2p.io/p2p/QmQCU2EcMqAqQPR2i9bChDtGNJchTbq5TbXJJ16u19uLTa")); | ||
BlockRequestAuthoriser authoriser = (cid, block, peerid, auth) -> CompletableFuture.completedFuture(true); | ||
HostBuilder builder = new HostBuilder().generateIdentity(); | ||
PrivKey privKey = builder.getPrivateKey(); | ||
PeerId peerId = builder.getPeerId(); | ||
IdentitySection identity = new IdentitySection(privKey.bytes(), peerId); | ||
boolean provideBlocks = true; | ||
|
||
SocketAddress httpTarget = new InetSocketAddress("localhost", 10000); | ||
Optional<HttpProtocol.HttpRequestProcessor> httpProxyTarget = | ||
Optional.of((s, req, h) -> HttpProtocol.proxyRequest(req, httpTarget, h)); | ||
|
||
EmbeddedIpfs ipfs = EmbeddedIpfs.build(new RamRecordStore(), | ||
new FileBlockstore(Path.of("/home/alice/ipfs")), | ||
provideBlocks, | ||
swarmAddresses, | ||
bootstrapAddresses, | ||
identity, | ||
authoriser, | ||
httpProxyTarget | ||
); | ||
ipfs.start(); | ||
|
||
List<Want> wants = List.of(new Want(Cid.decode("zdpuAwfJrGYtiGFDcSV3rDpaUrqCtQZRxMjdC6Eq9PNqLqTGg"))); | ||
Set<PeerId> retrieveFrom = Set.of(PeerId.fromBase58("QmVdFZgHnEgcedCS2G2ZNiEN59LuVrnRm7z3yXtEBv2XiF")); | ||
boolean addToLocal = true; | ||
List<HashedBlock> blocks = ipfs.getBlocks(wants, retrieveFrom, addToLocal); | ||
byte[] data = blocks.get(0).block; | ||
``` | ||
|
||
If you want a working example app you can fork, have a look at our [chat example](https://github.com/Peergos/nabu-chat). This is a simple CLI app where two users exchange peerid (out of band) and then connect and send messages via p2p http requests, which are printed to the console. | ||
|
||
## Future plans | ||
We still have lots planned for Nabu including the following: | ||
* NAT traversal with circuit-relay-v2, dcutr and AutoRelay | ||
* mDNS peer discovery | ||
* Android compatibility and demo app | ||
* Quic integration | ||
|
||
## Gratitude and Acknowledgments | ||
None of this would have been possible without the support of the [IPFS Implementations Fund](https://arcological.xyz/#ipfs-pool). We extend our heartfelt thanks for making this endeavor a reality. | ||
|
||
## Experience Nabu Today! | ||
We invite you to embark on an exploration of Nabu's capabilities. Feel free to give it a whirl, and we eagerly await your feedback and suggestions for improving Nabu. The easiest route is to open an issue on the github repo. | ||
|
||
[Discover Nabu on GitHub](https://github.com/peergos/nabu) and unlock a world of decentralized possibilities. |
42 changes: 42 additions & 0 deletions
42
src/_blog/2023-11_connect-with-us-in-istanbul-and-prague.md
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,42 @@ | ||
--- | ||
title: 'Connect with us in Istanbul and Prague' | ||
description: 'Connect with the PL IPFS Implementers in Istanbul and Prague for DevConnect and DCxPrague! We want to hear from IPFS users to shape our 2024 plans.' | ||
author: Cameron Wood | ||
date: 2023-11-06 | ||
permalink: '/2023-11-connect-in-istanbul-and-prauge' | ||
header_image: '' | ||
tags: | ||
- 'ipfs' | ||
- 'kubo' | ||
- 'helia' | ||
- 'event' | ||
- 'community' | ||
--- | ||
|
||
Hello, IPFS enthusiasts and users! We want to connect with you and hear your thoughts as we shape the future of IPFS for 2024. Your input is invaluable in guiding our efforts, so we're inviting you to meet with us in Istanbul and Prague at two exciting events: DevConnect / IPFS Connect in Istanbul πΉπ· and DCxPrague in Prague π¨πΏ. | ||
|
||
|
||
## π Who We Are: The PL IPFS Implementers and Network Infrastructure Operators | ||
|
||
We are the PL IPFS implementers and network infrastructure operators, working on projects like Kubo, Helia, and managing the IPFS.io gateway. Our goal is to create a better IPFS ecosystem, and your insights are a crucial part of this journey. | ||
|
||
|
||
## π We Want to Hear from You | ||
|
||
Your input matters! We would be thrilled to connect with as many of our current and prospective users as possible during these upcoming events. Your thoughts and experiences will help us understand your needs and use cases, ultimately guiding our plans for 2024. | ||
|
||
|
||
## π We're Eager to Listen | ||
|
||
Are you planning to attend any of these events? If so, we would love to connect with you and learn more about your experiences with IPFS. Whether you have feedback, insights, or simply want to share your thoughts, we're all ears. Your feedback will help us figure out how to make the most of our time and resources for the IPFS community. | ||
|
||
|
||
## β How Can You Get Involved? | ||
|
||
If you're interested in sharing your thoughts and connecting with us during these events, please fill out [this form](https://forms.gle/CxUQPsEUg2CGkLgh6). We're eager to schedule time to meet with you to discuss your current IPFS challenges, needs, and hopes. | ||
|
||
<br /> | ||
<a href="https://forms.gle/CxUQPsEUg2CGkLgh6" class="cta-button"> Fill out this form to connect</a> | ||
|
||
|
||
π Thank You! |
Oops, something went wrong.