One of the bravest things I did this week was to go to PingCAP's #sig-migrate to self-market go-storage. The reason for this is that I saw that the Write to cloud storage (S3, GCS) natively feature of pingcap/dumpling is not yet implemented, so I wanted to take advantage of this feature to bring go-storage to the market. We wanted to use this feature to bring go-storage in. After some communication, we decided to discuss it at the community meeting this week, and I wrote a Proposal: Use aos-dev/go-storage to replace storage.ExternalStorage. In the meeting, everyone mentioned some of their concerns, and then I responded to them one by one, and finally the Proposal was unanimously approved.

The PingCAP students at the conference were concerned with the following points.

• Compatibility: they expect that the current command line and configuration items must be compatible regardless of the implementation (this is not really related to go-storage, it's a dumpling implementation issue)
• functionality: they are concerned about whether go-storage is implemented or planned to implement their existing functionality (for the dumpling project, encryption-related functionality)
• Response time: The community is also concerned about response time, and they asked the project maintainers about response time for new features and bug fixes
• Documentation: The community expects go-storage to provide documentation on platform compatibility
• Licensing: They are very concerned about what protocol go-storage is licensed under (Apache 2.0), so that in the worst case they can Fork their own maintenance
• Testing: They expect go-storage to have a good test coverage
• Performance: they expect no more than 10% performance regression

This was actually the first time go-storage was used externally and the first time we received feedback from users.

• You can see that the API, which took a lot of time to refactor and design, is not the main focus of the users, as long as it meets the needs of their projects, the API is secondary to how good it looks. In other words, a beautiful API is useless if it does not achieve the functionality they want.
• What I thought was small and could be done in a few minutes was the main focus of users: for example, if go-storage wanted to support server-side encryption, all it needed to do was to add a new parameter, but without this feature, users would never have accepted to use go-storage.
• Another takeaway is that users don't actually want to maintain these repository docking themselves, and if go-storage can do this well, then users are actually happy to make the replacement.

So I've adjusted some of the next priorities for go-storage.

• Implementing Multiparter and Appender: The previous plan was to leave this part of the work to the community, but now it seems like a paradox. go-storage can't be adopted at scale until these core features are implemented; and until go-storage is adopted at scale, there can't be users to help implement these before go-storage is adopted. So we need to do the work ourselves.
  • Implement Appender Support
  • Implement Multipart Support
• Implement Server Side Encrypt: pingcap/dumpling is by far the largest user in our community, and their needs are the number one need, so we need to implement SSE support first
  • Implement SSE support
• Improve the test: The previous integration test only covered the common interfaces of Storager, the next will cover the interfaces of Multiparter and Appender, etc.
  • storage: Implement multipart tests
• Improve the documentation: I always thought that there was no hurry about the documentation, but now I find that no user will use the documentation if it is not written, so I am also improving the documentation
  • docs: Add docs for operations and pairs

Adoption by pingcap/dumpling is only the first step; after dumpling completes its migration, we will also implement rs-storage to bring this storage abstraction to the rust platform so that we can replace the storage implementation in pingcap/br. If adopted by br, we go-storage will be part of the tidb ecosystem and can grow along with the tidb project ~~ (start dreaming)~~.

If your application is developed in golang and you have the need to interface with various storage platforms, please feel free to come to our Matrix channel and talk to us at

• English
• 中文

See you next week!

Translated via DeepL with a bit modification.