Release 0.9 ("Astrakhan")

2015-12-30

Hello, long time no see. It's been over a year since our release, but we have fun stuff to share in our ninth release, named after the Russian city of Astrakhan.

For this release we have mainly focused on ease of use and ease of installation for new users. Recognizing that most users don't have their own servers, we've been working on making it easier to run Camlistore on cloud providers. Initially we've targeted the Google Cloud Platform and Scaleway, making sure Camlistore is well-adapted to those environments. For users wishing to use Google Cloud Platform, we now provide a web-based launcher for users to create their own Camlistore instances:

camlistore.org/launch

For users wishing to use Scaleway, see our early scaleway-camlistore image. Our Scaleway image does not yet use Scaleway's Object Storage service, however, and puts all Camlistore data on the same volume as the Camlistore code itself, potentially complicating upgrades. It should be considered an early preview.

Making Camlistore run well on cloud providers required two other major features: the blobpacked storage backend (for latency and cost reasons) and HTTP/2 (for latency reasons).

The blobpacked storage backend allows faster reading & serving of files because it stores related blobs contiguously within larger container blobs (which are also valid zip files) instead of many small randomly dispersed blobs. See the blobpacked package documentation.

The Go HTTP/2 implementation was developed for both Camlistore and the Go standard library and will be included in Go 1.6. Camlistore 0.9 only requires Go 1.5 but includes a vendored copy of golang.org/x/net/http2. See https://http2.golang.org/ for a demo.

Source release:

camlistore0.9-src.zip

Github: github.com/camlistore/camlistore/tree/0.9

Binary release, new in this release:

Linux, OS X, Windows

Please note that Go 1.5 or newer is now required for this release.

Release stats

33 total committers over 942 commits since Camlistore 0.8, including Aaron Bieber, Aaron Boodman, Andrew Gerrand, Antonin Amand, Bill Thiede, Brad Fitzpatrick, David Jack, Emil Hessman, Eric Drechsel, Fabian Reinartz, Fabian Wickborn, Felix Geller, Gina White, Govert Versluis, Jingguo Yao, Lindsey Simon, Mario Russo, Mathieu Lonjaret, Matthieu Rakotojaona, Patrick Mezard, Robert Kroeger, Salman Aljammaz, Salmān Aljammāz, Steven L. Speek, Tamás Gulácsi, Will Norris.

Thank you!

Changes in 0.9

Android

• Restrict upload by WiFI SSID.

Camget

• Do not fetch file if already exists on local disk.

Camput

• New option for the share command: -search.
• Detect and warn when using the wrong GPG key.
• New option for the file command: -contents_only.
• The init command can now initialize the client configuration automatically from the server (using the server’s /help/ handler).

Camtool

• New makestatic command.
• New packblobs command.
• New option for search command: -rawquery.

Cammount

• Use VolumeName on OS X.

Devcam

• Detect and warn when outdated.
• New hook command to install and run git hooks. The only currently available hook is pre-commit, which runs gofmt and checks for trailing whitespace.
• New option for server command: -wipecache.
• New option for server command: -memindex.

Deploy

• Launcher to help deploy Camlistore on Google Cloud Platform.

Docker

• misc/docker/dock.go: new program to generate a docker image for camlistored, as well as binaries in isolation.

Genfileembed

• New flag to generate build tags.

Images

• Optional faster resizing with djpeg (if available).
• Support for tiff decoding.
• Use golang.org/x/image/draw for YCbCr resizing.

Importers

• New importer: pinboard.

Indexer

• Improved MySQL connection handling (switch to go-mysql-driver).
• New sorted implementation: LevelDB. It is now the default index storage in the server configuration that is generated when no configuration file exists.
• Out of order reindexing finished/fixed.
• When indexing from images: always try a small prefix buffer first, then read full file if needed.
• Defined a (large) limit on key and value size for all sorted implementations.
• The wholeRef of a file is always indexed, for faster file stats everywhere.
• RDBMS-specific placeholder replacements (and table name replacements) are now cached.

Misc performance changes, bug fixes, clean-ups, etc.

• HTTP/2
• make.go: improvements, refactoring, and cleanup.
• Migration to new third_parties: golang.org/x/oauth2, google.golang.org/cloud, etc.
• goexif: features and fixes, then contributed upstream.
• Server system status is constantly refreshed over websocket.
• Most of the JSON maps (e.g. for server responses) should be gone in favor of annotated structs.
• More robust localhost detection.
• The setup wizard was removed.

Ports

• Solaris: Camlistore should at least build.
• Darwin/386: deprecated, and user is warned about it when trying to build.

Schema

• Cache last blob read when reading files -> major performance optimization.
• File/chunks related changes, often optimizations with blobpacked in mind.

Search

• New “Around” parameter for queries. Introduced to help web UI features.
• New “filename” search predicate.

Storage

• New blobpacked blobserver: keeps related blobs physically contiguous within zip files.
• Many Google Cloud Storage improvements, including using the official google.golang.org/cloud/storage package almost everywhere.
• New SubFetcher interface.
• New BlobStreamer interface.
• New WholeRefFetcher interface.
• New memory blobserver.

Vendoring

• With the arrival of the new vendoring mechanism in Go 1.5 (on by default in Go 1.6), we started moving our vendored libraries from the third_party directory to the vendor directory (as is the convention for the new mechanism). This means we do not have to modify the import path of the third party packages we use anymore.
• In addition, we also started moving a fair number of the Camlistore packages out of the camlistore.org namespace, to independent packages under the go4.org namespace. The goal is to make those packages more visible and usable as they can be useful to a larger audience.

Travis

• We are now using Travis CI, until we can rewrite our build bot based on the Go buildlet and trybots.

Website

• New launcher to deploy Camlistore on Google Cloud: https://camlistore.org/launch.
• CONTRIBUTORS are generated from git log.
• Issue tracker has moved to github.
• The camweb binary which runs the camlistore.org website has been updated to run easily on Google Compute Engine and auto-refresh from the Camlistore source tree. This means our website should be more easily updated going forward.
• Markdown is now supported, and some of the static html content has been converted to it.

Web UI

• New style: top search bar, pudgy navigation menu, vertical contextual actions sidebar.
• Simpler URL blobRef format: /ui/.
• Some new features:
• Improved uploading: context aware (add to an existing set), duplicate detection/prevention, better drop area, show bytes transferred.
• Aspects: generic attributes editor for permanode aspect. New directory aspect.
• Sidebar: mass tagging/deleting.
• Selections state is saved across navigation.
• Download: view original (non-resized) item.
• Videos rendering.
• Pages changes:
• Improved status page, with lots more info, and restart camlistored “button”.
• New help page: client tools configuration, and download links.
• New mobile auto-configuration page with QR code.
• New /debug/config page.
• misc:
• Piggy menu shows server errors.
• Left/right navigation restored.
• Better layout on mobile.
• Cleaned out most of the non-React, legacy code.

Wkfs

• New package that provides a filesystem abstraction layer.