Release 0.8 ("Tokyo")

2014-08-03

Forgive us Internet, for we have slacked. It has been over 5 months since our last release.

But we've been busy hacking away and have good stuff for you. The most exciting thing in this release is probably the new importer system and implementation of a bunch of importer types, slurping in your content from elsewhere on the web and mapping it into Camlistore's data model and letting you search it. In addition to importers we've improved search, started pulling code out of the core into separate "apps", and just made general improvements and bug fixes all over the place. Enjoy!

And if you find yourself in Tokyo for a GoCon conference, be sure to check out Robot Restaurant. It's wonderfully bizarre. This release may not have robots, but it does have importers which can run automatically, which is almost the same? Okay, maybe not.

Please note that Go 1.3 or newer is now required for this release.

Release stats

30 total committers over 610 commits since Camlistore 0.7, including Aaron Boodman, Andrew Gerrand, Antti Rasinen, Bill Thiede, Brad Fitzpatrick, Bret Comnes, Brian Gitonga Marete, Daniel Coonce, Daniel Erat, Edward Sheffler III, Eric Drechsel, Govert Versluis, Hernan Grecco, Joshua Gay, Mario Russo, Mathieu Lonjaret, Matt Jibson, Nick O'Neill, Philip Snowberger, Piotr S. Staszewski, Ritesh Sinha, Rob Young, Robert Kroeger, Steven L. Speek, Tamás Gulácsi.

Thank you!

New in 0.8

Apps

• Hello app, serves as a prototype and example. We’re in the process of removing more code from the main camlistored server process and kicking them out into client applications which can be sandboxed (both for CPU instructions, and for API data access) into separate processes. This will also force our APIs to improve if they’re found to be lacking.
• Publisher app, replaces publish handler. The publishing code (for putting photo galleries on the web for others to see) has been removed from the core of camlistored and is now an external process, but one that can be managed by camlistored.

Web UI

• Selected permanodes may deleted.
• Image representation for any kind of content (camliContentImage is now a well-supported permanode attribute in the importers, web UI, search, etc).
• “Aspects” … multiple view options for a given blob or permanode
• Debug UI window invoked by pressing the ”|” (pipe) key for doing bulk operations on selected items. Currently only tagging.

Storage

• Rewritten blob sync handler (the server’s background async replication), now bug fixes, better UI status, and option to run a full validation.
• Store blobs in MongoDB. (previously only the Camlistore index could be stored in MongoDB)
• Switched MySQL driver to go-mysql-driver
• Google Cloud Storage auth works automatically when run on Google Compute Engine.
• Google Cloud Storage supports reporting its storage generation, so camput will cache more.

Search

• Indexer now gracefully handles dependent blobs arriving out of order and reschedules indexing as dependencies are satisfied. This means full syncs in arbitrary orders don’t confuse the indexer.
• RelationConstraint implemented for Relation type “parent”
• Search operator syntax for searching permanodes for arbitrary attributes: attr:<attribute_name>:<attribute_value>
• Search operator syntax for searching permanodes by their parent permanode(s): childrenof:sha1-xxxxx
• Searches can contain parenthesized subexpressions and accept ‘and’ and ‘or’. A whitespace separation still means ‘and’.
• Permanode deletions now taken into account by index corpus, hence in search results too.
• Cleaned up subdirs in ./pkg/index. Cleaned up ./pkg/sorted/ API. Many tests now use Docker to test dependencies (mongo, mysql, postgres, etc) rather than using the host machine’s database.
• Describe rules, eliminating the previous inflexible and over-aggressive “describe depth” parameter.
• Indexing of MusicBrainz album IDs.
• Sorting of permanodes by time now recognizes a handful of common date-related schema attributes as used by the importers. The default search response’s sort order should be sane for most people now, showing newer content first.
• Allow searching for files by their “wholeref” (complete digest)

Importers

• Importer account setup UI at /importer/. Importers are no longer configured in the config file.
• Regular, recurring imports. (Optional, per account per importer type.)
• Foursquare venue and check-in importer.
• Picasaweb (Google Photos) importer.
• Twitter importer.
• Feed (RSS, Atom, RDF) importer.

Tools

• Added camtool search subcommand.
• camtool sync: warn about differing identities.
• camtool googinit: help setup Google Cloud Storage or Driver.

General

• kv: all the kv verifydb flags are on by default in dev mode, to help with detecting corruption. One known kv corruption bug remains.
• devcam server –makethings pre-populates your dev server with fake importer content to aid in UI development work
• Photos with GPS location in the EXIF but no timezone information (most cameraphone photos) now map the latitude and longitude to a timezone using the latlong package so they sort properly with respect to other events in your stream (e.g. Tweets, Foursquare checkins, etc.) Useful for anybody who travels or doesn’t live in UTC..