MaxBlobSize is the size of a single blob in Perkeep.
Variables
var (
ErrCorruptBlob = errors.New("corrupt blob; digest doesn't match")
// ErrNotImplemented should be returned in methods where the function is not implemented
ErrNotImplemented = errors.New("not implemented")
)
var ErrHandlerTypeNotFound = errors.New("requested handler type not loaded")
var ErrReadonly = errors.New("this blobserver is read only")
ErrReadonly is the error value returned by read-only blobservers.
CreateHandler instantiates an http Handler of type 'typ' from the
provided JSON configuration, and finding peer handlers and
configuration from the environment in 'loader'.
The handler 'typ' must have been previously registered with
RegisterHandlerConstructor.
EnumerateAll runs fn for each blob in src.
If fn returns an error, iteration stops and fn isn't called again.
EnumerateAll will not return concurrently with fn.
ListMissingDestinationBlobs reads from 'srcch' and 'dstch' (sorted
enumerations of blobs from two blob servers) and sends to
'destMissing' any blobs which appear on the source but not at the
destination.
destMissing is closed at the end.
If an invalid (zero) blob from srcch or dstch arrives,
ListMissingDestinationBlobs stops.
Receive wraps calling a BlobReceiver's ReceiveBlob method,
additionally providing verification of the src digest, and also
notifying the blob hub on success.
The error will be ErrCorruptBlob if the blobref didn't match.
RefTypes returns a list of blobref types appearing on the provided enumerator.
A blobref type is a string like "sha1", or whatever is on the left side
of the hyphen in a blobref.
To get the alphabet valid for the right side of the hyphen, use blob.TypeAlphabet(type).
StatBlobsParallelHelper is for use by blobserver implementations
that want to issue stats in parallel. This runs worker in multiple
goroutines (bounded by gate), but calls fn in serial, per the
BlobStatter contract, and stops once there's a failure.
The worker func should return two zero values to signal that a blob
doesn't exist. (This is different than the StatBlob func, which
returns os.ErrNotExist)
WaitForBlob waits until deadline for blobs to arrive. If blobs is empty, any
blobs are waited on. Otherwise, those specific blobs are waited on.
When WaitForBlob returns, nothing may have happened.
type BlobEnumerator interface {
// EnumerateBobs sends at most limit SizedBlobRef into dest,// sorted, as long as they are lexigraphically greater than// after (if provided).// limit will be supplied and sanity checked by caller.// EnumerateBlobs must close the channel. (even if limit// was hit and more blobs remain, or an error is returned, or// the ctx is canceled)
EnumerateBlobs(ctx context.Context,
dest chan<- blob.SizedRef,
after string,
limit int) error
}
type BlobHub interface {
// NotifyBlobReceived notes that a storage target has successfully received// a blob and asynchronously notifies registered listeners.//// If any synchronous receive hooks are registered, they're run before// NotifyBlobReceived returns and their error is returned.
NotifyBlobReceived(blob.SizedRef) error
// AddReceiveHook adds a hook that is synchronously run// whenever blobs are received. All registered hooks are run// on each blob upload but if more than one returns an error,// NotifyBlobReceived will only return one of the errors.
AddReceiveHook(func(blob.SizedRef) error)
RegisterListener(ch chan<- blob.Ref)
UnregisterListener(ch chan<- blob.Ref)
RegisterBlobListener(blob blob.Ref, ch chan<- blob.Ref)
UnregisterBlobListener(blob blob.Ref, ch chan<- blob.Ref)
}
type BlobReceiver interface {
// ReceiveBlob accepts a newly uploaded blob and writes it to// permanent storage.//// Implementations of BlobReceiver downstream of the HTTP// server can trust that the source isn't larger than// MaxBlobSize and that its digest matches the provided blob// ref. (If not, the read of the source will fail before EOF)//// To ensure those guarantees, callers of ReceiveBlob should// not call ReceiveBlob directly but instead use either// blobserver.Receive or blobserver.ReceiveString, which also// take care of notifying the BlobReceiver's "BlobHub"// notification bus for observers.
ReceiveBlob(ctx context.Context, br blob.Ref, source io.Reader) (blob.SizedRef, error)
}
BlobReceiver is the interface for receiving blobs.
type BlobRemover interface {
// RemoveBlobs removes 0 or more blobs. Removal of// non-existent items isn't an error. Returns failure if any// items existed but failed to be deleted.// ErrNotImplemented may be returned for storage types not implementing removal.// If RemoveBlobs returns an error, it's possible that either// none or only some of the blobs were deleted.
RemoveBlobs(ctx context.Context, blobs []blob.Ref) error
}
type BlobStatter interface {
// Stat checks for the existence of blobs, calling fn in// serial for each found blob, in any order, but with no// duplicates. The blobs slice should not have duplicates.//// If fn returns an error, StatBlobs returns with that value// and makes no further calls to fn.//// StatBlobs does not return an error on missing blobs, only// on failure to stat blobs.
StatBlobs(ctx context.Context, blobs []blob.Ref, fn func(blob.SizedRef) error) error
}
BlobStatter is the interface for checking the size and existence of blobs.
type BlobStreamer interface {
// BlobStream is an optional interface that may be implemented by// Storage implementations.//// StreamBlobs sends blobs to dest in an unspecified order. It is// expected that a Storage implementation implementing// BlobStreamer will send blobs to dest in the most efficient// order possible.//// The provided continuation token resumes the stream at a// point. To start from the beginning, send the empty string.// The token is opaque and must never be interpreted; its// format may change between versions of the server.//// If the content is canceled, the error value is// context.Canceled.//// StreamBlobs must unconditionally close dest before// returning, and it must return context.Canceled if// ctx.Done() becomes readable.//// When StreamBlobs reaches the end, the return value is nil.
StreamBlobs(ctx context.Context, dest chan<- BlobAndToken, contToken string) error
}
type FindHandlerByTyper interface {
// FindHandlerByType finds a handler by its handlerType and// returns its prefix and handler if it's loaded. If it's not// loaded, the error will be ErrHandlerTypeNotFound.//// This is used by handlers to find siblings (such as the "ui" type handler)// which might have more knowledge about the configuration for discovery, etc.//// Note that if this is called during handler construction// time, only the prefix may be returned with a nil handler// and nil err. Unlike GetHandler and GetStorage, this does// not cause the prefix to load immediately. At runtime (after// construction of all handlers), then prefix and handler will// both be non-nil when err is nil.
FindHandlerByType(handlerType string) (prefix string, handler interface{}, err error)
// AllHandlers returns a map from prefix to handler type, and// a map from prefix to handler.
AllHandlers() (map[string]string, map[string]interface{})
}
A GenerationNotSupportedError explains why a Storage
value implemented the Generationer interface but failed due
to a wrapped Storage value not implementing the interface.
type Generationer interface {
// Generation returns a Storage's initialization time and// and unique random string (or UUID). Implementations// should call ResetStorageGeneration on demand if no// information is known.// The error will be of type GenerationNotSupportedError if an underlying// storage target doesn't support the Generationer interface.
StorageGeneration() (initTime time.Time, random string, err error)
// ResetGeneration deletes the information returned by Generation// and re-generates it.
ResetStorageGeneration() error
}
Generationer is an optional interface and an optimization and paranoia
facility for clients which can be implemented by Storage
implementations.
If the client sees the same random string in multiple upload sessions,
it assumes that the blobserver still has all the same blobs, and also
it's the same server. This mechanism is not fundamental to
Perkeep's operation: the client could also check each blob before
uploading, or enumerate all blobs from the server too. This is purely
an optimization so clients can mix this value into their "is this file
uploaded?" local cache keys.
type HandlerIniter interface {
InitHandler(FindHandlerByTyper) error
}
HandlerIniter is an optional interface which can be implemented
by Storage or http.Handlers (from StorageConstructor or HandlerConstructor)
to be called once all the handlers have been created.
type Loader interface {
FindHandlerByTyper
// MyPrefix returns the prefix of the handler currently being constructed,// with both leading and trailing slashes (e.g. "/ui/").
MyPrefix() string
// BaseURL returns the server's base URL, without trailing slash, and not including// the prefix (as returned by MyPrefix).
BaseURL() string
// GetHandlerType returns the handler's configured type, but does// not force it to start being loaded yet.
GetHandlerType(prefix string) string // returns "" if unknown// GetHandler returns either a Storage or an http.Handler.// It forces the handler to be loaded and returns an error if// a cycle is created.
GetHandler(prefix string) (interface{}, error)
// GetStorage is like GetHandler but requires that the Handler be// a storage Handler.
GetStorage(prefix string) (Storage, error)
}
type MaxEnumerateConfig interface {
Storage
// MaxEnumerate returns the max that this storage interface is// capable of enumerating at once.
MaxEnumerate() int
}
MaxEnumerateConfig is an optional interface implemented by Storage
interfaces to advertise their max value for how many items can
be enumerated at once.
type ShutdownStorage interface {
Storage
io.Closer
}
ShutdownStorage is an optional interface for storage
implementations which can be asked to shut down
cleanly. Regardless, all implementations should be able to survive
crashes without data loss.
type WholeRefFetcher interface {
// OpenWholeRef returns a ReadCloser reading from offset bytes// into wholeRef (the blobref of an entire file).//// The returned wholeSize is the size of the file, without// subtracting any offset.//// The err will be os.ErrNotExist if the wholeref is not// known.
OpenWholeRef(wholeRef blob.Ref, offset int64) (rc io.ReadCloser, wholeSize int64, err error)
}
WholeRefFetcher is an optional fast-path interface exposed by the
'blobpacked' blob storage implementation, which packs pieces of
files together and can efficiently serve them contigously.
Package blobpacked registers the "blobpacked" blobserver storage type, storing blobs initially as one physical blob per logical blob, but then rearranging little physical blobs into large contiguous blobs organized by how they'll likely be accessed.
Package cond registers the "cond" conditional blobserver storage type to select routing of get/put operations on blobs to other storage targets as a function of their content.
Package diskpacked registers the "diskpacked" blobserver storage type, storing blobs packed together into monolithic data files with an index listing the sizes and offsets of the little blobs within the large files.
Package encrypt registers the "encrypt" blobserver storage type which stores all blobs and metadata with age encryption into other wrapped storage targets (e.g.
Package overlay registers the "overlay" blobserver storage type that presents storage that is the result of overlaying a storage ("upper") on top of another storage ("lower").
Package sftp registers the "sftp" blobserver storage type, storing blobs one-per-file in a forest of sharded directories to a remote SFTP server over an SSH connection.