Overview ▹
Overview ▾
Package blob defines types to refer to and retrieve low-level Perkeep blobs.
Index
- Constants
- Variables
- func HashFuncs() []string
- func NewHash() hash.Hash
- func ReaderAt(ctx context.Context, sf SubFetcher, br Ref) io.ReaderAt
- func TypeAlphabet(typ string) string
- func ValidRefString(s string) bool
- type Blob
- func FromFetcher(ctx context.Context, fetcher Fetcher, br Ref) (*Blob, error)
- func FromReader(ctx context.Context, br Ref, r io.Reader, size uint32) (*Blob, error)
- func NewBlob(ref Ref, size uint32, readAll func(ctx context.Context) ([]byte, error)) *Blob
- func (b *Blob) IsUTF8(ctx context.Context) (bool, error)
- func (b *Blob) ReadAll(ctx context.Context) (*bytes.Reader, error)
- func (b *Blob) Ref() Ref
- func (b *Blob) Size() uint32
- func (b *Blob) SizedRef() SizedRef
- func (b *Blob) ValidContents(ctx context.Context) error
- type ByRef
- func (s ByRef) Len() int
- func (s ByRef) Less(i, j int) bool
- func (s ByRef) Swap(i, j int)
- type ChanPeeker
- func (cp *ChanPeeker) Closed() bool
- func (cp *ChanPeeker) ConsumeAll()
- func (cp *ChanPeeker) MustPeek() SizedRef
- func (cp *ChanPeeker) MustTake() SizedRef
- func (cp *ChanPeeker) Peek() (sr SizedRef, ok bool)
- func (cp *ChanPeeker) Take() (sr SizedRef, ok bool)
- type DirFetcher
- func NewSimpleDirectoryFetcher(dir string) *DirFetcher
- func (df *DirFetcher) Fetch(ctx context.Context, r Ref) (file io.ReadCloser, size uint32, err error)
- type Fetcher
- func NewSerialFetcher(fetchers ...Fetcher) Fetcher
- type Ref
- func MustParse(s string) Ref
- func Parse(s string) (ref Ref, ok bool)
- func ParseBytes(s []byte) (ref Ref, ok bool)
- func ParseKnown(s string) (ref Ref, ok bool)
- func ParseOrZero(s string) Ref
- func RefFromBytes(b []byte) Ref
- func RefFromHash(h hash.Hash) Ref
- func RefFromString(s string) Ref
- func (r Ref) Digest() string
- func (r Ref) DigestPrefix(digits int) string
- func (r Ref) DomID() string
- func (r Ref) EqualString(s string) bool
- func (r Ref) HasPrefix(s string) bool
- func (r Ref) Hash() hash.Hash
- func (r Ref) HashMatches(h hash.Hash) bool
- func (r Ref) HashName() string
- func (r Ref) IsSupported() bool
- func (r Ref) Less(o Ref) bool
- func (r Ref) MarshalBinary() (data []byte, err error)
- func (r Ref) MarshalJSON() ([]byte, error)
- func (r Ref) String() string
- func (r Ref) StringMinusOne() string
- func (r Ref) Sum32() uint32
- func (r Ref) Sum64() uint64
- func (r *Ref) UnmarshalBinary(data []byte) error
- func (r *Ref) UnmarshalJSON(d []byte) error
- func (r Ref) Valid() bool
- type SizedByRef
- func (s SizedByRef) Len() int
- func (s SizedByRef) Less(i, j int) bool
- func (s SizedByRef) Swap(i, j int)
- type SizedRef
- func (sr SizedRef) HashMatches(h hash.Hash) bool
- func (sr SizedRef) Less(o SizedRef) bool
- func (sr SizedRef) String() string
- func (sr SizedRef) Valid() bool
- type SubFetcher
Package files
blob.go chanpeek.go fetcher.go ref.go
Constants
const Pattern = `\b([a-z][a-z0-9]*)-([a-f0-9]+)\b`
Pattern is the regular expression which matches a blobref. It does not contain ^ or $.
Variables
var (
ErrNegativeSubFetch = errors.New("invalid negative subfetch parameters")
ErrOutOfRangeOffsetSubFetch = errors.New("subfetch offset greater than blob size")
)
var ErrUnimplemented = errors.New("optional method not implemented")
ErrUnimplemented is returned by optional interfaces when their wrapped values don't implemented the optional interface.
func HashFuncs
func HashFuncs() []string
HashFuncs returns the names of the supported hash functions.
func NewHash
func NewHash() hash.Hash
NewHash returns a new hash.Hash of the currently recommended hash type. Currently this is SHA-224, but is subject to change over time.
func ReaderAt
func ReaderAt(ctx context.Context, sf SubFetcher, br Ref) io.ReaderAt
ReaderAt returns an io.ReaderAt of br, fetching against sf. The context is stored in and used by the returned ReaderAt.
func TypeAlphabet
func TypeAlphabet(typ string) string
TypeAlphabet returns the valid characters in the given blobref type. It returns the empty string if the typ is unknown.
func ValidRefString
func ValidRefString(s string) bool
type Blob
type Blob struct {
// contains filtered or unexported fields
}
Blob represents a blob. Use the methods Size, SizedRef and ReadAll to query and get data from Blob.
func FromFetcher
func FromFetcher(ctx context.Context, fetcher Fetcher, br Ref) (*Blob, error)
FromFetcher fetches br from fetcher and slurps its contents to memory. It does not validate the blob's digest. Use the Blob.ValidContents method for that.
func FromReader
func FromReader(ctx context.Context, br Ref, r io.Reader, size uint32) (*Blob, error)
FromReader slurps the given blob from r to memory. It does not validate the blob's digest. Use the Blob.ValidContents method for that.
func NewBlob
func NewBlob(ref Ref, size uint32, readAll func(ctx context.Context) ([]byte, error)) *Blob
NewBlob constructs a Blob from its Ref, size and a function that returns the contents of the blob. Any error in the function newReader when constructing the io.ReadCloser should be returned upon the first call to Read or Close.
func (*Blob) IsUTF8
func (b *Blob) IsUTF8(ctx context.Context) (bool, error)
IsUTF8 reports whether the blob is entirely UTF-8.
func (*Blob) ReadAll
func (b *Blob) ReadAll(ctx context.Context) (*bytes.Reader, error)
ReadAll reads the blob completely to memory, using the provided context, and then returns a reader over it. The Reader will not have errors except EOF.
The provided is only ctx is used until ReadAll returns.
func (*Blob) Ref
func (b *Blob) Ref() Ref
Ref returns the blob's reference.
func (*Blob) Size
func (b *Blob) Size() uint32
Size returns the size of the blob (in bytes).
func (*Blob) SizedRef
func (b *Blob) SizedRef() SizedRef
SizedRef returns the SizedRef corresponding to the blob.
func (*Blob) ValidContents
func (b *Blob) ValidContents(ctx context.Context) error
ValidContents reports whether the hash of blob's content matches its reference.
type ByRef
type ByRef []Ref
ByRef sorts blob references.
func (ByRef) Len
func (s ByRef) Len() int
func (ByRef) Less
func (s ByRef) Less(i, j int) bool
func (ByRef) Swap
func (s ByRef) Swap(i, j int)
type ChanPeeker
type ChanPeeker struct {
Ch <-chan SizedRef
// contains filtered or unexported fields
}
ChanPeeker wraps a channel receiving SizedRefs and adds a 1 element buffer on it, with Peek and Take methods.
func (*ChanPeeker) Closed
func (cp *ChanPeeker) Closed() bool
Closed reports true if no more SizedRef values are available.
func (*ChanPeeker) ConsumeAll
func (cp *ChanPeeker) ConsumeAll()
ConsumeAll drains the channel of all items.
func (*ChanPeeker) MustPeek
func (cp *ChanPeeker) MustPeek() SizedRef
MustPeek returns the next SizedRef or panics if none is available.
func (*ChanPeeker) MustTake
func (cp *ChanPeeker) MustTake() SizedRef
MustTake returns the next SizedRef, else panics if none is available.
func (*ChanPeeker) Peek
func (cp *ChanPeeker) Peek() (sr SizedRef, ok bool)
Peek returns the next SizedRef and whether one was available.
func (*ChanPeeker) Take
func (cp *ChanPeeker) Take() (sr SizedRef, ok bool)
Take returns the next SizedRef and whether one was available for the taking.
type DirFetcher
type DirFetcher struct {
// contains filtered or unexported fields
}
func NewSimpleDirectoryFetcher
func NewSimpleDirectoryFetcher(dir string) *DirFetcher
func (*DirFetcher) Fetch
func (df *DirFetcher) Fetch(ctx context.Context, r Ref) (file io.ReadCloser, size uint32, err error)
type Fetcher
type Fetcher interface {
// Fetch returns a blob. If the blob is not found then
// os.ErrNotExist should be returned for the error (not a wrapped
// error with a ErrNotExist inside)
//
// The contents are not guaranteed to match the digest of the
// provided Ref (e.g. when streamed over HTTP). Paranoid
// callers should verify them.
//
// The caller must close blob.
//
// The provided context is used until blob is closed and its
// cancellation should but may not necessarily cause reads from
// blob to fail with an error.
Fetch(context.Context, Ref) (blob io.ReadCloser, size uint32, err error)
}
Fetcher is the minimal interface for retrieving a blob from storage. The full storage interface is blobserver.Storage.
func NewSerialFetcher
func NewSerialFetcher(fetchers ...Fetcher) Fetcher
type Ref
type Ref struct {
// contains filtered or unexported fields
}
Ref is a reference to a Perkeep blob. It is used as a value type and supports equality (with ==) and the ability to use it as a map key.
func MustParse
func MustParse(s string) Ref
MustParse parse s as a blobref and panics on failure.
func Parse
func Parse(s string) (ref Ref, ok bool)
Parse parse s as a blobref and returns the ref and whether it was parsed successfully.
func ParseBytes
func ParseBytes(s []byte) (ref Ref, ok bool)
ParseBytes is like Parse, but parses from a byte slice.
func ParseKnown
func ParseKnown(s string) (ref Ref, ok bool)
ParseKnown is like Parse, but only parse blobrefs known to this server. It returns ok == false for well-formed but unsupported blobrefs.
func ParseOrZero
func ParseOrZero(s string) Ref
ParseOrZero parses as a blobref. If s is invalid, a zero Ref is returned which can be tested with the Valid method.
func RefFromBytes
func RefFromBytes(b []byte) Ref
RefFromBytes returns a blobref from the given string, for the currently recommended hash function.
func RefFromHash
func RefFromHash(h hash.Hash) Ref
RefFromHash returns a blobref representing the given hash. It panics if the hash isn't of a known type.
func RefFromString
func RefFromString(s string) Ref
RefFromString returns a blobref from the given string, for the currently recommended hash function.
func (Ref) Digest
func (r Ref) Digest() string
Digest returns the lower hex digest of the blobref, without the e.g. "sha224-" prefix. It panics if r is zero.
func (Ref) DigestPrefix
func (r Ref) DigestPrefix(digits int) string
func (Ref) DomID
func (r Ref) DomID() string
func (Ref) EqualString
func (r Ref) EqualString(s string) bool
EqualString reports whether r.String() is equal to s. It does not allocate.
func (Ref) HasPrefix
func (r Ref) HasPrefix(s string) bool
HasPrefix reports whether s is a prefix of r.String(). It returns false if s does not contain at least the digest name prefix (e.g. "sha224-") and one byte of digest. It does not allocate.
func (Ref) Hash
func (r Ref) Hash() hash.Hash
Hash returns a new hash.Hash of r's type. It panics if r is zero.
func (Ref) HashMatches
func (r Ref) HashMatches(h hash.Hash) bool
func (Ref) HashName
func (r Ref) HashName() string
HashName returns the lowercase hash function name of the reference. It panics if r is zero.
func (Ref) IsSupported
func (r Ref) IsSupported() bool
func (Ref) Less
func (r Ref) Less(o Ref) bool
Less reports whether r sorts before o. Invalid references blobs sort first.
func (Ref) MarshalBinary
func (r Ref) MarshalBinary() (data []byte, err error)
MarshalBinary implements Go's encoding.BinaryMarshaler interface.
func (Ref) MarshalJSON
func (r Ref) MarshalJSON() ([]byte, error)
func (Ref) String
func (r Ref) String() string
func (Ref) StringMinusOne
func (r Ref) StringMinusOne() string
StringMinusOne returns the first string that's before String.
func (Ref) Sum32
func (r Ref) Sum32() uint32
func (Ref) Sum64
func (r Ref) Sum64() uint64
func (*Ref) UnmarshalBinary
func (r *Ref) UnmarshalBinary(data []byte) error
UnmarshalBinary implements Go's encoding.BinaryUnmarshaler interface.
func (*Ref) UnmarshalJSON
func (r *Ref) UnmarshalJSON(d []byte) error
func (Ref) Valid
func (r Ref) Valid() bool
type SizedByRef
type SizedByRef []SizedRef
SizedByRef sorts SizedRefs by their blobref.
func (SizedByRef) Len
func (s SizedByRef) Len() int
func (SizedByRef) Less
func (s SizedByRef) Less(i, j int) bool
func (SizedByRef) Swap
func (s SizedByRef) Swap(i, j int)
type SizedRef
type SizedRef struct {
Ref Ref `json:"blobRef"`
Size uint32 `json:"size"`
}
SizedRef is like a Ref but includes a size. It should also be used as a value type and supports equality.
func (SizedRef) HashMatches
func (sr SizedRef) HashMatches(h hash.Hash) bool
func (SizedRef) Less
func (sr SizedRef) Less(o SizedRef) bool
Less reports whether sr sorts before o. Invalid references blobs sort first.
func (SizedRef) String
func (sr SizedRef) String() string
func (SizedRef) Valid
func (sr SizedRef) Valid() bool
type SubFetcher
type SubFetcher interface {
// SubFetch returns part of a blob.
// The caller must close the returned io.ReadCloser.
// The Reader may return fewer than 'length' bytes. Callers should
// check. The returned error should be: ErrNegativeSubFetch if any of
// offset or length is negative, or os.ErrNotExist if the blob
// doesn't exist, or ErrOutOfRangeOffsetSubFetch if offset goes over
// the size of the blob. If the error is ErrUnimplemented, the caller should
// treat this Fetcher as if it doesn't implement SubFetcher.
SubFetch(ctx context.Context, ref Ref, offset, length int64) (io.ReadCloser, error)
}
A SubFetcher is a Fetcher that can retrieve part of a blob.