# `MobMedia`
[🔗](https://github.com/GenericJam/mob_media/blob/0.1.0/lib/mob_media.ex#L1)

Media helpers for Mob apps, through the platform's own decoders: what's in
a media file, its audio as PCM for speech-to-text, and a video frame as a
JPEG.

    {:ok, %{duration_ms: 12_480, has_video: true, width: 1080, height: 1920}} =
      MobMedia.info(path)

    {:ok, pcm} = MobMedia.audio_pcm(path)          # 16 kHz mono s16le
    {:ok, text} = MobWhisper.transcribe(pcm)

    {:ok, jpeg} = MobMedia.frame(path, 4_000, max_px: 1024)

Android decodes with `MediaExtractor` + `MediaCodec` and grabs frames with
`MediaMetadataRetriever`; iOS uses `AVAssetReader` and
`AVAssetImageGenerator`. Anything the platform's decoders handle works
(verified on both: m4a/AAC, mp3, wav, ogg/opus, mp4/H.264). No
ffmpeg, no permission: `path` is a local file the app can already read
(from `Mob.Files.pick/2`, mob_photos, mob_camera, `Mob.Audio`, ...), not a
`content://` URI.

## Sync, off the schedulers

Every function blocks **the calling process** until the answer is ready and
returns it. The work itself runs on a native thread (a Kotlin thread pool on
Android, an operation queue on iOS); the NIF only queues it and returns, and the
native side sends the result back as a message. No BEAM scheduler, normal or
dirty, is held while a file decodes.

Decoding a long recording takes a while (10-30% of its length on a
mid-range Android phone), so don't call `audio_pcm/2` from a screen's
`handle_event`/`handle_info` directly: the screen would stop answering
taps until it returns. Run it in a task and send yourself the result:

    screen = self()
    Task.start(fn -> send(screen, {:pcm_ready, MobMedia.audio_pcm(path)}) end)

Each call runs in its own short-lived process, so a call that times out
(`:timeout` option) returns `{:error, :timeout}` at once while the native
work finishes in the background and its temporary output is deleted. A
caller that dies is handled the same way.

## Memory

`audio_pcm/2` never holds the whole decode in native memory: the platform
side streams the PCM into a file as it decodes. Without `:to`, that file is
temporary and is read into the binary `audio_pcm/2` returns (16 kHz mono is
1.9 MB a minute, 115 MB an hour). For long recordings pass `to: path`: the
PCM stays in that file (`{:ok, path}`), and you can read it a window at a
time (whisper works in 30 s windows: 960 000 bytes at 16 kHz mono):

    {:ok, pcm_path} = MobMedia.audio_pcm(video, to: Path.join(dir, "audio.pcm"))
    {:ok, fd} = :file.open(pcm_path, [:read, :binary, :raw])
    {:ok, window} = :file.pread(fd, 0, 960_000)

## Errors

  * `:not_found` — `path` doesn't exist.
  * `:unsupported` — not a format the platform can open or decode.
  * `:no_audio` / `:no_video` — the file has no such track.
  * `:decode_failed` — the decoder failed part way (a damaged file).
  * `:io_error` — the output couldn't be written or read back.
  * `:timeout` — the `:timeout` option ran out.
  * `:unavailable` — the NIF isn't linked into this build (or you're on the
    host without `MobMedia.Engine.Fake`).
  * `{:jni, reason}` (Android) / `:alloc` (iOS) — the NIF couldn't queue
    the job; see `MobMedia.Engine.Native`.

## Engines

The decoding is done by a `MobMedia.Engine`: `MobMedia.Engine.Native` when
the NIF is linked, `MobMedia.Engine.Unavailable` otherwise. Tests and
agents use `MobMedia.Engine.Fake` (real WAV decoding, scripted answers for
anything else):

    config :mob_media, engine: MobMedia.Engine.Fake

# `info`

```elixir
@type info() :: %{
  duration_ms: non_neg_integer() | nil,
  width: pos_integer() | nil,
  height: pos_integer() | nil,
  has_audio: boolean(),
  has_video: boolean(),
  mime: String.t() | nil
}
```

What `info/2` returns. Fields the file doesn't have are `nil`.

# `reason`

```elixir
@type reason() ::
  :not_found
  | :unsupported
  | :no_audio
  | :no_video
  | :decode_failed
  | :io_error
  | :timeout
  | :unavailable
  | {:jni, atom()}
  | term()
```

Why an operation failed; see the moduledoc.

# `audio_pcm`

```elixir
@spec audio_pcm(Path.t(), keyword()) ::
  {:ok, binary() | Path.t()} | {:error, reason()}
```

Decode the audio of `path` (an audio file, or a video's audio track) to
signed 16-bit little-endian interleaved PCM, the format
`MobWhisper.transcribe/2` takes.

Options:

  * `:rate` — sample rate in Hz (default 16000).
  * `:channels` — `1` (default; channels averaged) or `2`.
  * `:to` — write the PCM to this file instead of returning it, and return
    `{:ok, to}`. The file appears only when the decode succeeded (it's
    written beside it and renamed). See "Memory" in the moduledoc.
  * `:timeout` — ms, default `:infinity`.

`{:error, :no_audio}` when the file has no audio track. The PCM covers the
whole track: its length (`MobMedia.PCM.duration_ms/3`) matches the duration
within a few tens of milliseconds, except AAC files, whose container
duration includes encoder priming and padding (about 130 ms longer).

# `available?`

```elixir
@spec available?() :: boolean()
```

Whether media can be decoded in this build (the NIF is linked, or an engine is configured).

# `engine`

```elixir
@spec engine() :: module()
```

The engine in use: `config :mob_media, :engine` if set, otherwise
`MobMedia.Engine.Native` when the NIF is linked, otherwise
`MobMedia.Engine.Unavailable`.

# `frame`

```elixir
@spec frame(Path.t(), non_neg_integer(), keyword()) ::
  {:ok, binary()} | {:error, reason()}
```

The video frame of `path` at `at_ms` milliseconds, as a JPEG binary.

The frame is the one shown at that time (exact, not the nearest keyframe),
upright (the video's rotation applied), scaled down so neither side exceeds
`:max_px`. A time past the end gives the last frame; if the platform can't
decode the exact frame there, the nearest earlier keyframe.

Options:

  * `:max_px` — longest side in pixels (default 1280); never
    scales up. `nil` keeps the full size.
  * `:quality` — JPEG quality 1..100 (default 85).
  * `:timeout` — ms, default `:infinity`.

`{:error, :no_video}` for a file without a video track.

# `info`

```elixir
@spec info(Path.t(), keyword()) :: {:ok, info()} | {:error, reason()}
```

Duration, video dimensions, which tracks the file has, and its MIME type.

    {:ok, %{duration_ms: 3_021, width: nil, height: nil,
            has_audio: true, has_video: false, mime: "audio/mp4"}}

`width`/`height` are the size the video is **shown** at: a portrait phone
video recorded as 1920x1080 with a 90° rotation reports 1080x1920, the same
orientation `frame/3` returns. `mime` is the container type (Android reads
the container; iOS goes by the file name's extension) with common aliases folded: `audio/x-wav` is `audio/wav`,
`audio/x-m4a` is `audio/mp4`; `nil` when unknown.

Options: `:timeout` (ms, default `:infinity`).

---

*Consult [api-reference.md](api-reference.md) for complete listing*