Release 4.0.0

@wcandillon

4.0.0 (2024-04-22)

VisionCamera V4 is finally here! 🥳🥳🥳🥳🚀💪

The most anticipated features are:

Skia Frame Processors: Finally it's reality - draw directly onto a Frame using react-native-skia! It has never been simpler to draw something onto a Frame, with a native iOS/Android app you'd spend days if not weeks on this.

const frameProcessor = useSkiaFrameProcessor((frame) => {
  'worklet'

  // draw frame itself
  frame.render()

  // draw red rectangle into center of frame
  const centerX = frame.width / 2
  const centerY = frame.height / 2
  const rect = Skia.XYWHRect(centerX, centerY, 150, 150)
  const paint = Skia.Paint()
  paint.setColor(Skia.Color('red'))
  frame.drawRect(rect, paint)
}, [])

CameraX rewrite: muuuuch more stability!!
GPS Location Tags: Adds location tags to photos and videos.

If you followed along the VisionCamera V2 -> V3 journey, you might remember that I switched from the (at-that-time-) immature CameraX library, to the low-level almost C-like Camera2 API.
The motivation behind that was that CameraX was too limiting at that time, and I wanted to gain full control over the hardware sensor and output streams.
While this worked great on my test devices, it seems like Camera2 wasn't implemented the same on all devices, and countless of weird issues arised, such as Samsung phones completely crashing, photos being rotated, frame processors running really slow on Huawei, some non-null Camera properties actually being null, etc. - the list goes on.
While I was fighting my way through Android's low-level Camera2 API, CameraX started gaining more and more interesting features, and just recently they built a new API: stream sharing.
With stream sharing, CameraX now supports all the features that VisionCamera has, and so I decided to fully rewrite the Android codebase to CameraX.
I was pretty impressed with what I built in just a few months using Camera2 APIs, but in reality this is becoming unmaintainable as countless of phone manufacturers just don't really care about Camera2 APIs, and I can't be the person monkeypatching that all the time.
Compared to VisionCamera V3, VisionCamera V4 is now much more stable, and brings a ton of new features.

✨ Features

Fully rewrite the Android codebase to CameraX. This brings huge stability improvements overall, and should fix countless of bugs 🥳
Add Skia integration to provide drawable Frame Processors! 🥳🎨
- Add useSkiaFrameProcessor hook to create a drawable Frame Processor
- Inside a useSkiaFrameProcessor the caller can directly draw to the frame as it extends a SkCanvas
- Use all Skia.* APIs
- Render the Frame with optional shaders using frame.render(shader)
Add GPS/EXIF Location Tags to captured photos or videos 📍
- Add useLocationPermission hook to get location permission
- Add enableLocation prop to <Camera>
- Add $VCEnableLocation config option to Podfile to disable location APIs (e.g. if Apple Review is not happy with unused location APIs in your code)
Add photoQualityBalance prop to <Camera> to optimize the Camera pipeline for the given prioritization (speed, balanced (default) or quality)
Add preview prop to <Camera> to enable or disable the preview view/stream.
Add androidPreviewViewType prop to <Camera> to specify which preview implementation to use (surface-view is faster and supports HDR, texture-view supports transparency, masks, rotations and clipping)
New takeSnapshot(..) API to take instant snapshots of the video/preview streams. This can be used for intermediate results before takePhoto(..) completes. 📸
Use stream sharing to drop limitation of concurrent outputs - now photo, video, frameProcessor, preview and codeScanner can all be enabled at the same time!
Add photo HDR support to Android (using vendor-specific extensions) 🥳
Add video HDR support to Android (using 10-bit video HDR) 🥳
Always support yuv and rgb pixel-formats
- yuv is more efficient (uses ~60% less memory than rgb, and does not need conversion)
- rgb requires explicit conversion from yuv and will increase latency
Rewrite FPS Graph in JS to make it look more sexy and simplify native build (less native UI code)
Add support for legacy Camera1 devices
Lower minSdk level from 23 to 21.
Separate VisionCamera into three submodules/subspecs:
- VisionCamera/Core: The core library which could be used in a native iOS/Android app
- VisionCamera/React: React Native bindings for VisionCamera/Core
- VisionCamera/FrameProcessors: React Native bindings for synchronous onFrame callbacks using react-native-worklets-core

❌ Deprecations

Remove qualityPrioritizaion parameter in takePhoto(..) in favor of new photoQualityBalance prop on <Camera>
Remove enablePrecapture parameter in takePhoto(..) because CameraX now properly runs precapture sequences with device-quirks ironed out
Remove native pixel-format since yuv is native
Remove enableGpuBuffers since CameraX now managed such optimizations

💨 Performance Improvements

Optimize for concurrent configure { ... } abort calls and give higher priority to newer update
Improve performance for some JNI calls (SharedArray and FrameProcessor::call)

🐛 Bug Fixes

Fully rewrite the Android codebase to CameraX. This brings huge stability improvements overall, and should fix countless of bugs.
Fix onStarted/onStopped sometimes being called a bit too early
Fix shutter sound playing when phone is in silent mode
Fix Camera not starting and appearing black on some devices
Fix Camera crashing because no resolution could be found
Fix Camera crashing the entire Android OS because GPU buffers were enabled
Automatically fall-back to a supported videoCodec if the given video-codec is not supported (e.g. h265 -> h264)
Fix Camera flipping not working sometimes

Use correct UIManager in findCameraView depending on the architecture (#2702) (5223f5b)

📚 Documentation

Update docs for Skia Frame Processors
Update docs for takeSnapshot
Update docs for V4
Add a ton of new Frame Processor plugins by @gev2002 (#2698) (f396ea7)
Update FRAME_PROCESSOR_PLUGINS.mdx (#2705) (04f89c4)

Check out the VisionCamera V4 PR (#2623) for a full list of changes.

Huge shoutout to @wcandillon for helping me get the required changes into react-native-skia! We spent a lot of time debugging weird GPU issues and optimizing the code.
These are the Skia PRs needed for the VisionCamera V4 Skia Frame Processors feature:

Provide feedback

Saved searches

Use saved searches to filter your results more quickly