Angular with NativeScript: Creating the Blackout Lighting Console

Lighting consoles are electronic devices used in theatrical lighting design to control multiple stage lights at once. Traditionally, lighting consoles are standalone physical devices. The nstudio team wanted to bring the power and versatility of the lighting console to mobile devices. This is the story of how we used Angular and NativeScript to bring that vision to reality.

We developed Blackout, an iPad app that provides the same features as standalone lighting consoles on a table. The app supports a versatile DMX (Digital Multiplex) control platform, compatible with industry-standard sACN E1.31 and Art-Net protocols via WiFi or Ethernet. Additional Bluetooth control is also available via LumenRadio TimoTwo chip devices.

The main interface for Blackout with Fixture Controls active in the left sidebar and the “pills” in the center which represent patched light fixtures.

Since Blackout’s launch, the app has been used by experts in TV, film, and live events across the globe. Blackout is the choice of industry professionals such as Jeff Brink (product owner), Jason Elliott — Mr. Beast YouTube Channel, Andrew Korner, Mike Bauman, Bill Almeida, Len Levine, Shane Hurlbut (Cinematographer), Dion Beebe (Cinematographer), Rodney Charters (Cinematographer).

Blackout Teaser

How we built it

One of the constraints of apps in this space is that they must maintain a constant and uninterrupted ~23ms transmission rate of data to hundreds of wirelessly connected lights while delivering uncompromised 60–120fps of interactive UI with hundreds of real time data indicators alongside detailed fixture controls. DMX has a max refresh rate of 44Hz and the app must be in lock step with user control across realtime lighting updates.

Using Angular 17+ with NativeScript 8+ in tandem with ngrx state management offered a powerful set of features to meet our development demands.

We were able to combine Angular’s DI, expressive directives, control flow templating and reactive view bindings with SwiftUI open source libraries like DynamicColor, ColorPicker and FoggyColors. We were able to integrate with iOS CoreMIDI directly with NativeScript to bring the UI and advanced capabilities to reality.

Tech Stack

The team wanted to leverage our existing Angular development skills and the latest improvements in Angular, so we used v17. In order to take advantage of platform optimized views with UIKit we used NativeScript 8+.

NativeScript enabled the team to use platform optimized views — UIKit — using declarative JavaScript view templating. We could also target SwiftUI just where desired with @nativescript/swift-ui. You can learn more in this great introduction to SwiftUI for NativeScript.

Here is a list of some of the other technologies we used:

- Tailwind CSS of all platform optimized views via NativeScript

- State Management via NgRx 17.x

- SQLite for local offline database

- Supabase for remote syncable database

- Mobile UI testing with Maestro

The Angular Advantage

Angular offers a complete development surface from routing to templating to dependency injection to help project organization as well as features to aid in complex orchestration of UI like directives. The declarative component architecture possible with Angular provided for intuitive control flow around sophisticated visual state trees helping achieve some of the more challenging aspects of the project. This interactive advanced color mode display which translates in real time between RGB to CMYK to HSB to Chromaticity XY Coordinates:

Blackout Advanced Color Mode

This view in particular demonstrates the fluid nature of Angular components paired with SwiftUI possible with NativeScript:

We are able to drive completely isolated SwiftUI state via Angular component Inputs and process events from SwiftUI via Outputs to orchestrate an array of Angular component interactions.

Angular has a robust ecosystem including libraries like ngrx for state management. Blackout uses ngrx to drive all the complex data requirements.

We were able to use Angular 17 the day of release due to how well NativeScript is maintained. This enabled us to use the new control flow syntax to improve templating conditions while at the same time gaining performance benefits from the syntax.

There were two UI features in particular where Angular directives delivered immense value:

1. MIDI Learn Mode

“MIDI Learn Mode” is a UI state the user can enable which provides an interactive UI state where MIDI controls can be assigned to various highlighted UI controls. The feature itself was managed by a single Angular directive which orchestrates the connections between a `MIDIService` and the various UI controls.

The directive’s implementation was not overly complex, here’s a snippet of the code:

import { Directive } from '@angular/core'

@Directive({
  selector: '[midiLink]'
})
export class MidiLinkDirective {
  midiService = inject(MidiService)

  activate() {
    // impl
  }
}

In the app code, the directive can be used by referencing the it as an attribute.:

<GridLayout midiLink="midi-link-fader" />

Blackout MIDI Learn Mode

Here’s a video clip of MIDI in action at the LDI Show in Las Vegas:

Blackout MIDI in action at LDI in Las Vegas

2. nsIf directive

There is a performance difference between working with DOM views and UIKit views. UIKit views are expensive to create and remove.

The layout can generally be much slower on mobile devices and, unlike in Angular web, ngFor becomes rare meaning you will often use ngFor (or @for) far less for large lists due to the platform-native recycled row controls being available through NativeScript. Virtualized lists (also known as Collection Views or Recycler Views) are the norm on mobile devices (due to their highly optimized memory usage) vs. web desktop interfaces. Also, stutters while scrolling virtualized lists can be very perceptible if not architected properly from a view bindings perspective.

To solve this issue, we created a very specialized nsIf directive (available to study and use in this gist), which always creates the view but detaches it from change detection until it’s ready. Instead, it sets the visibility of the view to hidden or collapsed, so the overhead is reduced, and only impacts the application during re-layout when the condition/context reevaluates.

Pairing Best of Web with Best of Platform

Pairing Angular with NativeScript also allowed us to style all UIKit views with Tailwind CSS which enabled us to compose all the screens significantly faster than we would have been able to do with UIKit or SwiftUI alone.

Solidifying state management with NgRX

Workers (aka, multithreading) are a necessity in the app as we need constant transmission rate where any thread interruption, like by the UI thread, is noticeable. Having a rock-solid state layer which can provide the correct data to workers and the UI is absolutely paramount. Previous iterations of the app relied on mutable state reacting to changes, where our approach now relies heavily on an immutable state layer with carefully crafted selectors and memoization to ensure we don’t waste valuable CPU cycles with unnecessary calculations. NgRX was the right tool for the job.

Each fixture has to go through many layers until it gets to the final value that needs to be transmitted to lights. The values must be calculated in order of priority through each fader that affects it, including mapping from the user facing value to the DMX (8/16 bit) value. Since this is a costly operation, we must consider custom NgRx memoization techniques for both inputs and outputs of selectors, as well as handling potential back pressure from the state.

NgRx memoizes the inputs of a selector and the output, so it doesn’t execute if the inputs didn’t change, and also doesn’t emit if the output didn’t change. As seen in the example:

// wrong way to select. This will fire every time the state changes
const selectSecond = createSelector(selectState, (s) => s.first.second);
// this will fire every time the state changes
const selectFirst = createSelector(selectState, (s) => s.first);
// this will fire every time "first" changes
const selectSecond = createSelector(selectFirst, (s) => s.second);

Selectors can also be seen as computed values (or RxJs map operators with a lot of clever distinctUntilChanged operators). You might need some finer grained memoization for costly operations. Consider the following selector:

const selectCachedLookById = (id: string) =>
  createSelector(selectLookById(string), (look) => ({
    …look,
    cachedData: generateCache(look.values),
  }));

Every time the look changes, we cache the look values again so they can be used by other selectors later. The issue is that you might just have renamed a look and not affected the values, in which case, every look rename will trigger a costly caching operation. In such scenarios, a good approach would be to manually memoize the `generateCache(…)` function so it doesn’t recalculate if the values themselves didn’t change.

With the selectors optimized, we also need to limit the amount of times they calculate (back pressure). This is done in a two ways:

1. Use selectSignal for displaying in components.
2. selectSignal is only calculated when the signal is evaluated, reducing the overall amount of calculation as it’ll only be evaluated during Angular’s change detection, and not on every state change.
3. We only need to send network data in 44hz, so every action that changes a fixture fires a FixtureChanged action with the affected fixtures, and around every 23ms we switchMap into a selectFixtureByIds(…) with take(1) to ensure we only listen to that selector once
4. This data is then sent to the network worker, which we will talk a bit more on the next section.

This approach allows us to fire all the actions that are needed while keeping the amount of recalculations to a minimum.

The NativeScript Advantage

NativeScript enables us to use the preferred platform rendering engine (iOS UIKit and SwiftUI in this case) which is optimized for ideal platform performance and behaves completely natural to the target device with the wonderful pairing ability to architect the app entirely with Angular best practices.

Beyond being able to achieve ideal platform rendering, we also gain the development benefits of having the best of all worlds:

1. The ability to use the platform-native APIs where desired from our Angular components and services.
2. While also being able to combine platform techniques covering Swift, Objective C and C++ with our Angular app where desired.

The ability to bring together vast talent across the entire JavaScript ecosystem (not just one area) into unity with platform development is unparalleled in expressiveness, creative control, performance and options to deliver no matter the objective.

To illustrate a small example which brought joy to our developers was the MIDI integration challenge involving interwoven Bluetooth and hard wired devices. iOS presents a unique condition with MIDI integrations which will cache connected devices at a low level (in the C library level) and presents challenges in getting newly connected devices available for use in various MIDI systems. Most off the shelf MIDI plugins do not handle this condition well as it can vary based on various Bluetooth and hard wired device interaction UX flows per app requirements. For example, if a user flow involves selecting from a list of available bluetooth devices and a new device is paired, this case may need special consideration in the flow to not cause user confusion or invalid operational behavior due to the fact that Core MIDI will not know about the pairing right away until notified programmatically.

With NativeScript we were able to reference the iOS docs directly to find that MIDIRestart can be called to force Core MIDI to ask its drivers to rescan for hardware.

Without changing IDE’s or languages, we were able to invoke it directly using TypeScript (the app’s base language) without searching for 3rd party plugins or other integrations, saving us critical time. NativeScript enabled us to use the platform directly.

In the app code, we were able to add the following line to rescan for devices:

// With NativeScript, true platform development is enabled.
// Allow CoreMIDI devices to retrieve accurate state.
MIDIRestart()

This platform API is defined in TypeScript via objc!CoreMIDI.d.ts from @nativescript/types whereby you can configure any project to include types for more advanced APIs per needs which you can learn more about in this blog post from the NativeScript team.

NativeScript also allows us to use Workers for multi-threading. As explained earlier in this post, around every 23ms, if there are any network changes needed, data is sent from the main JS thread to the worker. This is responsible for transmitting the data. Using Worker threads gives us two advantages: first, it needs to transmit at a constant rate, and second, it needs to call blocking C++ APIs from the underlying DMX net libraries.

Since NativeScript allows direct access to native APIs, we can call all the native C++ and Objective-C APIs directly, and the runtime handles conversions (like from a JS array to NSArray). We can also use a SharedArrayBuffer and the underlying (from V8) C++ storage can be used directly by the libraries.

When to consider Angular with NativeScript?

NativeScript and Angular may be the right choice for your project whenever the app needs to target native device platform features and have an interactive fluid UI.

When you can leverage strong Angular skills it provides you the ability to architect the entire app the way you know how with components, services, directives, pipes and the router.

Route handling is often an unspoken benefit because architecting a large app with routes in SwiftUI or UIKit may be completely foreign to an Angular developer but with NativeScript you can configure routing just as you are used to.

Also for demanding cross-platform projects, using NativeScript is a great way to increase talent hiring potential because it leverages innovation from all popular ecosystems (SwiftUI, Kotlin/Compose, React Native, Capacitor or Flutter including your desired choice to architect with Angular) removing the barriers to building dynamic mobile applications.

Conclusion

In summary, the ability to build an Angular app using all the syntax and architecture you’re familiar with is an incredible advantage when paired with NativeScript because you broaden your target delivery options with more creative directions. The practicality of being able to use Angular skills to build UIKit and SwiftUI based interfaces is vast in demanding business-critical situations.

The open source family of the OpenJS Foundation rests at the bedrock of how innovation continues to evolve and grow with NativeScript. You can become involved through the Discord Community if you’d like to see new directions come from the TSC (Technical Steering Committee) which meets on a monthly basis to plan and prepare releases.