MobileVLCKit and VLCKit, part 3

This is part of an article series covering VLC’s Objective-C framework, which we provide to allow inclusion of all its features in third party applications as well as VLC for iOS and Apple TV.

Previously published:

Today, we will discuss thumbnailing of video content. We need to differenciate two key aspects: saving still images of a currently playing video (snapshot) and previewing media stored somewhere without being played (thumbnail). While either way, VLCKit will open the resource, decode the bitstream and provide you with a image, performance and usability will differ.

Thumbnailing

Let’s start with thumbnailing a non playing media source, which can be stored locally or remotely.

@implementation DummyObject <VLCMediaThumbnailerDelegate>

- (void)workerMethod
{
    // 1
    NSURL *url = [NSURL urlWithString:@""];
    VLCMedia *media = [VLCMedia mediaWithURL:url];

    // 2
    VLCMediaThumbnailer *thumbnailer = [VLCMediaThumbnailer thumbnailerWithMedia:media delegate:self];

    // 3
    CGSize thumbSize = CGSizeMake(800.,600.);
    thumbnailer.thumbnailWidth = thumbSize.width;
    thumbnailer.thumbnailHeight = thumbSize.height;

    // 4
    [thumbnailer fetchThumbnail];
}

- (void)mediaThumbnailer:(VLCMediaThumbnailer *)mediaThumbnailer didFinishThumbnail:(CGImageRef)thumbnail
{
    // 5
    if (thumbnail) {
        UIImage *thumbnailImage = [UIImage imageWithCGImage:thumbnail scale:[UIScreen mainScreen].scale orientation:UIImageOrientationUp];
        if (thumbnailImage) {
            // TODO: do something with the thumbnail!
        }
    }
}

- (void)mediaThumbnailerDidTimeOut:(VLCMediaThumbnailer *)mediaThumbnailer
{
     // TODO: Show a reaction
}

@end
  1. We need to create a NSURL instance along with its VLCMedia representation. Note that the URL may point to both a local or a remote resource.
  2. We create the thumbnailer instance for our media and point to ourselves as a delegate to receive the thumbnail.
  3. We define the size of the resulting thumbnail.  If width and height are set to zero, the video’s original size will be used. If you set either width or height to zero, the aspect-ratio is preserved.
  4. Finally, we call the thumbnailer’s worker function.
  5. Asynchronously, after about two to twenty seconds, we will receive a response from the thumbnailer to the delegate. It is important to check the thumbnail for NULL before trying to bridge it to a UIImage or NSImage as well as afterwards as the translation can fail. That’s all.

You might be wondering how the thumbnailer decides which frame to return. This is based on a more complex algorithm currently depending on the media’s duration and availability of key frames. Future versions may also analyze the image content.
You can overwrite this algorithm with the thumbnailer’s snapshotPosition property (with a 0.0 to 1.0 range).

Snapshots

The VLCMediaPlayer class includes a very basic API, which allows the creation of an infinite number of snapshots during playback, which will be asynchronously stored as local files. The size parameters follow the same pattern as for the thumbnailer.

- (void)workerMethod
{
    // ...
    [_mediaplayer saveVideoSnapshotAt:(NSString *)path withWidth:(int)width andHeight:(int)height];
    // ...
}

As soon as the snapshot was stored, a VLCMediaPlayerSnapshotTaken notification is emitted and mediaPlayerSnapshot: is called on the media player’s delegate. Note that the delegate call is available on iOS and tvOS only.
As a convenience starting in VLCKit 3.0 on iOS and tvOS, the media player class exposes the lastSnapshot and snapshots properties, which provide a UIImage instance of the last shot as well as a list of files of the taken shots.

That’s all for today. Enjoy using VLCKit!

VLCKit 3.0

10 days ago, we published VLC media player 3.0 for all platforms. It’s the first major release in three years and brings a huge number of features, improvements and fixes. Get an overview here and the full changelog there.

For VLCKit, we improved performance and memory management, added new APIs and you get all improvements from the underlying libvlc including full support for decoding H264 and H265 using VideoToolbox in hardware. Instead of using all cores of your iPhones CPU at 100%, decoding a 4K video uses less than 20%.
Further you can look at all aspects of a 360° video with touch gesture based controls, discover and browse shares on your network with UPnP, NFS, FTP, SFTP and SMB and more.

As you remember, we published VLC for Apple TV in January 2016, but so far, we never made VLCKit available on tvOS. In addition to MobileVLCKit for iOS, we now introduce TVVLCKit for tvOS!

For macOS, iOS and tvOS, VLCKit 3.0 is available through Cocoapods as a precompiled binary under the LGPLv2.1 license. You can find the source code on our website – contributions welcome!

We are looking forward to all your feedback and the apps deploying VLCKit to deliver multimedia to their users.
Do you want to learn more about integrating VLCKit? Have a look at the tutorials I wrote not too long ago (Part 1, Part 2).

So what did we change in VLCKit, API-wise? Continue reading VLCKit 3.0

Announcing VLC for Apple TV

Today, we are proud to announce VLC on the Apple TV. It’s a full port of VLC media player combined with platform specific features.

VLC for Apple TV integrates with a plethora of devices and services on your local network and includes a custom way of casting files directly to the TV from your other computers using a web browser!

Background
Immediately after Apple announced the future of television, we started to develop for the new box and now it is finally ready to be published. This is the first app we release with the increased limit of 2000 TestFlight beta testers, who had access to our app since mid November and greatly influenced the result. We were excited about the amazing amount of feedback we received.

Our mission for VLC for the Apple TV was to fulfill our claim “plays it all” and we got close:  it supports virtually all formats under the sun (like the iOS version of VLC). Before demonstrating VLC’s advanced playback experience on the Apple TV, let’s discuss about ways how to actually access content!

Content Discovery
VLC on the Apple TV automatically discovers storage and shares on your local network (using the SMB, UPnP media server, FTP and PLEX protocols). You can browse them with the clicker or the Remote, select the content you want and it will play.

During playback, you’ll be able to add subtitles from OpenSubtitles.org. Login credentials can be securely stored on your Apple TV and will be synchronized to your iPad or iPhone (and the other way around) to ease access.

Remote Playback
However, what about network streams and files you don’t have stored on your server? This is where Remote Playback steps in. On this view, VLC shows an address (and an IP) on your Apple TV. Open it using your web browser. Drop a file to play it on the big screen, drop multiple files to queue them. Until storage hits its limits on the Apple TV, the files will remain available locally on device.

For playback of a network stream URL on your TV, just paste it to the field and push return. Once playback concludes, you will find it in the Network Stream view for future use. Like for files, you can send multiple URLs rows in a row and they will be attached to the playback queue.

Finally, the web interface can be used for play/pause and seeking.

Speaking about network streams. URLs recently played on iPad or iPhone will appear along with the ones played on the big screen automatically.

Playback
VLC for the Apple TV supports the basic play, pause and seek. Additionally, you can navigate by chapter and titles or pick from multiple audio and subtitles tracks.
Watching a talk and you’d wish the speaker would be a bit quicker? Set your custom playback speed, a feature we missed from any other player on the Apple TV so far. As VLC is increasingly used to watch talks and documentation in education, it was naturally of high importance for us to include this feature from the start.
You enjoy listening to web radio or music? We believe that just showing album art is a bit boring so we integrated with the community service hatchet.is to show artist imagery and biographies – give it a go!

Let’s go back to subtitles for a second: VLC on the Apple TV supports any kind of text (srt, SSA, WebVTT, …) or bitmap subtitles including full support for Right-to-Left languages like Arabic and Hebrew, complex text layout for Malayalam as well as further Asian languages.
What to do when subtitles are not included with your media? Within the playback view, you can find VLC’s first integration on embedded platforms ever with OpenSubtitles.org!

Outlook
When looking closely, one content source known from VLC’s iOS port is missing so far: Cloud Services! Support for Dropbox, OneDrive and Box is currently in beta testing using the TestFlight platform. It will appear shortly after.

The day Apple announced the new device, we immediately started porting VLC to their new operating system and finally, our initial version was approved! We are excited about this first new port of VLC in 2016 after concluding 2015 with a port to Chrome OS and are really looking forward to your feedback!

I can’t thank Tobias enough for his dedication during the last quarter to make this possible.

VLC for iOS, version 2.7.0

Today, we will release a new major version of VLC for iOS. Note that due to adapted review guidelines, it is branded “VLC for Mobile” on the iOS App Store now.

We have been working on this update for the last few months, so it adds a lot of features:

  • full iOS 9 support incl. split screen and iPad Pro
  • re-written Apple Watch app for watchOS 2
  • system-wide search for content stored locally within the app using “CoreSpotlight”
  • Touch ID support to unlock the app
  • full support for Right-To-Left languages

On the playback and networking side, we added

  • support for SMB  file servers (Windows shares)
  • highly improved subtitles rendering with font fallback for non-western languages
  • desktop quality SSA subtitles rendering (initially introduced in 2.6.6)
  • list of recently played network streams is shared across devices using iCloud
  • stored login credentials for servers is shared across devices using iCloud Keychain
  • stored login credentials for cloud services are shared across devices using iCloud Keychain

This release includes a lot of UI improvements, bug fixes and last but not least a newly designed sidebar menu!

Finally, this release drops support for iOS 6.1. VLC now requires iOS 7.0 or later.

This major update is the result of 4 months of work and I would like to thank TobiasCarola and Pierre for their dedication to make this possible.

Stay tuned, we have more exciting things in the pipeline, which we will happily present very soon.

VLC for iOS, version 2.6.0

Today, we will release version 2.6.0 of VLC for iOS adding support for Apple Watch. With Apple Watch, you can fully control VLC on iPhone, browse the library and retrieve info about stored or playing media items.

Since playback control using Apple Watch allows you to start playback while the app is in background, we took the occasion to completely rework the playback process which culminated in the introduction of a mini player. This way, you can finally browse the media library or network services without interrupting playback.

Further, we added the ability to loop playlists and access to PLEX shares requiring authentication. Finally, thumbnails displayed in the local media library now reflect playback progress and will show the last played frame.

Version 2.6.0 of VLC for iOS includes the largest number of bug fixes and improvements we ever shipped in a single update for this platform. They are affecting virtually all parts of the app and will lead to a greatly improved user experience. Please scroll down if you are curious about the details.

One last thing: since VLC for iOS includes a large amount of white artwork which is virtually invisible in Xcode, Tobias developed a fix for that.

This major update is the result of 3 months of work and I would like to thank Tobias, Carola and Pierre for their dedication to make this possible.


Full change log:

  • Added support for Apple Watch – control VLC from your watch!
    • playback control
    • media info
    • library browsing
  • Added mini-player to browse the library during playback (#13367)
  • Added support for looping playlists
  • New ingest mechanism for audio files
  • Improved remote command support
  • Improved thumbnail generation
  • Improved thumbnail and web interface performance on devices with A5 CPU and above
  • Fixed crash when playing media from a folder or music album on iPad (#14394, #14706)
  • Fixed crash when app goes to background while a video is playing (#14643)
  • Fixed privacy leak when using a passcode to protect the library (#14159, #14615)
  • Removed stray popup announcing VLC’s crash all the time while it actually
    didn’t crash before, but was terminated by the user (#13194)
  • Added support for the PLEX Web API
  • Thumbnails displayed in the media library are updated to the last playback position (#14462)
  • Improved reliability when sharing media library on the local network
  • Improved media library search delivering more accurate results (#14593)
  • Fixed ‘crop to fill screen’ on iPad (#14575)
  • Fixed issue which prevented downloading of a few files via UPnP (#11123)
  • Fixed crash when screen is being locked during playback (#14610)
  • A large number of bug fixes affecting most parts of the app (#13194, #14056#14270, #14284, #14355, #14477, #14588, #14589, #14609, #14623, #14624#14628, #14629, #14635, #14638, #14641, #14642, #14654, #14663, #14687#14688, #14713, #14715, #14716, #14733, #14736, #14795, #14800, #14801, #14829)

VLC for iOS 2.5.0

Today, we released version 2.5.0 of VLC for iOS with a strong focus on cloud integration. We added support for iCloud Drive, box.com and OneDrive. As part of iCloud Drive, you can have access to any cloud storage enabled app on iOS 8 and later. For box.com and OneDrive, we added native support, which allows us direct streaming from the cloud without having the user download a media first.

Furthermore, we cleaned-up the playback experience matching recent changes in VLC for Android and added the full 10-band equalizer known from VLC’s desktop ports. While reworking playback, we also integrated support for titles and chapters included in a few Matroska / MKV and MP4 files. Due to popular request, you can also lock the UI and rotation during playback.

One last thing: you can share your media library between multiple devices on the same local network now! If you enable WiFi sharing and keep VLC active, it will appear in the “Local Network” tab of all iOS devices running VLC supporting both device to device streaming as well as downloads.

Finally, we added a couple of new translations to Portuguese (Portugal), Portuguese (Brazil), Khmer, Faroese, Belarusian, Serbian (Latin), Tamil and Afrikaans along with the usual set of bug fixes and minor improvements.

We hope you like this new version as much as we do.

VLC for iOS 2.4.0 & 2.4.1

After a 5 month hiatus and a couple of delays in between, we are very happy to release version 2.4.1 of VLC for iOS to the general public today. Please see our press release for cross-platform information.

As an introductory remark, make sure to get the legitimate version of VLC. There are a number of clones on the iOS App Store right now, which violate both our copyright and our trademarks, include advertisement and usually charge $1 to $5 per download. We sent 39 DMCA take down notices against 48 apps over the cause of the last 12 months and in the end, we always succeeded. However, this can take up to 3 months, even for seemingly obvious violations.

As a consequence to those continued violations, VLC for iOS’s source code repository will no longer be publicly accessible, but archives of the stable releases only.

So, what’s new in VLC for iOS?

  • Support for iPhone 6 and 6+
  • Native support for Plex using their custom protocol
  • File Sharing with third party apps
  • Media library search
  • Custom URL scheme based on x-callback-url
  • Greatly improved UPnP support
  • In addition to uploading media via WiFi, you can also download all the media stored within VLC using a simple web browser
  • Streaming and folder support for Google Drive
  • Flat folder synchronization via iTunes
  • Options for default playback speed and FTP text encoding
  • On-the-fly audio and subtitles track synchronization
  • And a large number of small features, improvements and bug fixes as well as a new translation to Traditional Chinese.

This release removes support for Dolby Digital (AC-3), Dolby Digital Plus (E-AC-3) and Dolby TrueHD (MLP) for the foreseeable future due to a content dispute with Dolby Laboratories, which could not be resolved in a different way.

We are very happy with this major version of VLC for iOS and hope that you’ll like it as much as we do. Further, we are excited about the things we have in our pipeline for the forthcoming releases. A first beta of version 2.5 will be released to our testers today.

VLC media player for Mac OS X 2.2.0

Today, we will also a major new version of VLC for Mac OS X. For cross-platform changes, please have a look at the release notes and our press release.

What’s new in VLC for Mac?

  • Support for OS X Yosemite
  • Completely re-written web plugin for Chrome, Safari and Firefox is back!
    Improved fullscreen behavior
  • Continue playback where you left off
  • Improved playlist adding a file size column and an option to increase the font size
  • In addition to iTunes, Spotify can be paused on playback start
  • New encryption and decryption modules for SSL based on OS X’s SecureTransport library for FTP and HTTP connections. This greatly improves speed and security.
  • A lot of improvements in VLCKit for use in third party applications, notably
    • Switched the code base to ARC and added support for Swift projects
    • Support for HLS and HTTPS playback on iOS
    • Improved thumbnailing
    • Various new APIs for playlist handling, the equalizer, thumbnailing and meta data handling

We are excited about this major update of VLC for Mac and hope that you’ll like it as much as we do.

MobileVLCKit and VLCKit, part 2

This is part of an article series covering VLC’s Objective-C framework, which we provide to allow inclusion of all its features in third party applications as well as VLC for iOS and Apple TV.

Currently published:

Today, we will discuss meta data processing.

In VLCKit, every item you play is a VLCMedia object. For typical use cases, it can be created with an NSURL or an NSString containing a path.

We differentiate between two types of meta data: technical information describing the media such as codec, bitrate, video size and user-visible/-provided information such as artist, publisher name, album title.

Let’s start with the technical information, which can be retrieved from any media object with a single API call:

@property (NS_NONATOMIC_IOSONLY, readonly, copy) NSArray *tracksInformation;

This gives you an array containing as many dictionaries as the media contains tracks, be it video, audio or subpictures (subtitles, teletext). The track type is defined by the VLCMediaTracksInformationType key. All tracks will include information about the respective codec, bitrate and encoding details. Depending on track type, keys for video dimensions, audio channel numbers or text encoding will be set as well as an optional key for language.
Retrieving this property can potentially be very expensive, especially if your media is stored remotely as VLCKit will synchronously parse the source to provide this information. Therefore, we recommend you to cache this data, as it will probably not change during the lifetime of the VLCMedia object.
Note: for codec information, you’ll receive an integer which is a raw FOURCC representation of the codec name. Releases following VLCKit 2.2 will include a convenience method to translate it to an end-user readable string.

On mobile devices, you might run into the question if a given device is powerful enough to decode a given video file. For this purpose, VLCMedia includes the isMediaSizeSuitableForDevice property which will provide a reasonable guess. Note that this property will always be true on OS X.

Now, what about non-technical information about the media contents? To retrieve them, VLCKit needs to parse the source. This can be done both synchronous and asynchronous depending on the needs of your application. We generally recommend you to use the asynchronous way so you don’t block the execution of any threads. VLCMedia includes an optional delegate protocol, which allows you to follow meta data processing by receiving notifications every time further information becomes available as well as once parsing finished.
VLCKit can provide up to 17 different meta data keys with more to come in subsequent releases. While it allows to fetch them one by one, we recommend you to fetch the full dictionary using:

@property (nonatomic, readonly, copy) NSDictionary * metaDictionary;

If you have write access to your media source, you can also set values for the respective keys and save them to disk.

Thanks for reading!

If you have questions, don’t hesitate to use the comment section or to shoot a mail. The next part of this series will be about thumbnail creation.

MobileVLCKit and VLCKit, part 1

This is part of an article series covering VLC’s Objective-C framework, which we provide to allow inclusion of all its features in third party applications as well as VLC for iOS and Apple TV.

Currently published:

VLCKit is a generic library for any audio or video playback needs on both Apple platforms and also supports streaming and media to file conversations on the Mac. It is open-source software licensed under the LGPLv2.1 or later, available in source code and binary form from the VideoLAN website. You can also integrate MobileVLCKit easily via CocoaPods.

The foundation of this framework was laid in 2007 by Pierre d’Herbemont as part of a Google Summer of Code student project, in fact kind of mentored by me. Since then, it played major roles in developing the Lunettes prototype in 2008/2009, MobileVLC in 2010/2011 as well as take-two of VLC for iOS in 2013 until present.

When do you need VLCKit? Frankly always when you need to play media not supported by QuickTime / AVFoundation or if you require more flexibility. You want to play something else besides H264/AAC files or HLS streams? You need subtitles beyond QuickTime’s basic support for Closed Captions? Your media source is not your mobile device and not a basic HTTP server either, but perhaps a live stream hailing from some weird media server or even a raw DVB signal broadcasted on a local network? Then, VLCKit is for you.

But this is open-source software right? What does this mean for me and the end-user? And wasn’t MobileVLC removed from the App Store in 2011 for some crazy licensing reason?

First of all, open-source means for you, that you get access to the whole stack. There is no blackbox, all the sources are there at your fingertips. No reverse-engineering needed, no private APIs.

Then again, this must not be the case for your software. The LGPLv2.1 allows our software to be included in proprietary apps, as long as you follow the license. As a start, make sure to publish any potential changes you do to our software, make sure that the end-user is aware that VLCKit is embedded within your greater work and that s/he is aware of the gained rights. S/he is granted access to our code as well as to your additions to our work. For further details, please read the license and consult your lawyer with any questions you might have.

Regarding the removal of MobileVLC, the world moved on a bit. libvlc and VLCKit are no longer under the GPLv2 but were relicensed to the LGPLv2.1. While this looks like a play on characters, it is of major importance regarding distribution on the Mac or iOS App Store. As of now, the Terms of Service of either store are still incompatible with the GPLv2 (despite major changes and improvements), while distribution under the LGPLv2.1 is absolutely permitted.

Let’s get going with a technical overview and start with a layer model:

Any layer can only communicate with the layer above and the layer below. Therefore, decoders are four abstraction layers away from the client app developer. Why is that a good thing? By having a decent amount of abstraction, it is no longer necessary for you to know which library is taking care of the decoding, rendering or parsing. You are wondering whether you need to decode your certain video in a multi-threaded or single-threaded way, think that decoding might be accelerated through the hardware in the iPhone 5S and above, but pondering what to do with older releases? libvlc will do the correct thing for you.
However, we allow flexibility. You know that you want to use a certain option on libavcodec because it’s cool (and for some reason, we decided against it?)? that’s a one-liner for you.

Okay, so how do you get going? Have a look at the SimplePlayback sample code available for download here. Ignoring the boilerplate code present in any empty iOS app project, this basic app shows you how to play a video in 8 lines of code and to allow the user to play or pause playback:

@interface RandomViewController ()
{
    VLCMediaPlayer *_mediaplayer;
}
@end
@implementation RandomViewController
- (void)viewDidLoad {
    [super viewDidLoad];
    /* setup the media player instance, give it a delegate and
     * something to draw into */
    _mediaplayer = [[VLCMediaPlayer alloc] init];
    /* assign a random UIView or NSView to the player to
     * draw the video */
    _mediaplayer.drawable = self.movieView;
    /* create a media object and give it to the player */
    _mediaplayer.media = [VLCMedia mediaWithURL:
     [NSURL URLWithString:@“http://yourserver.com/folder/file.mkv“]];
}
- (IBAction)playandPause:(id)sender {
    if (_mediaplayer.isPlaying)
        [_mediaplayer pause];
    [_mediaplayer play];
}
@end

Give it a spin!

Then again, chances are that you are on iOS and you want to have a slightly more advanced user experience. Need a seek slider and a time display or would you like to crop the video a bit? Your media includes more than one audio or subtitles track and the user should be able to switch? The code is there for you. We wrote a self contained class and distribute it under the BSD 2-clause license. If you are interested, checkout the Dropin-Player project available here.

That’s it for the first part of this article. The second part will cover more advanced use-cases of VLCKit and will follow shortly.

If you have questions, don’t hesitate to use the comment section or to shoot a mail.

One last word regarding VLC for iOS and taking code from that project. A contrario to VLCKit or the provided sample code, it is protected by different licenses and different obligations. It is dual licensed under the GPLv2 as well as the MPLv2. Most importantly, it is absolutely not permitted to take any bits of code it unless your product is distributed under a compatible license. We have a good DMCA track record and we are monitoring both the Mac and the iOS App Stores constantly.