Become an AceSign inSign up
Cay-Zhang/ SwiftSpeech (v0.9.3) 3 months ago
Ruby Swift swiftui speech-recognition iOS voice-recognition
View on GitHub
Need help with SwiftSpeech?
Click the “chat” button below for chat support from the developer who created it, or find similar developers for support.

About the developer

Cay-Zhang
198 Stars 19 Forks MIT License 60 Commits 1 Opened issues

Description

A speech recognition framework designed for SwiftUI.

Services available

!
?

Need anything else?

Contributors list

cayZ Cay-Zhang
# 37,988
Swift
voice-r...
instagr...
Twitter
 59 commits

SwiftSpeech

Speech Recognition Made Simple

A few lines of code to do this!

Recognize your user's voice elegantly without having to figure out authorization and audio engines.

SwiftSpeech Examples

Aside from the readme, the best way to learn more about SwiftSpeech and how speech recognition capabilities are implemented in apps like WeChat is to check out my new project SwiftSpeech Examples. For now, it contains a WeChat voice message interface mock and the three demos in SwiftSpeech.

WeChat

Features

SwiftSpeech is a wrapper for Apple's Speech framework with deep SwiftUI and Combine integration.

  • [x] UI control + speech recognition functionality in just several lines of code.
  • [x] Customizable cancelling.
  • [x] SwiftUI style reactive APIs and Combine support.
  • [x] Highly customizable but also keeping your code highly reusable via a composable structure.
  • [x] Fully open low-level APIs.

Installation

Swift Package Manager (Recommended)

In Xcode, select 

Add Packages...
from the 
File
menu and enter the following package URL: 
html
https://github.com/Cay-Zhang/SwiftSpeech

CocoaPods

pod 'SwiftSpeech'

Getting Started

1. Authorization

Although SwiftSpeech takes care of all the verbose stuff of authorization for you, you still have to state the usage descriptions and specify where you want the authorization process to happen before you start to use it.

Usage Descriptions in Info.plist

If you haven't, add these two rows in your 

Info.plist
: 
NSSpeechRecognitionUsageDescription
and 
NSMicrophoneUsageDescription
.

These are the messages your users will see on their first use, in the alerts that ask them for permission to use speech recognition and to access the microphone.

Here's an exmample: 

xml
NSSpeechRecognitionUsageDescription
This app uses speech recognition to convert your speech into text.
NSMicrophoneUsageDescription
This app uses the mircrophone to record audio for speech recognition.

Request Authorization

Place 

SwiftSpeech.requestSpeechRecognitionAuthorization()
where you want the request to happen. A common location is inside an 
onAppear
modifier. Common enough that there is a snippet called Request Speech Recognition Authorization on Appear exposed in the Xcode Modifiers library. 
swift
.onAppear {
    SwiftSpeech.requestSpeechRecognitionAuthorization()
}

2. Try some demos

You can now start to try out some light-weight demos bundled with the framework using Xcode preview. Click the "Preview on Device" button to try the demo on your device. ``

swift
static var previews: some View {
    // Two of the demo views below can take a
localeIdentifier: String` as an argument. // Example locale identifiers: // 简体中文（中国）= "zhHansCN" // English (US) = "enUS" // 日本語（日本）= "jaJP" 
Group {
    SwiftSpeech.Demos.Basic(localeIdentifier: yourLocaleString)
    SwiftSpeech.Demos.Colors()
    SwiftSpeech.Demos.List(localeIdentifier: yourLocaleString)
}

} ```

Here are the "previews" of your 

previews
:

Demos

3. Build it yourself

Knowing what this framework can do, you can now start to learn about the concepts in SwiftSpeech.

Inspect the source code of 

SwiftSpeech.Demos.Basic
. The only new thing here is this: 
swift
SwiftSpeech.RecordButton()                                        // 1. The View Component
    .swiftSpeechRecordOnHold(sessionConfiguration:animation:distanceToCancel:)  // 2. The Functional Component
    .onRecognizeLatest(update: $text)                             // 3. SwiftSpeech Modifier(s)
There are three parts here (and luckily, you can customize every one of them!): 1. The View Component: A 
View
that is only responsible for UI. 2. The Functional Component: A component that handles user interaction and provides the essential functionality of speech recognition. In the built-in one here, the first two arguments let you specify the configuration for the recording session (locales and more) and an animation used when the user interacts with the View Component. The third argument sets the distance the user has to swipe up in order to cancel the recording. The framework also provides another Functional Component: 
.swiftSpeechToggleRecordingOnTap(sessionConfiguration:animation:)
. 3. SwiftSpeech Modifier(s): One or more components allowing you to receive and manipulate the recognition results. They can be stacked together to create powerful effects.

For now, you can just use the built-in View Component and Functional Component. Let's explore some SwiftSpeech Modifiers first since every app handles its data differently:

Important: Chaining multiple or identical SwiftSpeech Modifiers together doesn't override any behavior. All actions of the modifiers will be executed in the order where the closest to the Functional Component executes first and the farthest executes last.

// 1
// All three demos use these modifiers.
// Inspect the source code of them if you want examples!
.onRecognizeLatest(
    includePartialResults: Bool = true,
    handleResult: (SwiftSpeech.Session, SFSpeechRecognitionResult) -> Void,
    handleError: (SwiftSpeech.Session, Error) -> Void
)


.onRecognize(
    includePartialResults: Bool = true,
    handleResult: (SwiftSpeech.Session, SFSpeechRecognitionResult) -> Void,
    handleError: (SwiftSpeech.Session, Error) -> Void
)


// This one simply assigns the recognized text to the binding in handleResult and ignores errors.
.onRecognizeLatest(
    includePartialResults: Bool = true,
    update: Binding
)


// This one prints the recognized text and ignores errors.
.printRecognizedText(includePartialResults: Bool = true)

The first group of modifiers encapsulates the core value of SwiftSpeech. It does all the publisher transformation and subscription for you and calls the closures with enough information to facilitate a sophisticated task when a recognition result is yielded.

onRecognizeLatest
ignores recognition results from the last recording session (if any) when a new session is started, while 
onRecognize
subscribes to results from every recording session.

In 

handleResult
, the first closure parameter is a 
SwiftSpeech.Session
, which has a unique 
id
for every recording. Use it to distinguish the recognition result from one recording and that from another.

The second is a 

SFSpeechRecognitionResult
, which contains rich information about the recognition. Not only the recognized text (

result.bestTranscription.formattedString
), but also interesting stuff like speaking rate and pitch!

In 

handleError
, you will handle the errors produced in the recognition process and also during the initialization of the recording session (such as a microphone activation failure). 
// 2
.onStartRecording(appendAction: (SwiftSpeech.Session) -> Void)
.onStopRecording(appendAction: (SwiftSpeech.Session) -> Void)
.onCancelRecording(appendAction: (SwiftSpeech.Session) -> Void)

The second group gives you utter control over the whole lifespan of a 

SwiftSpeech.Session
. It runs the provided closures after a recording was started/stopped/cancelled. Inside the closures, you will have access to the corresponding 
SwiftSpeech.Session
, which is discussed below. 
// 3
// `SwiftSpeech.ViewModifiers.OnRecognize` uses these modifiers.
// Inspect the source code of it if you want examples!
.onStartRecording(sendSessionTo: Subject)
.onStopRecording(sendSessionTo: Subject)
.onCancelRecording(sendSessionTo: Subject)

The third group might be useful if you prefer a reactive programming style. The only new argument here is a 

Combine.Subject
(e.g. 
CurrentValueSubject
and 
PassthroughSubject
) and the modifier will send the corresponding 
SwiftSpeech.Session
to the 
Subject
after a recording is started/stopped/cancelled.

SwiftSpeech.Session

Configuration

A session can be configured using a 

SwiftSpeech.Session.Configuration
struct. A configuration contains information such as the locale, the task hint, custom phrases to recognize, options for on-device recognition, and audio session configurations. Inspect 
SwiftSpeech.Session.Configuration
for more details.

Customized Subscription to Recognition Results

If the built-in 

onRecognize(Latest)
modifiers do not satisfy your needs, you can subscribe to recognition results via 
onStart/Stop/CancelRecording
.

A 

Session
publishes its recognition results via its 
resultPublisher
. It has an 
Output
type of 
SFSpeechRecognitionResult
and an 
Failure
type of 
Error
.

You will receive a completion event when the 

Session
finishes processing the user's voice (i.e. 
result.isFinal == true
), an error happens, or you have explicitly called the 
cancelRecording()
on the session.

A 

Session
also has a convenient publisher called 
stringPublisher
that maps the results to the recognized string.

Independent Use

Here's an example of using 

Session
to recognize user's voice and receive updates. 
swift
let session = SwiftSpeech.Session(configuration: SwiftSpeech.Session.Configuration(locale: Locale(identifier: "en-US"), contextualStrings: ["SwiftSpeech"]))
try session.startRecording()
session.stringPublisher?
    .sink { text in
        // do something with the text
    }
    .store(in: &cancelBag)
For more, please refer to the documentation of 
SwiftSpeech.Session
.

Customized View Components

A View Component is a dedicated 

View
for design. It does not react to user interaction directly, but instead reacts to its environments, allowing developers to only focus on the view design and making the view more composable. User interactions are handled by the Functional Component.

Inspect the source code of 

SwiftSpeech.RecordButton
(again, it's not a 
Button
since it doesn't respond to user interaction). You will notice that it doesn't own any state or apply any gestures. It only responds to the two variables below. 
@Environment(\.swiftSpeechState) var state: SwiftSpeech.State
@SpeechRecognitionAuthStatus var authStatus

Both are pretty self-explanatory: the first one represents its current state of recording, and the second one indicates the authorization status of speech recognition.

Here are more details of 

SwiftSpeech.State
: 
enum SwiftSpeech.State {
    /// Indicating there is no recording in progress.
    /// - Note: It's the default value for `@Environment(\.swiftSpeechState)`.
    case pending
    /// Indicating there is a recording in progress and the user does not intend to cancel it.
    case recording
    /// Indicating there is a recording in progress and the user intends to cancel it.
    case cancelling
}


authStatus
here is a 
SFSpeechRecognizerAuthorizationStatus
. You can also use 
$authStatus
for a short hand of 
authStatus == .authorized
.

Combined with a Functional Component and some SwiftSpeech Modifiers, hopefully, you can build your own fancy record systems now!

Support SwiftSpeech Modifiers

The library provides two general functional components that add a gesture to the view it modifies and perform speech recognition for you: ```swift // They already support SwiftSpeech Modifiers. func swiftSpeechRecordOnHold( sessionConfiguration: SwiftSpeech.Session.Configuration = SwiftSpeech.Session.Configuration(), animation: Animation = SwiftSpeech.defaultAnimation, distanceToCancel: CGFloat = 50.0 ) -> some View

func swiftSpeechToggleRecordingOnTap( sessionConfiguration: SwiftSpeech.Session.Configuration = SwiftSpeech.Session.Configuration(), animation: Animation = SwiftSpeech.defaultAnimation ) 

If you decide to implement a view that involves a custom gesture other than a hold or a tap, you can also support SwiftSpeech Modifiers by adding a delegate and calling its methods at the appropriate time:
swift var delegate = SwiftSpeech.FunctionalComponentDelegate() ``
For guidance on how to implement a custom view for speech recognition, refer to
ViewModifiers.swift` and SwiftSpeechExamples. It is not that hard, really.

License

SwiftSpeech is available under the MIT license.

We use cookies. If you continue to browse the site, you agree to the use of cookies. For more information on our use of cookies please see our Privacy Policy.